Financial Services · Databricks
Capital-markets firm rebuilds intraday risk on a Databricks lakehouse
4x lift in fraud-model recall · sub-second risk refresh on the trading desk
At a glance
Key metrics
Challenge
The situation
Intraday risk refreshed on a 15-minute cycle — too slow for desk traders and not reproducible for the model-validation team. The existing Hadoop cluster was expensive, brittle, and out of vendor support.
Approach
How we delivered
- 01 Migrated the risk feature store to Delta Lake with change-data-feed, enabling reproducible point-in-time training sets.
- 02 Re-engineered the fraud model on Databricks ML, adding gradient-boosted ensembles and graph-based signals.
- 03 Rolled out Unity Catalog for lineage + fine-grained access control acceptable to the CRO.
Architecture
Solution architecture
Streaming trades land via Kafka into Delta Lake tables using Spark Structured Streaming. Feature store served from Databricks Feature Engineering. Models trained and deployed through MLflow + Mosaic AI Model Serving with champion-challenger routing. Risk dashboards rebuilt on Databricks SQL with sub-second interactive query.
Outcomes
Measured results
- 4x recall improvement on the priority fraud segment, validated on 6 months of out-of-time data.
- Intraday risk refresh moved from 15 minutes to sub-second on the trading desk.
- 40% compute-cost reduction versus the retired Hadoop estate.
Technology
Tech stack
Platform
- Databricks on AWS
- Delta Lake
- Unity Catalog
Streaming
- Apache Kafka
- Spark Structured Streaming
ML
- Databricks ML
- MLflow
- Mosaic AI Model Serving
Analytics
- Databricks SQL
- Power BI
“We were done patching Hadoop. Apptad gave us a lakehouse our quants actually want to build on.”
Related
More outcomes
Chasing a similar outcome?
Tell us your target — cycle time, cost, risk, adoption. We'll walk you through what we've delivered closest to it.