Data & AI Engineering
Data warehouse migration to a Databricks lakehouse
Challenge
- 10+ year-old on-premise data warehouse with rising licence and operating costs
- Management and regulatory reports that could not be disrupted during the move
- Long daily load windows; new data requests waiting for months
Solution
- Mapping of report and pipeline dependencies; migration plan by business domain
- Databricks lakehouse build; ETL rewritten in Spark
- Parallel runs with automated reconciliation in every domain
- Legacy warehouse decommissioned; platform handed over to the bank's data team