01
Select one pipeline
Choose a legacy Snowflake/Airflow pipeline as the next migration unit rather than changing the full platform at once.
Selected Engineering Work · Project 03
Enterprise Data Platform Modernization
Migrating complete data pipelines to a single Databricks Lakehouse platform.
Modernized a centralized retail data platform by migrating complete pipelines from Snowflake and Airflow to Databricks, consolidating ingestion, orchestration and transformation while preserving business logic and data consistency.
The challenge
Modernize the platform without losing the behavior embedded in legacy pipelines.
The existing data platform combined Snowflake for data processing and storage with Airflow for file ingestion, dependency management and scheduling. The modernization goal was to consolidate those responsibilities onto Databricks and create a more scalable single-platform architecture.
The hardest part was not simply translating code. Each legacy pipeline contained business logic and assumptions that had to be understood before the pipeline could be rewritten safely. Data consistency between the legacy and modernized implementations was a core requirement throughout the migration.
- Understand business transformations embedded in existing pipelines.
- Rebuild complete pipeline behavior on Databricks.
- Preserve data consistency during a gradual migration.
- Remove the operational dependency on both Snowflake and Airflow.
Architecture transition
From a split Snowflake + Airflow stack to a unified Databricks platform.
Databases, files and retail transaction data continued to feed the platform, while ingestion, orchestration and transformation responsibilities moved into Databricks.
Sources
Databases + files
Retail transaction and enterprise source data.
Orchestration
Airflow
File ingestion, schedules and pipeline dependencies.
Legacy platform
Snowflake
Existing pipeline processing, transformations and data workloads.
Consumption
Analytics + reporting
Enterprise reporting and downstream data use.
Sources
Databases + files
Retail transaction and enterprise source data.
Unified platform
Databricks ingestion + orchestration
Schedules, dependencies and ingestion moved onto the target platform.
Lakehouse
Bronze → Silver → Gold
Clear transformation boundaries from raw to business-ready data.
Consumption
Analytics + reporting
Validated outputs delivered from the Databricks-based platform.
Migration approach
A controlled migration cycle repeated one pipeline at a time.
The platform was modernized gradually rather than through a big-bang cutover. Each pipeline moved through the same sequence before production deployment.
02
Understand legacy logic
Analyze transformations, dependencies and business assumptions before introducing implementation changes.
03
Rewrite for Databricks
Rebuild the complete pipeline on Databricks and structure transformations using Bronze, Silver and Gold layers.
04
Test and reconcile
Run row-count checks and old-vs-new data comparisons against the legacy implementation.
05
Validate with business
Review the migrated output with business stakeholders before replacing the production pipeline.
06
Deploy and repeat
Move the validated pipeline into production, then continue the same controlled process with the next pipeline.
Medallion Architecture
Clear transformation boundaries from source data to business-ready output.
The rewritten pipelines were organized using a Bronze, Silver and Gold approach. This gave the target platform a consistent structure and separated ingestion concerns from cleansing, transformation and business-facing outputs.
- Bronze: ingested source data retained as the raw platform layer.
- Silver: cleaned and transformed data with reusable processing logic.
- Gold: business-ready datasets used by analytics and reporting.
Validation strategy
Prove equivalence before replacing each legacy pipeline.
Each migrated pipeline was validated against the Snowflake-based implementation before production deployment. Technical reconciliation focused on row counts and old-vs-new output comparison to expose missing, unexpected or inconsistent data.
Technical validation was followed by business review. Only after the migrated result was accepted was the Databricks pipeline deployed to production.
- Row-count reconciliation between legacy and Databricks outputs.
- Old-vs-new data comparison to verify consistency.
- Business validation before production cutover.
Key engineering decisions
Reduce migration risk while simplifying the long-term platform.
Migrate incrementally
Move one pipeline at a time so each migration could be understood, validated and deployed independently.
Preserve behavior before replacing architecture
Understand existing business logic first and use the legacy output as the reference during the rewrite.
Consolidate onto Databricks
Move ingestion, orchestration and transformation responsibilities onto one platform so Snowflake and Airflow could be removed from the final architecture.
Outcome
A consolidated Databricks platform built through controlled, validated migration.
The migration progressively replaced the Snowflake and Airflow architecture with a Databricks-based Lakehouse platform. Complete pipelines were redesigned around Bronze, Silver and Gold layers, while ingestion, orchestration and transformation responsibilities were consolidated onto Databricks.
Each pipeline moved through legacy-logic analysis, implementation, row-count and old-vs-new validation, business review and production deployment. This controlled approach helped preserve data consistency while simplifying the final platform architecture.