01
Establish the baseline
Use current production data as the trusted reference against which candidate pipeline output is validated.
Selected Engineering Work · Project 01
Enterprise Data Validation Framework
Validating business-logic changes before production release.
Designed and built an end-to-end validation framework to compare output produced by code-freeze pipeline logic against the current production data baseline, helping identify expected and unexpected impacts before release.
The challenge
Understand the impact of changing business logic.
During code freeze, pipeline logic could include added, removed or modified business rules. Before those changes reached production, the resulting data needed to be compared with the current production baseline.
The validation process had to expose both structural and data-level differences while making it possible to separate intended changes from unexpected discrepancies that could indicate a regression or data-quality problem.
- Detect added or removed columns.
- Identify data-type changes such as
INT → BIGINT. - Compare aligned records at row and column level.
- Apply configurable rules to focus validation on relevant differences.
Validation architecture
Candidate output compared against the production baseline.
The framework brought both sides into a consistent comparison process, aligned records, evaluated schema and data differences, and persisted structured results for review.
Baseline
Production data
Current trusted output used as the comparison reference.
Candidate
Code-freeze output
Result of added, removed or modified business logic.
Step 1
Record alignment
Surrogate keys establish comparable record identity.
Step 2
Schema + data comparison
Structural checks plus row- and column-level difference logic.
Output
Delta tables + dashboards
Configuration-filtered results exposed for validation and review.
Engineering approach
A reusable comparison process, not a one-off check.
The implementation evolved from SQL-based comparison logic toward PySpark and used configuration-driven rules to keep validation behavior separate from the processing code.
02
Generate candidate output
Execute the code-freeze version containing newly added, removed or modified business logic.
03
Align comparable records
Create surrogate keys to establish a reliable basis for comparing corresponding records.
04
Compare schema and data
Detect schema drift and evaluate differences at row and column level. The comparison logic started in SQL and later moved to PySpark.
05
Apply configurable rules
Drive filtering and validation behavior from a configuration table rather than embedding every condition in the processing logic.
06
Publish validation results
Persist structured results in Delta tables and surface them through dashboards for downstream review.
Key engineering decisions
Design choices that made the validation repeatable.
Surrogate-key based alignment
Record comparison requires stable identity. Surrogate keys were introduced to align corresponding records before row- and column-level checks were applied.
SQL → PySpark evolution
The comparison logic was initially implemented in SQL and later moved to PySpark, keeping the solution aligned with the Databricks processing model.
Configuration-driven validation
A configuration table separated validation rules from processing logic, making the framework easier to adapt as comparison requirements changed.
Outcome
A structured release-validation workflow for data changes.
The framework provided a repeatable way to assess the impact of business-logic changes before release, detect unexpected schema and data-level differences, filter results using configurable rules, and make validation findings available through Delta tables and dashboards.