Question

Corrupted‑to‑Gold: MDM Foundations

This build takes intentionally corrupted CSV files and transforms them into clean, trusted, analytics‑ready Delta tables. The pipeline demonstrates real‑world data engineering constraints: inconsistent delimiters, malformed dates, broken encodings, duplicate entities, and multi‑format ingestion.

The solution includes:

✔ RAW → BRONZE ingestion

• Normalized delimiters

     Removed non‑ASCII noise
     Safe splitting logic
     Header detection
     Schema validation
  

✔ SILVER cleansing

• Email, phone, country, and status normalization

     Multi‑format date parsing
     Null handling and type correction
  

✔ Survivorship scoring (MDM‑light)

• Email validity

    Phone completeness
    Date reliability
    Status correctness
    Row‑level scoring + best‑record selection
  

✔ GOLD outputs

• Clean, deduped, resequenced customer entities

     Delta‑optimized storage
     Ready for downstream MDM and ADF Mapping Data Flows
  

This project sets the foundation for the full MDM pipeline (matching, clustering, survivorship, and golden‑record creation) and the upcoming ADF MDF suite demonstrating schema drift, parameterization, joins, window functions, branching, SCD‑light, partitioning, and KPI generation.

If you want the full code, architecture, or notebooks, the repo is here:

github.com/usman19zafar/Project_corrupted-to-gold-data-journey