Data Engineering
Data Engineering14 min read•
Deep Dive into Medallion Architecture
Advanced patterns for implementing Bronze, Silver, Gold data layers.
By Administrator
The medallion architecture provides a structured approach to data refinement, moving from raw ingestion to business-ready analytics.
The Three Layers
Bronze (Raw) Landing zone for ingested data: - Exact copy from sources - Full history preserved - Minimal transformations - Schema-on-read flexibility
Silver (Cleansed) Validated and standardized data: - Data quality applied - Business rules enforced - Deduplication completed - Standardized schemas
Gold (Curated) Business-ready consumption layer: - Aggregations and metrics - Star schema models - Optimized for reporting - Semantic layer aligned
Bronze Layer Patterns
Full Snapshots Copy entire tables on each run: - Simple to implement - Good for small tables - Higher storage costs
Incremental Loads Capture only changes: - Efficient for large tables - Requires change tracking - Lower storage and compute
Raw Format Preservation Keep original format: - JSON as-is - CSV with original encoding - Enables reprocessing
Silver Layer Patterns
Type 2 Slowly Changing Dimensions Track historical changes: - Start and end date columns - Current flag indicator - Full history preserved
Data Quality Gates Stop bad data from progressing: - Null checks - Range validations - Referential integrity - Quarantine failures
Standardization Consistent data formats: - Date formats - Currency handling - Code standardization
Gold Layer Patterns
Star Schema Optimized for BI: - Fact tables with measures - Dimension tables with attributes - Clear relationships
Aggregation Tables Pre-computed summaries: - Daily, weekly, monthly - By common groupings - Performance optimization
Frequently Asked Questions
Is medallion architecture required in Fabric?
No, medallion is a recommended pattern but not required. It provides clear data lineage and progressive quality improvement. Smaller projects might simplify, but enterprise implementations benefit from the structure.
How do I handle data quality failures in medallion architecture?
Implement quarantine tables in the Bronze or Silver layer. Records failing quality checks go to quarantine for review while valid data progresses. Track and resolve quarantined records through a defined process.
Microsoft FabricMedallion ArchitectureData EngineeringBest Practices