- Synthetic source services emit JSON events to Kafka topics.
- The landing consumer writes messages to S3 or MinIO JSONL partitions.
- Bronze ingestion keeps raw JSON and adds metadata such as ingestion timestamp, source topic, source file, batch ID, and processing date.
- Silver transforms parse fields, standardize timestamps, deduplicate event IDs, normalize payment statuses, validate required fields, and write invalid records to quarantine.
- Gold aggregations produce revenue, customer, product, inventory, payment reliability, and DQ scorecard tables.
- Plotly Dash reads the Gold layer and presents executive analytics.