databricks-streaming-cdc-governance-lakehouse

Databricks-idiomatic streaming CDC lakehouse on open-source Delta Lake + PySpark Structured Streaming + Scala UDFs, governed by Apache Polaris (OSS Unity Catalog), with full Databricks Asset Bundles + Delta Live Tables-style declarative pipelines. Runs end-to-end on docker-compose (Redpanda + Debezium + delta-spark + Polaris + Azurite + LocalStack) at zero-dollar cost.

---

Architecture

flowchart LR
    subgraph OLTP["🗄️ OLTP"]
        PG[(Postgres<br/>orders/customers)]
        MY[(MySQL<br/>inventory)]
    end

    subgraph CDC["🌀 CDC"]
        DBZ[Debezium 2.6<br/>connectors]
        KC[Kafka Connect]
        KFK{{Redpanda<br/>Kafka API}}
    end

    subgraph Stream["⚡ Structured Streaming"]
        BZ[Bronze PySpark<br/>cdc_consumer.py<br/>Delta + CDF]
        SV[Silver PySpark<br/>MERGE + Scala UDFs<br/>+ liquid clustering]
        GD[Gold PySpark<br/>Aggregations +<br/>Z-ORDER + OPTIMIZE]
    end

    subgraph Storage["🪣 Multi-cloud storage"]
        S3[(AWS S3<br/>LocalStack)]
        ADLS[(Azure ADLS Gen2<br/>Azurite)]
    end

    subgraph Gov["🔐 Governance"]
        POL[Apache Polaris<br/>REST catalog]
        GLUE[AWS Glue Catalog<br/>LocalStack]
        HMS[Hive Metastore<br/>fallback]
    end

    subgraph Decl["📜 Declarative"]
        DAB[Databricks Asset<br/>Bundles YAML]
        DLT[DLT-style SQL<br/>dlt_pipelines/]
    end

    subgraph Obs["🔍 Observability"]
        CDF[Delta Change<br/>Data Feed]
        TT[Time Travel<br/>snapshot queries]
    end

    subgraph Orch["🎯 Orchestration"]
        AF[Airflow backfill<br/>DAG]
        NB[Databricks<br/>notebooks]
    end

    PG --> DBZ
    MY --> DBZ
    DBZ --> KC --> KFK
    KFK --> BZ --> S3
    BZ --> ADLS
    BZ --> SV --> GD
    SV --> S3
    GD --> S3
    SV --> CDF
    GD --> CDF
    POL -. governs .-> BZ
    POL -. governs .-> SV
    POL -. governs .-> GD
    GLUE -. external .-> SV
    HMS -. fallback .-> SV
    DAB --> SV
    DLT --> SV
    NB --> SV
    AF -. backfill .-> BZ
    TT -. query .-> GD

Loading

Full sequence diagrams in docs/architecture.md.

---

Tech highlights

Capability	Tools
Streaming	PySpark Structured Streaming (exactly-once, watermarks, stream-stream joins, stateful aggregations)
Storage format	Delta Lake 3.1 — CDF, MERGE, Z-ORDER, liquid clustering, time travel, generated columns
CDC	Debezium 2.6 + Kafka Connect
Catalogs	Apache Polaris (OSS Unity Catalog), AWS Glue (LocalStack), Hive Metastore
Databricks idioms	Databricks Asset Bundles (databricks.yml), DLT-style declarative pipelines, notebook-style .py
Multi-cloud	S3 (LocalStack) + ADLS Gen2 (Azurite) write paths
Maintenance	OPTIMIZE, VACUUM, Z-ORDER, liquid clustering rewrites
Languages	Python, Scala (UDFs + MERGE predicates), SQL, Shell
IaC	Terraform (AWS/Azure local + Databricks provider examples)
Orchestration	Airflow backfill DAG + Databricks Workflows (via Asset Bundles)
CI/CD	GitHub Actions (lint + unit tests + YAML validate + chispa spark transform tests), Jenkinsfile, GitLab CI

---

Quickstart

make install
make lint
make test              # pytest unit + chispa Spark tests
make compose-up        # redpanda + debezium + delta-spark + polaris + azurite + localstack
make seed              # load sample CDC events into Kafka
make run-streaming     # start the CDC streaming consumer
make notebooks-run     # execute notebooks in sequence
make asset-bundle-validate

---

Project layout

databricks-streaming-cdc-governance-lakehouse/
├── .github/workflows/ci.yml
├── databricks.yml              # Databricks Asset Bundle root
├── bundles/pipelines/          # DLT-style pipeline YAML
├── bundles/jobs/               # Workflows / jobs YAML
├── dlt_pipelines/              # DLT-style declarative SQL
├── notebooks/                  # Databricks-style .py notebooks (% magic)
├── src/
│   ├── pyspark/                # streaming / bronze / silver / gold / maintenance / helpers
│   ├── scala/                  # SBT Scala UDFs module
│   ├── debezium/               # connector JSON
│   └── polaris/                # catalog definitions
├── dags/                       # Airflow backfill DAG
├── include/                    # shared constants / helpers
├── infra/terraform/            # Terraform (aws + azurerm + databricks providers)
├── tests/unit/                 # chispa-based + pure-python
├── configs/                    # governance policies
└── scripts/                    # seed + bootstrap + rpk helpers

---

Zero-cost posture

• Databricks — open-source delta-spark + pyspark standin for runtime; Asset Bundle YAML is real, just not deployed to a paid workspace in CI.
• AWS Glue / Unity Catalog — Polaris + Glue-on-LocalStack + Hive Metastore parity.
• Azure ADLS — Azurite emulator.

License

MIT.