knowledge-dataplane

An open, modular data platform control plane for Teradata.

knowledge-dataplane closes the gap between "assembled integrations" and a unified, product-native data platform experience. Domain teams write YAML and SQL. Everything else — governance, lineage, catalog registration, CI/CD — is generated and enforced by the platform.

The 30-second pitch

# A domain team onboards a new data product
copier copy gh:Teradata/knowledge-dataplane/scaffold ./flight-ops-dp

# Answer 6 questions. Get a complete, governed data product repo.
# Push to GitHub. CI validates, deploys, and registers the catalog entry.
# Done in under 30 minutes.

Architecture

knowledge-dataplane is built as a modular framework with stable service interfaces and swappable backend adapters. You can run OpenMetadata for a lightweight MVP and swap to DataHub for production — with a single config change.

┌─────────────────────────────────────────────────┐
│              Platform consumers                  │
│   Control plane UI · Backstage · GitHub Actions  │
└────────────────┬────────────────────────────────┘
                 │  stable internal APIs
┌────────────────▼────────────────────────────────┐
│           Service interfaces (core/)             │
│  CatalogService · LineageService                 │
│  OrchestrationService · ContractService          │
└────────────────┬────────────────────────────────┘
                 │  adapter pattern
┌────────────────▼────────────────────────────────┐
│              Adapters                            │
│  openmetadata/ · datahub/ · marquez/ · dagster/  │
└────────────────┬────────────────────────────────┘
                 │
┌────────────────▼────────────────────────────────┐
│           Teradata Vantage                       │
│   On-premises · VantageCloud Lake                │
└─────────────────────────────────────────────────┘

Quick start (demo)

# 1. Clone the repo
git clone https://github.com/Teradata/knowledge-dataplane
cd knowledge-dataplane

# 2. Start the MVP stack (OpenMetadata + Marquez + Dagster + Postgres)
docker compose -f docker/docker-compose.mvp.yml up -d

# 3. Run the flight-ops demo
cd examples/flight-ops-demo
./run-demo.sh

The demo runs the full 30-minute story — scaffold → dbt models → CI pipeline → catalog registration — against a local stack.

Repository structure

knowledge-dataplane/
├── core/              # Python ABCs — the stable service interfaces
├── adapters/          # One folder per backend implementation
│   ├── openmetadata/  # MVP catalog + lineage backend
│   ├── datahub/       # Production catalog + lineage backend
│   ├── marquez/       # Lightweight lineage backend (MVP)
│   └── dagster/       # Orchestration adapter
├── scaffold/          # Copier template — generates data product repos
├── platform/          # Thin React control plane UI
├── docker/            # Compose files for MVP and production stacks
├── examples/          # Runnable demo scenarios
│   └── flight-ops-demo/
└── docs/              # Architecture decisions, module specs, ADRs

Swapping backends

Backend selection is controlled by kdp.config.yaml in your deployment:

catalog:
  adapter: openmetadata          # swap to: datahub
  base_url: http://localhost:8585

lineage:
  adapter: marquez               # swap to: datahub (consolidates with catalog)
  base_url: http://localhost:5000

orchestration:
  adapter: dagster
  base_url: http://localhost:3000

contracts:
  adapter: datacontract_cli

No application code changes. Only the adapter resolves the interface differently.

Contributing

knowledge-dataplane is an open-source Teradata field engineering project. Contributions welcome — especially new adapters, scaffold template improvements, and example data products.

See docs/CONTRIBUTING.md for the contribution guide.

License

Apache 2.0 — see LICENSE.