🚀 Flink PostgreSQL to StarRocks CDC Pipeline

This project demonstrates a complete Flink CDC pipeline using:

• 📦 Debezium for PostgreSQL Change Data Capture
• 🔄 Apache Flink to process and transform changes in real-time
• 📊 StarRocks as the analytical sink

🧱 Architecture Overview

To render the diagram below, use Kroki.io or a Markdown PlantUML renderer.

📜 PlantUML Source

@startuml
!theme spacelab

actor Developer as Dev
database "PostgreSQL\n(Test Data)" as PG
component "Debezium\n(Postgres CDC)" as Debezium
component "Flink SQL\nJob (DDL + Logic)" as FlinkSQL
component "Flink\nJobManager" as JM
component "Flink\nTaskManager" as TM
database "StarRocks\n(Frontend + Backend)" as StarRocks

Dev --> JM : Deploys SQL Jobs (DDL)

PG --> Debezium : Emits WAL Changes
Debezium --> FlinkSQL : CDC Stream (Kafka-style)

FlinkSQL --> JM : Submit Flink Plan
JM --> TM : Dispatch Executable Plan
TM --> StarRocks : Insert Streamed Data

note right of PG
PostgreSQL seeded with sample data\nRuns on port 5433
end note

note right of Debezium
Debezium watches WAL logs\nand emits change streams
end note

note right of FlinkSQL
Defines CDC pipeline logic\nand table DDL mappings
end note

note right of JM
JobManager builds optimized\nexecution graphs from SQL
end note

note right of TM
TaskManager runs parallel\nexecution subtasks
end note

note right of StarRocks
StarRocks FE+BE ingest real-time\ndata from Flink for analytics
end note
@enduml

🛠 Components

Component	Description
PostgreSQL	Source DB generating WAL logs for Debezium
Debezium	Captures WAL logs and produces CDC events
Flink SQL	Defines pipeline transformations (Flink DDL)
JobManager	Translates jobs into physical pipelines
TaskManager	Executes jobs and pushes to StarRocks
StarRocks	Real-time analytical store

---

▶️ How to Run This Pipeline

1. Start Flink & StarRocks (external PostgreSQL is assumed to be already running):

./setup_pipeline.sh

2. Open Flink SQL Client:

./open_sql_client.sh

3. Load the CDC job:

Flink SQL> source sql/01_init.sql;

This will:

• Create a Flink CDC table from PostgreSQL
• Create a StarRocks sink table
• Continuously mirror changes
• Also print changes via a debug sink

---

🌐 Connect to External PostgreSQL

This project now connects to an external PostgreSQL database (e.g. a production-like supervisor-db-v1) and uses Flink CDC to stream all changes from that DB into StarRocks.

🔑 Connection Properties:

dbname = "postgres"
user = "postgres"
password = "postgres"
host = "localhost"
port = 5432

Make sure the external PostgreSQL is reachable and has logical replication enabled with wal_level = logical.

---

🧪 Local Development

./setup_pipeline.sh

This will:

• Stop & clean existing containers
• Download required JARs (Debezium + StarRocks + Runtime)
• Start PostgreSQL, Flink, and StarRocks containers
• Deploy SQL CDC pipeline