Spark Declarative Pipelines and Lakeflow Designer on Databricks

Learn how to build production-grade data pipelines on Databricks Free Edition using Declarative Pipelines and the Lakeflow Designer. This hands-on course covers everything from Delta Lake fundamentals and Unity Catalog governance to building a complete medallion architecture (Bronze, Silver, Gold) pipeline — all with SQL and no infrastructure setup required.

You'll work with a real e-commerce dataset of 500,000+ transactions, use Databricks AI tools (Assistant, Genie) to accelerate development, and extend the pipeline with streaming data from AWS Kinesis. Ideal for data engineers and analysts who want a practical, end-to-end introduction to modern lakehouse pipelines on Databricks.

Ready to follow along? Sign up for Databricks Free Edition here — no credit card required.

---

Contents

1. Orientation, Setup, and Dataset Exploration
   1. Course Overview & What You Will Build
   2. Getting Started in Databricks and Exploring the Dataset
2. Delta Lake & Unity Catalog Foundations
   1. Delta Lake Essentials (ACID, Time Travel, Versioning)
   2. Unity Catalog Basics
3. Building Declarative Pipelines With Pipeline Editor
   1. Introduction to Declarative Pipelines & Medallion Architecture
   2. Building the Bronze Layer
   3. Building the Silver Layer (Cleaning & Modeling)
   4. Building the Gold Layer (Business Aggregations)
   5. Scheduling and Running the Pipeline
4. Building Declarative Pipelines with Lakeflow Designer
   1. Introduction to Lakeflow Designer
   2. Createing a Pipeline Inside Lakeflow Designer
5. Bonus: Streaming with Declarative Pipelines in Lakeflow Designer
   1. Streaming Modes in Declarative Pipelines
   2. Connecting Databricks to an AWS Message Queue
   3. Building a Simple Streaming Pipeline in Lakeflow Designer
6. Summary & Next Steps
   1. Summary & Next Steps