AI Data Engineering

Learning objectives

• Can write typed, tested, packaged Python fit for a pipeline, not a notebook.
• Can use window functions, CTEs, and the query-engine mental model to reason about SQL cost.
• Can prompt an AI copilot to write this code, and review what it writes.

Key concepts

• Typing, dataclasses and pydantic, iterators and generators, packaging with Poetry or uv.
• Advanced SQL: window functions, CTEs, and how a query engine plans work.
• The enterprise lens and the AI-copilot lens, introduced on the smallest possible surface.

Hands-on (live-lab)

Stand up a tooled, typed starter package for the commerce data platform: reads, transforms, and writes, with the quality gate wired in.

Deliverable

a tooled, typed starter package with a green quality gate.

Learning objectives

• Can work a clean Git flow, containerise a job with Docker, and reason about a cloud account.
• Can express infrastructure as code from first principles.
• Can explain why reproducible environments matter for a platform.

Key concepts

• Git workflow, Docker, Linux and the CLI, a cloud account.
• Infrastructure as code first principles with Terraform.
• Reproducibility as a platform property, not an afterthought.

Hands-on (live-lab)

Containerise the starter platform and define its cloud footprint as code.

Deliverable

a containerised platform with its infrastructure declared as code.

Learning objectives

• Can reason about partitioning, replication, and consistency, and the trade-offs in practice.
• Can choose a file format and storage layout for a workload, and say why.
• Can name the enterprise-quality lens: idempotency, reproducibility, testing, observability, cost.

Key concepts

• Distributed-systems fundamentals for DE: partitioning, replication, the CAP trade-offs.
• Rows versus columns, Parquet, Avro, ORC, compression, object storage (S3, GCS, ADLS).
• The quality lens applied to a pipeline, made concrete.

Hands-on (live-lab)

Lay out the platform's storage in a columnar format on object storage, partitioned deliberately.

Deliverable

M0 capstone: a small, fully-tooled, tested hello data platform that reads, transforms, and writes.

Learning objectives

• Can ingest from files, APIs, and databases, with incremental and idempotent loads.
• Can choose an EL tool and reason about when to build versus buy.
• Can make a re-run of ingestion safe by design.

Key concepts

• Batch extract, API and database sources, incremental loads, change tracking.
• EL tooling and the build-versus-buy call.
• Idempotent, re-runnable ingestion as the default.

Hands-on (live-lab)

Build an incremental, idempotent ingestion layer for the commerce sources.

Deliverable

a re-runnable ingestion layer landing raw commerce data.

Learning objectives

• Can explain data lake versus warehouse versus lakehouse and when each fits.
• Can use a table format's ACID, time travel, and schema evolution correctly.
• Can lay out a lakehouse on object storage.

Key concepts

• The lakehouse pattern and object-storage layout.
• Table formats in depth: Delta, Iceberg, Hudi (ACID, time travel, schema evolution).
• Reference tool: Delta; the field: Iceberg and Hudi named and compared.

Hands-on (live-lab)

Turn the raw landing zone into a lakehouse table with ACID, time travel, and enforced schema.

Deliverable

a lakehouse table with time travel and schema enforcement.

Learning objectives

• Can explain the driver, executors, partitions, and lazy execution.
• Can read a query plan and the Spark UI to find shuffles and skew.
• Can tune a slow join by fixing its plan, not by adding compute.

Key concepts

• The execution model: partitions, transformations versus actions, jobs, stages, tasks.
• Shuffles, skew, broadcast joins, caching, adaptive query execution.
• Cost as a first-class concern.

Hands-on (live-lab)

Build the platform's transformation layer in PySpark and cut a slow join by fixing its plan.

Deliverable

a tuned Spark transformation stage with a measured speedup.

Learning objectives

• Can model data with dbt: models, tests, docs, and lineage.
• Can design a medallion (bronze, silver, gold) and dimensional model.
• Can express business logic as tested, documented transformations.

Key concepts

• dbt models, tests, docs, and lineage.
• Data modeling: dimensional, Data Vault, and the medallion architecture.
• Reference tool: dbt; modeling patterns named.

Hands-on (live-lab)

Model the commerce data into a medallion with dbt, with tests and generated lineage.

Deliverable

a tested, documented medallion model with lineage.

Learning objectives

• Can orchestrate a multi-step pipeline with dependencies, schedules, backfills, and retries.
• Can enforce data quality and contracts as a gate between layers.
• Can make the whole pipeline safe to re-run.

Key concepts

• Orchestration with Airflow (reference), Dagster and Prefect (the field): DAGs, scheduling, backfills, idempotency.
• Data quality and contracts: expectations and dbt tests, schema enforcement.
• Idempotency and backfills done safely.

Hands-on (live-lab)

Orchestrate the batch platform end to end with quality gates and a safe backfill.

Deliverable

M1 capstone: a complete, tested, orchestrated batch lakehouse with lineage and quality gates.

Learning objectives

• Can reason about Kafka topics, partitions, consumer groups, and delivery semantics.
• Can stream an operational database into the platform with change data capture.
• Can choose delivery guarantees for a use case.

Key concepts

• Event streaming with Kafka: topics, partitions, consumer groups, delivery semantics.
• Change data capture with Debezium: streaming the operational database in.
• At-least-once versus exactly-once, and what each costs.

Hands-on (live-lab)

Stream the commerce clickstream through Kafka and the orders database in via CDC.

Deliverable

a streaming ingestion feeding the platform from events and CDC.

Learning objectives

• Can process streams with windows, watermarks, and state, exactly once.
• Can keep a lakehouse table current from a stream.
• Can reason about Kappa versus Lambda.

Key concepts

• Stream processing: Spark Structured Streaming and Flink (windowing, watermarks, state, exactly-once).
• Streaming and the lakehouse in motion; Kappa versus Lambda.
• Reference tool: Structured Streaming; the field: Flink named.

Hands-on (live-lab)

Build a windowed clickstream aggregation with a watermark that updates a gold table continuously.

Deliverable

a real-time aggregation keeping a gold table current.

Learning objectives

• Can define SLAs, SLOs, and freshness, and respond to a data incident.
• Can instrument observability and lineage across the platform.
• Can govern access, PII, and cost at scale.

Key concepts

• Data reliability engineering: SLAs and SLOs, freshness, incident response.
• Observability and lineage: OpenLineage, DataHub, OpenMetadata; real-time stores (ClickHouse, Druid, Pinot).
• Governance, security, and cost at scale: catalogs (Unity Catalog), access control, PII.

Hands-on (live-lab)

Add observability, lineage, and a governance layer to the real-time platform.

Deliverable

M2 capstone: a real-time platform (clickstream plus inventory CDC) with observability and governance.

Learning objectives

• Can parse and chunk documents into a pipeline-ready form.
• Can generate, batch, and store embeddings at scale.
• Can reason about chunking strategies and their retrieval impact.

Key concepts

• Unstructured and semi-structured pipelines: parsing documents, chunking strategies.
• Embeddings pipelines: generating, batching, and storing embeddings at scale.
• The data side of retrieval quality.

Hands-on (live-lab)

Build an embeddings pipeline over the product catalog, chunked and batched.

Deliverable

an embeddings pipeline producing a searchable catalog index.

Learning objectives

• Can choose and operate a vector database and index for a workload.
• Can build RAG data infrastructure: ingestion, indexing, freshness.
• Can evaluate retrieval quality on the data side.

Key concepts

• Vector databases in depth: pgvector, Qdrant, Milvus, Weaviate (indexing, ANN, trade-offs).
• RAG data infrastructure: ingestion, indexing, freshness, retrieval-quality evaluation.
• Freshness and re-indexing as a pipeline concern.

Hands-on (live-lab)

Index the catalog embeddings in a vector store and evaluate retrieval quality.

Deliverable

a RAG-ready catalog index with a retrieval-quality report.

Learning objectives

• Can build a feature store with online and offline stores and point-in-time correctness.
• Can reason about data for training versus inference, and data versioning.
• Can build and review AI-written pipeline code with judgment.

Key concepts

• Feature engineering platforms and feature stores: Feast and Tecton, online versus offline, point-in-time correctness.
• Data for training versus inference; the MLOps data plane; data versioning (lakeFS, DVC); drift and monitoring on the data side.
• AI-assisted data engineering: building and hardening pipelines with LLM copilots, and reviewing AI-written code.

Hands-on (live-lab)

Build a recommendation feature store with point-in-time-correct features on the commerce platform.

Deliverable

M3 capstone: an end-to-end AI data platform, a RAG-ready catalog index plus a recommendation feature store, on M1 and M2.

What it is

The same layer you build in the live-lab, rebuilt in a high-volume sensor domain, where throughput and cost dominate.

Deliverable

a parallel data platform for IoT telemetry, layer by layer, with a worked solution.

What it is

The same layer, rebuilt in a domain where every record must be right, so idempotency, contracts, and reconciliation are non-negotiable.

Deliverable

a parallel data platform for financial transactions, with a worked solution.

What it is

Every module ends in a complete, valuable artifact you defend. Stop after M0 (a tooled hello platform), M1 (a batch lakehouse), M2 (a real-time platform), or M3 (an end-to-end AI data platform): each is a real competency on its own.

Deliverable

a defended capstone at each module boundary, culminating in an AI-ready data platform.