Data Engineering Foundations (DE-101)
A dashboard shows revenue that finance swears is wrong. A nightly job that finished in four minutes last quarter now runs for four hours. A "one-time" data fix quietly becomes a script three people are afraid to touch. Behind each of these is a pipeline that someone designed — or failed to design — on purpose. This course teaches you to be the person who designs it well.
DE-101 is the mental-model course for the School of Data Engineering. It uses almost no heavy tooling. Instead, it gives you the vocabulary and the judgment that every later course lands on: what a pipeline is, how data moves through its lifecycle, and how to reason about the trade-offs — latency, cost, and correctness — that decide how you build. When you finish, you can sketch a pipeline, name its failure modes, and defend a batch-versus-streaming decision without hand-waving.
- Explain what a data engineer does and why the pipeline is treated as a product - Describe the data engineering lifecycle from generation through serving - Distinguish batch from streaming and choose between them using latency, cost, and correctness - Sketch a simple end-to-end pipeline and name the stages data passes through - Identify common pipeline failure modes such as late data, duplicates, and schema drift
Who this course is for
This is the entry point to the whole school. It is written for four kinds of learner:
- The fresh graduate with coursework Python and no production experience, building toward a first junior role.
- The backend engineer comfortable with Python and Git, new to data-specific concerns, on the way to DE-201 and beyond.
- The analyst crossing over who writes SQL daily and keeps hitting the limits of BI tooling.
- The cohort learner who wants the mental models under the migrated PySpark, modelling, and AWS material they are already working through.
Prerequisites
None. DE-101 gates the School of Data Engineering, so it assumes only that you can read a short Python snippet and a line of SQL without panic. You do not need a cloud account, a cluster, or a paid tool. Everything runs either in your browser through the Tapasya playgrounds or in a few lines you can reproduce locally.
Modules
DE-101 is roughly twelve hours of effort across seven modules. This first slice of the course delivers the three foundational lessons and your first hands-on lab.
| # | Module | What you leave with |
|---|---|---|
| 1 | What data engineers actually do | The pipeline-as-product mental model |
| 2 | Source systems | How data is born: databases, events, files, APIs |
| 3 | Storage fundamentals | Row vs. column, and CSV/JSON/Parquet trade-offs |
| 4 | Batch vs. streaming | A decision framework, not a technology tour |
| 5 | The modern data stack, mapped | Where each tool in this school fits |
| 6 | Pipeline failure modes | Late data, duplicates, schema drift, backfills |
| 7 | Working like an engineer | Git, environments, and reproducibility basics |
The three lessons and the lab below cover the core of Modules 1, 2, and 4 and give you the pipeline-modelling skill the rest of the course builds on.
Outcomes
By the end of DE-101 you can:
- Describe a data engineer's job as owning a pipeline end to end, not writing one-off scripts.
- Walk any dataset through the five stages of the lifecycle and say what each stage is responsible for.
- Choose batch or streaming for a real scenario and justify the choice with a latency budget and a cost argument.
- Draw a pipeline diagram that a teammate could implement.
Where this leads: the capstone
Everything in this school converges on DE-350: Production-Grade Pipeline, where you build and operate a complete pipeline against a realistic source scenario with seeded data defects and late-arriving records. The pipeline sketch you produce in this course's lab is the same artifact — a source-to- serving model — that the capstone asks you to build for real, with modelling (DE-201), PySpark transformations (DE-202), orchestration (DE-204), and a data quality suite (DE-301) layered on top. Start here, and the capstone lands on prepared ground.
Do the lessons in order. Each one introduces vocabulary the next assumes, and the lab at the end pulls all three together into a single diagram you will reuse throughout the school.