Data Engineering Zoomcamp
30.3K subscribers
7 photos
166 links
Download Telegram
We start Module 3 on Data Warehousing and BigQuery

Reminder: the previous homework deadline is in less than 24 hours.

This module covers:

🔸 Data warehouse fundamentals and BigQuery basics
🔸 Query performance and internals (partitioning, clustering, best practices)
🔸 Machine learning in BigQuery, including SQL for ML
🔸 Deploying models from BigQuery using Docker

Learn here: https://github.com/DataTalksClub/data-engineering-zoomcamp/tree/main/03-data-warehouse

We're finalizing Homework 3 and will notify you when it's ready.
❤23😁2👍1👎1
Reminder: today at 17:00 CET (in 10 hours from now) we have a live stream

We will share the link 5-10 minutes before the start

See you soon!
❤21🔥3
We start Module 4 on Analytics Engineering (dbt)

Reminder: the previous homework deadline is in less than 24 hours.

This module focuses on transforming warehouse data into analytical models using dbt.

You’ll build a dbt project on NYC yellow and green taxi data (2019-2020) and cover analytics engineering fundamentals, data modeling, and dbt in practice.

Setup options:

• Local: DuckDB + dbt Core (free, no prerequisites)
• Cloud: BigQuery + dbt Cloud (requires Module 3)

👉🏼 Materials

Homework 4:
• Homework assignment
• Submit here

Deadlone for this homework: 17 February 12 AM CET

Big thanks to Juan Manuel Perafan for updating this module, originally recorded by Victoria Perez Mola!
❤19👍5
We received a few reported cases of people misusing the "Learning in Public" scoring. These course participants were submitting links that were not social media posts sharing their progress, but rather random links they found on the internet while researching topics.

This is against the rules. As a consequence, these two participants can no longer earn points for "Learning in Public." I still want to encourage them to continue the practice because it is for their own benefit, but we are against misusing the system just for the sake of getting points. That is why the "Learning in Public" rewards have been disabled for them.

If you come across anyone on the leaderboard who is misusing this system by posting links that are not their personal posts about what they are learning in the course, please report it in the course channel.
👍56❤5
This week, we're starting Module 5: Data Platforms.

Reminder: the previous homework deadline is in less than 24 hours

In this module, you'll learn how modern data platforms manage the full data lifecycle, from ingestion to transformation, orchestration, data quality, and metadata.

We'll use Bruin as a practical example of a unified data platform. You'll install it, create a project, and build an end-to-end pipeline using NYC taxi data with a three-layer architecture: ingestion, staging, and reporting.

You'll also set up connections, configure pipelines in YAML, run Python and SQL assets, and see how orchestration and data quality fit into a single workflow.

Homework:

• Homework 5
• Deadline: 1st March, 12:00 AM CET

🎁 Bonus: Bruin is offering a Claude Pro subscription to participants who complete their final project using Bruin.
👍19❤10🔥7🙈1
Reminder: tomorrow we’re running a hands-on workshop on AI-Assisted Data Ingestion with dlt as part of the Data Engineering Zoomcamp.

Date: Tuesday, February 17
Time: 4:30 PM CET
Place: Live on YouTube

Join Aashish Nair, who’ll lead the session, to build a reliable ingestion pipeline into a data warehouse (for example, Snowflake) using dlt from dltHub.

We’ll go through:

• Extracting data from APIs, files, and databases
• Normalizing it into consistent schemas
• Writing it to a warehouse
• Using LLMs to accelerate pipeline development
• Validating data and schema changes with the dlt dashboard and dlt MCP

By the end, you’ll understand how to design maintainable ingestion pipelines and use AI and validation tools to build them faster.

Register to get the join link:
https://luma.com/hzis1yzp
👍11❤9🔥7👎1
On Monday at 16:00 CET we will have a live session

Add it to your calendar! We will share the link to the stream 5-10 minutes before the start

If you want a reminder, I created a luma event: https://luma.com/m258fy6w
❤11
This week, we continue working on Module 5: Data Platforms.

In this module, you'll learn how modern data platforms handle the data lifecycle, including ingestion, transformation, orchestration, and data quality.

Using Bruin as a practical example, you'll install the platform, create a project, and build an end-to-end pipeline with NYC taxi data. The pipeline will include three layers: ingestion, staging, and reporting.

You can submit Homework 5 before March 1.
👍7❤3🔥2
We also recently had a workshop with dlt on AI-assisted data ingestion.

Watch the recording and check out the code if you missed it.

Practice what you learned in the homework assignment, and submit it here.
👍5❤3🔥3
Today at 16:00 CET (in 90 minutes) we have office hours

You can use this link if you want to ask any question in advance:

https://app.sli.do/event/r22cR71X67cQ7gHacVu7LG

See you soon!
❤4
This is the link I promised to share:

https://getbruin.com/zoomcamp-project/

You get a chance to win Claude Code subscription if you use Bruin in your project
👍12❤3
The course management platform is currently down

I'm working on restoring it
❤19🙏3🔥1
Update: I've spent one hour on a call with AWS and they said they needed more time to restore the database.

Hopefully tomorrow it'll get sorted out
❤21👍10😨9
The course management platform is back!

If you submitted any homework yesterday, the data may be lost. Please check your homework submissions.

Apologies for the inconvenience!

I'm working on the incident report and will publish it soon. TLDR: don't let your AI agents do terraform apply -auto-approve
❤25🤣19🔥3