menu

Workflow Orchestration

●10 ●85 ●140
calendar_today • schedule1 min read

Just completed Module 2 of the Data Engineering Zoomcamp 2026. Built production-ready data pipelines using Kestra to process 26 million NYC taxi trip records.
What I accomplished:

Orchestrated ETL workflows with Kestra
Ingested data from GitHub to GCS to BigQuery
Implemented partitioned tables for query optimization
Built MERGE operations for data deduplication
Automated monthly data loads with schedule triggers
Completed all homework questions (6/6)

Key learnings:

Template rendering with Pebble syntax
Handling trigger.date for manual vs scheduled runs
GCS storage class compatibility (REGIONAL vs STANDARD)
IANA timezone format for DST handling
BigQuery partitioning strategies

Cost efficiency:
Processed 2GB of data for less than one dollar, staying well within GCP's free tier.
Check out my project on GitHub: https://github.com/Derrick-Ryan-Giggs/module-2-workflow-orchestration
Thanks to DataTalksClub for this amazing free course.

Derrick Ryan Giggs
6.7k Points • 235 Badges • ●10 ●85 ●140
Kenya
71Posts
23Comments
147Followers
6Connections
Aspiring Data Engineer | Learning Python, Java & Oracle Databases

On an exciting journey to become a data engineer.

Currently diving deep into Python programming, Java developme... Show more
✨ Build your own developer journey
Track progress. Share learning. Stay consistent.
Create your profile

1 Comment

•

Nice work, processing 26 million records for under a dollar is impressive and makes me want to try Kestra myself, how was the learning curve?

1 vote
🔥 Join developers growing publicly
Share your knowledge, build in public, and grow your developer presence with a global community.
Join CoderLegion

More Posts

Tech Ecosystem Observatory: How I Built a Cloud-Native Data Pipeline to Track Global Tech Layoffs

Derrick Ryan - Mar 30

Sibling Rivalry? How to Make Kestra Tasks Talk to Each Other

Amara Graham - Apr 23

From APIs to Warehouses: AI-Assisted Data Ingestion with dlt

Derrick Ryan - Mar 1

Batch Processing with Apache Spark

Derrick Ryan - Mar 7

I Built a Real-Time Crypto Analytics Pipeline for $0.01/Month — Here's the Full Architecture

Derrick Ryan - Apr 27
chevron_left

More From Derrick Ryan

I Built a Real-Time Crypto Analytics Pipeline for $0.01/Month — Here's the Full Architecture

Oracle GoldenGate 23ai: Powering Distributed AI with Real-Time Data Replication

Building the Sovereign Debt Observatory: An End-to-End ELT Pipeline on World Bank Debt Data for Low

Related Jobs

View all jobs →

Commenters (This Week)

4 comments
1 comment
1 comment

Contribute meaningful comments to climb the leaderboard and earn badges!