While building my ETL pipeline project, I ran into a common beginner problem:
Running Apache Airflow locally on Windows with Docker was painful.
Problems included:
- Low disk space
- Docker setup issues
- Linux compatibility problems
- Environment debugging overhead
With only ~17GB free on my laptop, running multiple Airflow containers locally became difficult.
So I moved the entire setup to GitHub Codespaces.
What Codespaces Provided
Out of the box:
- Ubuntu Linux environment
- Docker pre-installed
- VS Code in browser
- Auto-cloned GitHub repo
- Port forwarding for Airflow UI
Workflow
docker-compose up
Then:
- Open Airflow UI
- Trigger ETL DAG
- Verify successful execution ✔️
Airflow was running in ~90 seconds.
Important Security Lesson
I accidentally committed my GCP service account key once.
GitHub Secret Scanning blocked the push automatically.
Immediately added:
*.json
.env
Never commit cloud credentials.
Why This Setup Helps Beginners
Codespaces removes a huge amount of local environment friction and lets you focus more on:
- Airflow orchestration
- ETL pipelines
- Docker workflows
- Cloud integrations
If you'd like:
- Beginner-friendly walkthrough → check Medium
- Engineering-focused breakdown → check Hashnode
Medium: Link
Hashnode: Link
Project Repo:ETL Pipeline
DataEngineering #Docker #ApacheAirflow #GitHubCodespaces
