As Data Engineers, we often live in the cloud. Our daily lives are governed by S3 buckets, Glue jobs, and Kafka clusters. But there is a silent pressure that every senior professional feels: the cost of experimentation.
How many times have you hesitated to spin up a resource just to validate a simple schema or a transformation logic? We've all been there—balancing the need for agility with the responsibility of cost-awareness.
I’ve always believed that autonomy is a superpower. True technical mastery comes when you can prototype "blindly," without an internet connection, directly on your machine, and with zero impact on your company's AWS bill.
Why I moved away from heavy VMs
For years, the choice was either a heavy Virtual Machine (VM) that drained my laptop’s RAM or a Docker container that, while fast, sometimes lacked the persistence and system-level isolation I needed for a long-term practice environment.
When I started working with Ubuntu 24.04, I found the perfect middle ground: LXD. It offers the "system feel" of a VM with the "instant speed" of a container. It became my go-to for building local SQL sandboxes where I can fail fast, learn deep, and spend zero.
The Blueprint: Setting up your Local Engine
The goal is simple: a functional database server running in minutes.
- The Foundation
We start by initializing the hypervisor to manage our internal networking.
sudo snap install lxd
sudo lxd init --auto
- Launching the Instance
We choose a lightweight Ubuntu image as our dedicated server.
lxc launch ubuntu:24.04 sql-practice
- The "Hardware" Setup
We enter the container and transform that empty shell into a robust PostgreSQL engine.
lxc exec sql-practice -- bash
apt update && apt install -y postgresql postgresql-contrib
From Infrastructure to Impact
Setting up the "pipes" is only half the battle. As engineers, we care about the data. In my experience building event-driven architectures and streaming pipelines with Kafka, I’ve learned that a local sink is the best place to test idempotency and schema evolution before touching a production environment.
Using the COPY command for massive local ingestion or pushing CSV files directly into the container (lxc file push) are the types of "unsexy" but highly efficient workflows that define a professional.
Closing thoughts: Design first
My philosophy is simple: Systems must be resilient and observable from day one. That includes our local dev environments. Complexity is a cost, and simplicity scales better.
By mastering these local sandboxes, we aren't just saving a few dollars on an AWS bill; we are reclaiming our creative freedom as builders.
About the Author
I am Rocío Baigorria, a Senior Data Engineer and co-leader of AWS Girls Argentina. I specialize in building "institutional-grade" data platforms where reliability, cost-efficiency, and business impact are core design principles.
Connect with me on LinkedIn: https://www.linkedin.com/in/rociobaigorria/
Explore my Resources: https://tr.ee/2t-m587551
