Building MyZubster in Public: The Part Nobody Sees
When people see a technology project online, they usually see the final layer:
the website.
The application.
The screenshots.
The AI.
The repository.
But there is another side of development that is much less glamorous.
Servers filling their disks.
Nodes falling behind.
CI pipelines failing.
Dependencies breaking.
Databases growing.
Services restarting.
Unexpected network problems.
Over the last few days, MyZubster reminded us of something important:
Infrastructure is part of the product.
A real example: our Monero node stopped synchronizing
MyZubster experiments with blockchain infrastructure as one of several technological components of the ecosystem.
We operate a Monero node, and recently its synchronization stopped progressing.
The daemon was alive.
CPU was working.
Memory wasn't exhausted.
The RPC reported:
busy_syncing: true
synchronized: false
But the blockchain height wasn't moving.
At first, this could look like a networking problem, a peer problem or even database corruption.
The actual problem was much simpler.
And much more dangerous.
Filesystem: 100%
Available space: 0
The server had run out of disk space.
Finding the bottleneck
The Monero LMDB database had grown to approximately:
/var/lib/monero/lmdb/data.mdb
≈ 50 GB
The server itself had a root filesystem of roughly 79 GB.
But Monero wasn't alone.
The machine also contained:
Operating system
Docker
MyZubster services
Development repositories
Build environments
Backups
Package caches
Blockchain data
Eventually:
Infrastructure demand > Available storage
And synchronization stopped.
The wrong solution would have been destructive
When a blockchain database stops progressing, it's tempting to immediately rebuild it.
Delete the database.
Restart synchronization.
Try database recovery.
But that would have thrown away millions of already synchronized blocks.
Instead, we investigated first.
Observe
↓
Measure
↓
Identify bottleneck
↓
Recover capacity
↓
Verify
This is exactly the philosophy we're trying to bring into the broader MyZubster architecture.
Evidence before assumptions.
We started cleaning the server
The first step was understanding where the storage was actually going.
We discovered several categories:
Monero LMDB ~50 GB
/root ~15 GB
/usr ~9 GB
Docker ~2 GB
Inside the development environment there were also multiple local repository clones.
Some could safely be recreated from GitHub.
Others could not.
And this created another interesting engineering problem.
Never assume "it's on GitHub"
A local Git repository isn't automatically disposable just because it has an origin.
Before deleting repository copies, we checked:
git status --short --untracked-files=all
git log --oneline --branches --not --remotes
Why?
Because a repository can contain:
uncommitted changes
untracked files
local-only commits
ignored configuration
temporary development artifacts
Deleting a clone without checking those states can destroy work that never reached the remote repository.
We actually encountered exactly this situation.
One repository contained an untracked file.
Another MyZubster checkout contained modified build artifacts and additional local files.
So those weren't treated as disposable.
Git became part of infrastructure recovery
Other repository clones were clean.
Their source existed remotely.
No untracked files.
No local-only commits.
Those copies could therefore be removed from the production server and cloned again later if needed.
The process became:
Local Repository
↓
Check remote
↓
Check working tree
↓
Check untracked files
↓
Check local-only commits
↓
├── Unique data → KEEP
│
└── Fully reproducible → REMOVE
This recovered several gigabytes.
And then something happened
The Monero node started moving again.
Before:
height: 2705556
After recovering disk capacity:
2708836
2708936
2710796
2711216
...
The node wasn't corrupted.
It wasn't dead.
It was suffocating.
Give the infrastructure resources again, and it continued working.
This is why observability matters
A process being "running" doesn't mean a service is healthy.
Consider:
systemctl/process status
↓
RUNNING
That tells us very little.
For a blockchain node, actual health might require:
Process alive
+
RPC responsive
+
Peers available
+
Blockchain height increasing
+
Disk capacity available
+
Database writable
Only then can we say:
SERVICE HEALTHY
The same principle applies to MyZubster.
"Online" is not the same as "working"
As the ecosystem grows, this distinction becomes increasingly important.
An AI agent responding doesn't necessarily mean its underlying data is correct.
An API returning HTTP 200 doesn't mean the pipeline is healthy.
A sensor being connected doesn't mean its measurements are valid.
A blockchain daemon running doesn't mean it's synchronized.
A CI pipeline passing doesn't prove the real-world system works.
That's why infrastructure needs measurable health indicators.
Toward evidence-based operations
Imagine representing every MyZubster service through observable state:
SERVICE
↓
HEALTH
↓
METRICS
↓
EVIDENCE
↓
ALERT
↓
RECOVERY
For example:
{
"service": "monero-node",
"status": "syncing",
"height": 2711216,
"disk_pressure": true,
"healthy": true,
"action_required": "increase-storage"
}
Now infrastructure becomes machine-readable.
And that's where AI becomes interesting again.
AI should diagnose evidence, not invent status
An AI operations agent shouldn't simply say:
"Everything looks fine."
It should be able to inspect measurable signals.
Disk → 96%
Blockchain → advancing
RPC → responding
Memory → available
Target height → ahead
Then conclude:
Node operational.
Synchronization progressing.
Storage capacity remains a risk.
That is a much more useful form of AI.
Not artificial confidence.
Evidence-driven reasoning.
The permanent fix
Cleaning files isn't the long-term solution.
The server needs more storage.
The infrastructure therefore has to evolve as the project evolves.
Today:
79 GB server
↓
Development grows
↓
Blockchain grows
↓
Disk pressure
Tomorrow:
Expanded storage
↓
More operational margin
↓
Dedicated infrastructure
↓
Better monitoring
↓
Automated capacity alerts
Eventually, blockchain data may deserve its own dedicated volume rather than competing with application infrastructure.
That's normal.
Architecture changes when reality provides new information.
Building in public means showing this too
It's easy to publish:
"We added AI."
"We integrated blockchain."
"We launched another service."
But engineering also means publishing:
"The disk filled up."
"Synchronization stopped."
"We investigated it."
"We found the bottleneck."
"We recovered capacity."
"The node started progressing again."
Those stories may actually be more useful to developers.
Because that's what running infrastructure really looks like.
MyZubster isn't finished
And we don't want to pretend otherwise.
We're still building.
We're testing.
We're finding bottlenecks.
We're changing architecture when reality tells us that our assumptions were wrong.
That's the difference between drawing a system diagram and actually operating the system.
The diagram says:
App → API → Blockchain
Reality says:
App
↓
API
↓
Node
↓
Database
↓
Filesystem
↓
Storage capacity
↓
Operating system
↓
Network
↓
Monitoring
↓
Recovery strategy
Every layer matters.
The lesson
One of the principles emerging from MyZubster development is becoming increasingly simple:
Don't ask only whether a service is running. Ask whether you can prove that it's working.
That's true for blockchain nodes.
It's true for AI.
It's true for IoT.
It's true for environmental monitoring.
And eventually, it should be true for the entire MyZubster ecosystem.
Build → Observe → Measure → Verify → Recover → Improve.
That's what building in public actually looks like.