Building MyZubster in Public: The Part Nobody Sees

Leader ●2 ●24 ●155
calendar_today • schedule4 min read

Building MyZubster in Public: The Part Nobody Sees

When people see a technology project online, they usually see the final layer:

the website.

The application.

The screenshots.

The AI.

The repository.

But there is another side of development that is much less glamorous.

Servers filling their disks.
Nodes falling behind.
CI pipelines failing.
Dependencies breaking.
Databases growing.
Services restarting.
Unexpected network problems.

Over the last few days, MyZubster reminded us of something important:

Infrastructure is part of the product.

A real example: our Monero node stopped synchronizing

MyZubster experiments with blockchain infrastructure as one of several technological components of the ecosystem.

We operate a Monero node, and recently its synchronization stopped progressing.

The daemon was alive.

CPU was working.

Memory wasn't exhausted.

The RPC reported:

busy_syncing: true
synchronized: false

But the blockchain height wasn't moving.

At first, this could look like a networking problem, a peer problem or even database corruption.

The actual problem was much simpler.

And much more dangerous.

Filesystem: 100%
Available space: 0

The server had run out of disk space.

Finding the bottleneck

The Monero LMDB database had grown to approximately:

/var/lib/monero/lmdb/data.mdb
≈ 50 GB

The server itself had a root filesystem of roughly 79 GB.

But Monero wasn't alone.

The machine also contained:

Operating system
Docker
MyZubster services
Development repositories
Build environments
Backups
Package caches
Blockchain data

Eventually:

Infrastructure demand > Available storage

And synchronization stopped.

The wrong solution would have been destructive

When a blockchain database stops progressing, it's tempting to immediately rebuild it.

Delete the database.

Restart synchronization.

Try database recovery.

But that would have thrown away millions of already synchronized blocks.

Instead, we investigated first.

Observe
↓
Measure
↓
Identify bottleneck
↓
Recover capacity
↓
Verify

This is exactly the philosophy we're trying to bring into the broader MyZubster architecture.

Evidence before assumptions.

We started cleaning the server

The first step was understanding where the storage was actually going.

We discovered several categories:

Monero LMDB ~50 GB
/root ~15 GB
/usr ~9 GB
Docker ~2 GB

Inside the development environment there were also multiple local repository clones.

Some could safely be recreated from GitHub.

Others could not.

And this created another interesting engineering problem.

Never assume "it's on GitHub"

A local Git repository isn't automatically disposable just because it has an origin.

Before deleting repository copies, we checked:

git status --short --untracked-files=all
git log --oneline --branches --not --remotes

Why?

Because a repository can contain:

uncommitted changes
untracked files
local-only commits
ignored configuration
temporary development artifacts

Deleting a clone without checking those states can destroy work that never reached the remote repository.

We actually encountered exactly this situation.

One repository contained an untracked file.

Another MyZubster checkout contained modified build artifacts and additional local files.

So those weren't treated as disposable.

Git became part of infrastructure recovery

Other repository clones were clean.

Their source existed remotely.

No untracked files.

No local-only commits.

Those copies could therefore be removed from the production server and cloned again later if needed.

The process became:

Local Repository

   ↓

Check remote

   ↓

Check working tree

   ↓

Check untracked files

   ↓

Check local-only commits

   ↓
    ├── Unique data → KEEP
    │
    └── Fully reproducible → REMOVE

This recovered several gigabytes.

And then something happened

The Monero node started moving again.

Before:

height: 2705556

After recovering disk capacity:

2708836
2708936
2710796
2711216
...

The node wasn't corrupted.

It wasn't dead.

It was suffocating.

Give the infrastructure resources again, and it continued working.

This is why observability matters

A process being "running" doesn't mean a service is healthy.

Consider:

systemctl/process status

    ↓
  RUNNING

That tells us very little.

For a blockchain node, actual health might require:

Process alive
+
RPC responsive
+
Peers available
+
Blockchain height increasing
+
Disk capacity available
+
Database writable

Only then can we say:

SERVICE HEALTHY

The same principle applies to MyZubster.

"Online" is not the same as "working"

As the ecosystem grows, this distinction becomes increasingly important.

An AI agent responding doesn't necessarily mean its underlying data is correct.

An API returning HTTP 200 doesn't mean the pipeline is healthy.

A sensor being connected doesn't mean its measurements are valid.

A blockchain daemon running doesn't mean it's synchronized.

A CI pipeline passing doesn't prove the real-world system works.

That's why infrastructure needs measurable health indicators.

Toward evidence-based operations

Imagine representing every MyZubster service through observable state:

SERVICE
↓
HEALTH
↓
METRICS
↓
EVIDENCE
↓
ALERT
↓
RECOVERY

For example:

{
"service": "monero-node",
"status": "syncing",
"height": 2711216,
"disk_pressure": true,
"healthy": true,
"action_required": "increase-storage"
}

Now infrastructure becomes machine-readable.

And that's where AI becomes interesting again.

AI should diagnose evidence, not invent status

An AI operations agent shouldn't simply say:

"Everything looks fine."

It should be able to inspect measurable signals.

Disk → 96%
Blockchain → advancing
RPC → responding
Memory → available
Target height → ahead

Then conclude:

Node operational.
Synchronization progressing.
Storage capacity remains a risk.

That is a much more useful form of AI.

Not artificial confidence.

Evidence-driven reasoning.

The permanent fix

Cleaning files isn't the long-term solution.

The server needs more storage.

The infrastructure therefore has to evolve as the project evolves.

Today:

79 GB server

↓

Development grows

↓

Blockchain grows

↓

Disk pressure

Tomorrow:

Expanded storage

    ↓

More operational margin

    ↓

Dedicated infrastructure

    ↓

Better monitoring

    ↓

Automated capacity alerts

Eventually, blockchain data may deserve its own dedicated volume rather than competing with application infrastructure.

That's normal.

Architecture changes when reality provides new information.

Building in public means showing this too

It's easy to publish:

"We added AI."

"We integrated blockchain."

"We launched another service."

But engineering also means publishing:

"The disk filled up."

"Synchronization stopped."

"We investigated it."

"We found the bottleneck."

"We recovered capacity."

"The node started progressing again."

Those stories may actually be more useful to developers.

Because that's what running infrastructure really looks like.

MyZubster isn't finished

And we don't want to pretend otherwise.

We're still building.

We're testing.

We're finding bottlenecks.

We're changing architecture when reality tells us that our assumptions were wrong.

That's the difference between drawing a system diagram and actually operating the system.

The diagram says:

App → API → Blockchain

Reality says:

App
↓
API
↓
Node
↓
Database
↓
Filesystem
↓
Storage capacity
↓
Operating system
↓
Network
↓
Monitoring
↓
Recovery strategy

Every layer matters.

The lesson

One of the principles emerging from MyZubster development is becoming increasingly simple:

Don't ask only whether a service is running. Ask whether you can prove that it's working.

That's true for blockchain nodes.

It's true for AI.

It's true for IoT.

It's true for environmental monitoring.

And eventually, it should be true for the entire MyZubster ecosystem.

Build → Observe → Measure → Verify → Recover → Improve.

That's what building in public actually looks like.

🔥 Join developers growing publicly
Share your knowledge, build in public, and grow your developer presence with a global community.

More Posts

AAIF Hits 10 Projects. Its CTO on What to Adopt Now and the Security Gaps Left Open

Tom Smithverified - Oct 23

The Sovereign Vault — A Comprehensive Guide to Protocol-Driven AI

Ken W. Algerverified - Jun 4

My 30-day log cleanup kept a file that was 30 and a half days old

BashSnippets - Oct 8

Why “Building in Public” Is Hollowing Out Your Developer Career

Karol Modelski - Jun 18

The log was deleted. Its 48M was still allocated.

BashSnippets - Oct 28
chevron_left
4.4k Points • 181 Badges
Rimini
112Posts
17Comments
33Connections

Related Jobs

View all jobs →

Commenters (This Week)

1 comment
1 comment
1 comment

Contribute meaningful comments to climb the leaderboard and earn badges!