This is the fourth entry in a curious builder's diary. In the first, a self-taught web developer borrowed a satellite engineer's intuition and asked whether frontier AI could run on the MacBook on his desk.
In the second, a 235-billion-parameter model said "Hello, world!" through a door we cut in the memory wall.
In the third, it learned to converse without making you wait minutes between turns.
This one is about the words themselves — and about the day we measured our founding idea against a blank wall, and the blank wall won.
1. The Moment
Third turn of a conversation. Same MacBook, same 48 GB, same 117 GB expert vault cold on the SSD:
A complete, correctly-constructed joke — setup, pause, anatomical pun — from a 235-billion-parameter model streaming its brain off a consumer SSD, three turns deep into a conversation, arriving at a bit over one word per second. That's roughly fifty words per minute. The mountain now speaks at the pace of a human typing.
But the speed is only half of this entry. The other half is why the joke got better — because the same change that made the engine faster also made it, for the first time, run the exact mathematics its creators trained. And that change was a deletion.
2. The Numbers, Plainly
Same hardware, same vault, same prompt discipline as every previous entry. Only the engine changed.
| Thing | Last article | Now |
|---|
| Generation speed | 0.70 tokens/s | 1.13 tokens/s |
| …since first light | 0.48 tokens/s | 2.4× total |
| One generated token costs | ~1,430 ms | ~885 ms |
| SSD reads per generated token | ~7.5 GB | ~3 GB |
| Expert routing | predicted, 51.5% right | exact — the true Qwen3 routing, every layer, every token |
| New hardware required | none | none |
And one number that doesn't fit in a table: the speed graph of this campaign was not monotone. It went up, then deliberately down, then honestly back up. That dip is the story.
3. The Instrument Panel: The Web Developer's Ultimate Weapon
In my previous articles, I noted how my web developer instincts managed to outmaneuver deep low-level code bugs. In Article 02, it was discovering a config mismatch with the chat template. In Article 03, it was hunting down stale caches, the most web-shaped bug imaginable.
For this phase, I relied on another classic web developer crutch: raw telemetry over systems theory. Everything in this entry came from one new flag: --instrument, which bucketed every millisecond of every generated token. It measured time spent in dense math, time spinning on the SSD, and one statistic nobody had ever actually measured: how often do adjacent tokens want the same experts?.
The answer: 46.4%. Nearly half of what each token needs, the previous token already fetched.
4. The Power of Deletion (Addition by Subtraction)
That 46.4% number illuminated a harsh truth: our biggest performance gains weren't going to come from building new features. They were going to come from ruthless deletions.
- Deleting the Wipe: We found a twin bug to the one from our last article. The generation loop was wiping the ring on every single token, re-reading 7.5 GB from the SSD against a 46% reuse opportunity. Deleting this wipe cut decode I/O by two-thirds, making the output bit-identical.
- Deleting the Corpse: We discovered our engine was maintaining about 950 lines of ceremony around a 1.6-gigabyte key-value cache that was written to on every token and read by absolutely nothing. It was write-only memory. We deleted it and gave that 1.6 GB straight back to the expert ring.
- Deleting the Ping-Pong: The profiler revealed the CPU and GPU were playing ping-pong 188 times per token (synchronous round-trips for projections, norms, and attention). We rewrote the attention block into one Metal command buffer per layer, bringing the engine down to 94 synchronizations and removing the attention math from the CPU entirely.
S-MoE was born from Ing. Filippo Biondi's beautiful SAR acoustic imaging metaphor: the Surface Scout, a resident seismic sensor listening to the conversation and predicting which experts the mountain would need next. It was the first article. It was my favorite part of the whole design.
But the reality of language models is that true intelligence requires exact math, not approximation. The instrument panel proved our expensive speculative prophet was costing ~265 ms per token and was only right 51.5% of the time. The passive ring buffer—which costs zero compute—covers 46.4% for free. The prophet, wrong as he was, queued ahead of the demand reads that had to happen anyway, actively delaying the truth.
So, we fired the prophet. We switched generation to compute routing exactly, from the true hidden state, with one tiny matrix multiply per layer.
Trading throughput for truth initially caused a dip in our speed, but the multi-turn output visibly improved. The irony is beautiful: by sacrificing speed to get the exact true Qwen3 routing mixing weights, we eventually unlocked the architectural simplicity needed to make it faster.
6. The Experiments That Failed, Published Anyway
This diary's standing promise. Two more for the pile:
Speculation is real, and it still lost. Ranks 9–16 genuinely cover 63.7% of the next token's experts. We built a low-priority prefetch queue for it, but at 4-bit, it was net-negative—speculative reads simply bid against demand reads on a saturated bus. It sits in the code, default-off, waiting for the 2-bit vault.
We grouped the GPU dispatches, and it changed nothing. Theory said batching each layer's expert kernels would hide wait times. Measurement said the command-buffer overhead was ~5 ms all along. The hypothesis is dead.
7. Hitting the Wall of Physics
Here is where the campaign shifts. Of the ~885 ms each token now costs, about 490 ms is the engine spinning on the SSD.
Three gigabytes of misses per token divided by 490 ms is 6.3 GB/s: the NVMe's actual delivered bandwidth, saturated. There is no more bloated software left between us and the flash memory. There are no dead caches or unnecessary speculative math. For the first time, the biggest cost is simply the raw physics of the hardware doing exactly what it says on the box.
This sets up a beautiful, concrete cliffhanger: you cannot make a saturated physical bus faster, but you can send half the bytes.
The upcoming 2-bit vault will halve every expert blob, halve the SSD bill, double the ring's capacity, and finally arm our speculative queue.
8. What This Entry Is Actually About
The pattern, four articles in, is impossible to miss: the exotic layers keep being innocent. What keeps falling is belief: caches that don't cache, insurance nobody claims, prophets who lose to blank walls. The instrument panel is the only law we've kept, and it has now voted against our own founding metaphor.
And the authorship remains the experiment inside the experiment: the AI agent wrote the NEON intrinsics, the Metal kernels, and the instrumentation. I brought the questions — what is it actually doing in there? who reads this cache? is the prophet even right? — and the willingness to bury the answer's body in public.
The mountain speaks at typing speed now. It no longer needs anyone to predict it.
It predicts itself.
S-MoE is open source under the MIT License. The code, the philosophy, and the mistakes are all public: github.com/melasistema/s-moe. Built by Luca Visciola and an AI agent who hopefully knows C++ and low level programming, because I do not 😅.
Acknowledgements: The seismic tomography concept remains a philosophical translation of the published work of Ing. Filippo Biondi on SAR-based subsurface imaging. The Scout retires with honor — the listening was never wrong; the mountain simply turned out to be its own best seismograph.