This is the third entry in a curious builder's diary. In the first one, a self-taught web developer borrowed a satellite engineer's intuition and asked whether frontier AI could run on the MacBook on his desk. In the second, a 235-billion-parameter model said "Hello, world!" through a door we cut in the memory wall — the SSD. That article admitted, honestly, that the door was slow to open: minutes of silence between pressing Enter and the first word. This one is about the silence.
1. The Moment
The first article proved the mountain could vibrate. The second proved it could speak. This one is smaller and, I think, more human: the mountain learned to converse.
You: Explain in one sentence what a Mixture of Experts model is.
S-MoE: A Mixture of Experts (MoE) model is... [first word after ~32 s]
You: Now say it in exactly three words.
S-MoE: Experts combine weights [first word after ~14 s]
Look at the second exchange. Not the speed — the obedience. A 235-billion-parameter frontier model, streaming its experts off a consumer SSD, understood "say it in exactly three words," reached back across the conversation for what it was, and answered in exactly three words. The context survived between turns. The memory held. And it answered in fourteen seconds instead of two minutes.
When we started this phase, the same second question would have cost more than the first — because the engine was reborn from nothing on every turn, re-reading everything, re-thinking everything, forgetting everything. Every exchange made the wait longer. Now every exchange makes it shorter, because the past is already paid for.
2. The Numbers, Plainly
Same MacBook. Same 48 GB. Same 117 GB vault cold on the SSD. Same model, same weights, same mathematics — the outputs are bit-for-bit identical where the math didn't change, and more correct where it did.
| Thing | Before | After |
|---|
| First token, cold 45-token prompt | 92.7 s | 32.4 s |
| First token, follow-up turn | ~96 s, growing every turn | ~14 s, constant |
| Generation speed | 0.48 tokens/s | 0.70 tokens/s |
| Prompt I/O per token | ~7 GB re-read from NVMe | each needed expert read once |
| New hardware required | — | none |
No new machine. No smaller model. No quantisation trick that trades away intelligence. Just sitting with the engine for long evenings and asking, over and over, the only question that ever matters in systems work: where does the time actually go?
3. The Prompt Is Not a Prophecy
The deepest fix wasn't code. It was noticing a category error.
S-MoE's whole philosophy is prediction — the Surface Scout listens to the conversation and forecasts which experts the model will need next, so the SSD can fetch them before the GPU asks. It's the phonon metaphor, and for generation it's right, because generation is genuinely unknown.
But we were running the prophet on the prompt. And the prompt is not the future — it's the archive. Every token of it is already known. We were paying a full 94-layer speculative forward pass per prompt token to predict routing decisions that could be computed exactly, from the true hidden state, with one tiny matrix multiply per layer. Prediction where certainty was available. A weather forecast for yesterday.
So during prefill, the Scout now stands down. Routing is exact — the miss rate is zero not because we got better at guessing, but because we stopped guessing. And that opened the door to the real structural change:
We turned the prompt loop inside out. Instead of marching token by token through all 94 layers (each token privately demanding its ~7 GB of experts from the SSD), the prompt now moves through the model layer by layer, in chunks of 64 tokens. At each layer, the chunk's tokens vote on which experts they need, the duplicates collapse, and each expert is read from the vault once — then applied to every token that asked for it, in a single GPU dispatch.
The prompt's I/O bill stopped scaling with its length. That one inversion is most of the 3×.
4. The Cache That Threw Everything Away
Here is the discovery that made me laugh out loud, and I share it because this diary promised the mistakes along with the victories.
The engine has a ring buffer — a cache of recently-fetched experts, the beating heart of the streaming design. While tracing the prefill path, we found that on every single prompt token, a housekeeping routine walked the entire ring and evicted everything. The cache was being emptied at precisely the moment it was most useful. Every prompt token re-read its full expert set from the SSD — seven gigabytes — into a cache with the memory span of the token itself.
Fixing that revealed two deeper cracks that the constant evictions had been hiding, the way a flood hides the potholes: a deadlock where the engine insisted an expert was "already in the ring" while simultaneously refusing to hand it over (we watched it spin at 500% CPU, saying nothing, for six minutes), and a genuine race where the streamer could start overwriting an expert's data while the GPU was still reading it. Both had been in the code all along. Both were unreachable until the cache started actually caching.
The lesson, again: the exotic layer was innocent. The Metal kernels were fine; the quantisation math was fine. What broke was cache invalidation — famously one of the two hard problems in computer science, and the most web-shaped bug imaginable. My decade of being betrayed by stale caches in staging environments turned out to be, once again, the useful instinct.
5. The Experiments That Failed, Published Anyway
Two things we built did not work, and I want them in the record — partly for honesty, partly because negative results are the most expensive knowledge to lose.
We tried prophecy-by-precedent, and it made things worse. Before committing to the layer-major inversion, we tried a cheaper trick: prefetch each prompt token's experts using the previous token's routing, on the theory that neighbouring tokens want similar experts. Measured result: 37% slower. In Qwen3-235B, adjacent tokens overlap far less in their expert choices than intuition suggests — the hints mostly streamed the wrong experts ahead of the demand fetches that had to happen anyway. The failed experiment is documented in a comment in the source, at the scene of the crime, so that no future contributor (human or agent) walks that road twice.
We built a learning cache, and today it changes nothing. The engine now remembers which experts fire most, persists that histogram across sessions, and quietly pre-warms the hottest ones while you're typing your next message. The machinery works. The measured benefit today: zero. The ring simply cannot hold enough of a 235B model's working set at 4-bit for popularity to matter — it's a bookshelf of 580 slots trying to cache a library of 12,000. We're keeping it, because the next chapter (2-bit vaults, bigger rings) doubles and triples the shelf space, and the histogram is already learning. But I won't dress a zero up as a win. In the process we also discovered the old pre-warmer had been silently doing nothing since the day it was written — its safety margin was larger than the entire ring. Nobody noticed, because how do you notice the absence of a warmth you never felt?
6. One Line, Twenty-Four Seconds
A confession about where the very first win came from.
For every prompt token, the engine was computing the full output vocabulary — a 151,936 × 4,096 matrix multiply on a single CPU thread — and then throwing the answer away, because a prompt token's next token is already known. It's the prompt. Six hundred million multiplications per token, per 44 tokens, into the void.
The fix is one if statement. It was worth twenty-four seconds — a quarter of the entire original wait.
I keep this example close because it is the whole project in miniature: the barrier was never the hardware. The hardware was heroically streaming gigabytes off the SSD the entire time. The barrier was a loop doing faithful, expensive, perfectly-executed work that nobody needed. I suspect the world's datacenters contain rather a lot of that.
7. What's Next
The system that converses today is, once again, the slowest version of itself that will ever exist. The wait is now ~85% raw SSD reads — we've stripped away nearly everything else — which means the roadmap is beautifully concrete: a 2-bit vault done carefully (our first attempt taught us how fine-grained experts punish careless quantisation) would halve the remaining I/O and finally arm the learning cache; a braver ring budget turns spare RAM directly into silence removed; and the generation loop still has one 256×-overprovisioned GPU dispatch pattern waiting to be humbled.
And the authorship remains what it has always been: the AI agent wrote the C++, the Metal kernels, the lock-free state machines, and found the races I could only squint at. I brought the questions — why is it quiet for so long? what is it actually doing in there? are we even caching anything? — the stubbornness, and the willingness to publish the failures next to the wins. Neither of us could have done this alone, and the pairing itself is still the experiment I find most notable.
The mountain talks.
Now it remembers the conversation — and it no longer makes you wait to find out.
S-MoE is open source under the MIT License. The code, the philosophy, and the mistakes are all public: github.com/melasistema/s-moe.
Built by Luca Visciola and an AI agent who hopefully knows C++ and low level programming, because I do not 😅.
Acknowledgements: The seismic tomography concept remains a philosophical translation of the published work of Ing. Filippo Biondi on SAR-based subsurface imaging. The prompt taught us the difference between prophecy and archive; the intuition to listen was his first.
