The Mountain Runs — I Don't Know If We're Chasing Something or Being Chased, but the Speed Won't Stop Climbing
This is the fifth entry in a curious builder's diary. In the first, a self-taught web developer borrowed a satellite engineer's intuition and asked whether frontier AI could run on the MacBook on his desk. In the second, a 235-billion-parameter model said "Hello, world!" through a door we cut in the memory wall. In the third, it learned to converse without making you wait minutes between turns. In the fourth, we fired our own founding metaphor and the model got faster and smarter. That article ended at the wall of physics: a saturated SSD, nothing left to delete. This one is about what happened next, which is that the speed kept climbing anyway — and for one terrifying evening, it looked like it had collapsed by 85% instead.
1. The Moment
A microbenchmark, late at night. Same MacBook. Same GPU. Same weights, same mathematics, same number of bytes read. Two kernels that compute the identical matrix-vector product:
K/V projection (512 × 4096, bf16)
old kernel: 12.3 GB/s new kernel: 285.3 GB/s (23×)
Twenty-three times faster. Nothing was added — no new hardware, no new math, no quantisation trick. We changed the shape of the question the GPU asks its own memory, and the same silicon that had been trudging along at 12 GB/s broke into a sprint.
Last article ended with me declaring, confidently, that we had hit the wall of physics — that the SSD was saturated and there was no more bloated software between us and the flash memory. That was true, and it remains true. What I had not understood is that the SSD was only one of the walls, and behind it the GPU had been quietly doing the memory-system equivalent of taking a taxi one block at a time.
2. The Numbers, Plainly
Same hardware, same 117 GB vault cold on the SSD, same canonical prompt discipline as every previous entry. Only the engine changed.
| Thing | Last article | Now |
|---|
| Generation speed | 1.13 tokens/s | 1.43 tokens/s |
| …since first light | 0.48 tokens/s | 3× total |
| One generated token costs | ~885 ms | ~700 ms |
| First token, cold 45-token prompt | ~32 s | ~29.4 s |
| GPU dense projections | 12–136 GB/s | 207–285 GB/s |
| GPU syncs during prefill | 2 per token | 2 per layer |
| Output token IDs | — | bit-identical, before and after |
| New hardware required | none | none |
And one number that isn't in the table, because it deserves its own section: for a few hours in the middle of this campaign, the official benchmark read 0.20 tokens/second — six times slower than where we started the evening. That story is below, and it is the most web-developer-shaped lesson of the entire diary.
3. The GPU Was Running N+1 Queries
Every web developer knows the N+1 query bug. You ask the database for a list of users, then loop over them asking for each one's profile — a thousand tiny round-trips where one JOIN would do. The database isn't slow. Your access pattern is rude.
It turns out a GPU's memory system has exactly the same personality.
The old kernels assigned one thread per output row of the weight matrix. Sounds reasonable — it's the obvious loop, the one you'd write first. But 32 neighbouring GPU threads execute in lockstep, and under that layout their simultaneous memory requests land at addresses two kilobytes apart. The memory system, which serves wide consecutive bursts beautifully, has to satisfy 32 scattered pleas one crumb at a time. A thousand tiny round-trips. N+1 queries against your own RAM.
The new kernels invert it: one simdgroup — 32 threads — owns one output row together, and its lanes read consecutive 16-byte chunks. Same bytes, same total work, but now every request is one coalesced 512-byte burst. The JOIN.
- The quantised expert FFN kernels — the ones that dequantise 2/4-bit codes in register space, the heart of the whole engine — jumped 7×, to 158 GB/s effective.
- The dense Q/K/V/O projections and the LM head jumped to 207–285 GB/s.
- The comedy prize goes to the K/V projection: its matrix is only 512 rows tall, and the old dispatch geometry handed the entire job to 2 threadgroups — on a GPU built to run thousands. We had booked a stadium and seated two people. That's the 23×.
The pattern of this diary holds for a fifth consecutive entry: the exotic layer was innocent. The dequantisation math was right, the Metal code was correct, every byte arrived. What was broken was the shape of the request — the most database-shaped bug imaginable, living in the middle of a GPU shader.
4. The Prompt Stopped Playing Ping-Pong
The second front was the prefill — the silence before the first word, which article three cut from 93 seconds to 32.
Reading the prompt, the engine was still submitting GPU work per token: every one of the 45 prompt tokens made its own little QKV round-trip and its own little o_proj round-trip, each ending with the CPU standing at the door waiting for the GPU to ring the bell. Roughly 8,460 command buffers for one prompt. Now each layer submits all tokens' QKV projections in one command buffer, and all their output projections in another — 188 submissions total, two per layer. The GPU finally gets asked the whole question at once.
And one more theft from the I/O timeline: during routing, the moment the first token votes for an expert, that expert fires at the SSD immediately — while the remaining tokens are still deciding. The reads were always going to happen; now they start milliseconds earlier, hidden under compute. After last article's lesson about speculation, I want to be precise about why this one is legal: replaying yesterday's routing as a guess measured 37% slower and stays banned. This is different — it is certain demand, pulled forward. Never bet. Just start walking earlier.
5. The Evening the Benchmark Lied
Here is the danger in the title.
We land the new coalesced kernel — the one the microbenchmark just crowned at 285 GB/s — into the real engine, run the canonical benchmark, and:
decode: 0.20 tokens/s
Not a typo. One-sixth of the previous speed. The kind of number that, at 11 PM, makes you believe you've corrupted something fundamental — a memory fence, an alignment, the laws of thermodynamics.
Old me would have started reverting commits. Instead, the instrument panel — still the only law of this project — got the first word. And the instrument said something astonishing: the dense math bucket had gotten faster, exactly as the microbench promised. The regression was entirely inside io-spin: the SSD, which reliably delivers 6.3 GB/s, was crawling at 4.4.
The kernel was innocent. The machine was tired. Running benchmark after benchmark back-to-back, each launch mapping a 16 GB Scout and churning gigabytes of Direct I/O, had driven macOS's memory pressure into the ground — and the OS quietly throttles you when you do that. The benchmark wasn't measuring my code. It was measuring my impatience.
One quiet run later: 1.43 tokens/second. The rule is now written into the project's law books: one launch per comparison, on a rested machine — the systems-programming equivalent of "never trust a load test you ran against staging while the nightly backup was running." My decade of being lied to by test environments finally paid rent in C++.
And in the spirit of publishing everything: the numbers in this article were measured on a machine whose memory was half-eaten by my own IDE, because a web developer's laptop is never actually quiet. The reference conditions may be slightly kinder. I'd rather publish the conservative number than wait for a silence that never comes.
6. The Wall, Revisited
So where does a generated token's ~700 ms actually go now?
| Where | ms | What it is |
|---|
| SSD spin | ~490 | demand misses at the NVMe's saturated bandwidth — pure physics |
| Dense math | ~155 | weight bandwidth through the new coalesced kernels |
| GPU wait | ~46 | expert kernels finishing on late-arriving misses |
| LM head | ~8 | one vocabulary-sized matvec |
Seventy percent of every token is now the raw, honest cost of flash memory delivering bytes at its rated speed. Last article's cliffhanger hasn't moved, it has sharpened: you cannot make a saturated bus faster, but you can send half the bytes. The 2-bit vault remains the next mountain — it halves the SSD bill, doubles the ring, and finally arms the speculative machinery that has been sitting in the code, patient and default-off, for two articles now. The remaining sliver of dense time has its own successor waiting: the last per-layer CPU round-trip, where the residual and norms still come home between command buffers.
7. What This Entry Is Actually About
The title is a genuine confession. When the speed keeps climbing after you've declared the wall of physics, you start to wonder which direction you're running. Are we chasing some optimal engine that's always one commit ahead? Or fleeing something — the 0.20 t/s ghost, the next benchmark that lies?
Four campaigns in, I finally looked over my shoulder. This is what I found.
And the answer is: neither. Nothing is chasing us. What's behind us is just everything we deleted — the wiped caches, the fired prophet, the write-only memory, and now the scattered reads and the 8,460 tiny GPU round-trips. The engine isn't accelerating toward or away from anything. It's shedding. Every entry of this diary, the mountain weighs less than the diary said it did.
The authorship remains the experiment inside the experiment: the AI agent wrote the simdgroup kernels, the barrier-free dispatch paths, and the microbenchmarks that isolated silicon truth from environmental noise. I brought the questions — why is the GPU only using two threadgroups? why did the benchmark collapse when the kernel got faster? — and the N+1 instincts of someone whose databases have been punishing rude access patterns for years.
The mountain vibrated. Then it talked, then it remembered, then it predicted itself.
Now it runs. And honestly — I've stopped trying to guess what from.
S-MoE is open source under the MIT License. The code, the philosophy, and the mistakes are all public: github.com/melasistema/s-moe.
Built by Luca Visciola and an AI agent who hopefully knows C++ and low level programming, because I do not 😅.
Acknowledgements: The seismic tomography concept remains a philosophical translation of the published work of Ing. Filippo Biondi on SAR-based subsurface imaging. He taught this project to listen to a surface for news of the interior; it turns out the same lesson applies to a benchmark on a tired machine.
