This is the seventh entry in a curious builder's diary. Over six chapters, we carved a door through Silicon Valley's artificial memory wall, coaxing a 235-billion-parameter mountain to speak, remember, run, and finally converse fluently in the OpenAI dialect. As we push past an average of 1.84 tokens per second and touch peaks of 2.01 on a consumer 48 GB MacBook, the engine is humming. But before we pack our gear for the next impossible summit, we are claiming a builder's privilege. After months of writing lock-free C++ to move heavy stone, it is time to take a breath of thin, fresh air.
1. The 477B Avalanche
S-MoE was built to prove a point: that the memory wall is a software convention, not a law of physics. We took Qwen3-235B — a model that standard runtimes insist requires cloud clusters — and shattered it into pieces that stream perfectly off a MacBook's SSD via Apple Silicon's Unified Memory Architecture.
Because we broke the diary dependency and invited standard frontends (like Open WebUI) into the architecture in Article 06, we are now ready to tackle an even more colossal peak: the GLM MoE 477B parameter model.
The climb will be grueling. A 477-billion-parameter model will push consumer hardware to an edge no one thought possible. It will test every byte of our zero-allocation loops, our Direct I/O DMA transfers, and our patience. But we have the map now.
However, before we shoulder that massive weight, we realized something crucial. S-MoE shouldn't just be an engine for the 235B. It should be an engine for any fine-grained MoE.
2. The Self-Describing Vault (SARC)
Until now, the engine was heavily hardcoded for Qwen3's specific math. If we were going to run different models — the daily driver 30B, the massive 235B, or the upcoming 477B — we couldn't keep recompiling C++ to change the RoPE theta or the routing Top-K.
We needed the engine to be model-agnostic.
The solution was elegantly simple: the Smoe ARChitecture (SARC) block. During the offline shatter process, the Python script now reads the model's Hugging Face config.json and extracts the mathematical DNA — RoPE theta, MoE Top-K, GQA head geometry, and normalization flags. It serializes these into a tiny, 128-byte binary block and stamps it directly into the .smoe vault's header.
Now, when you boot S-MoE, there are no config files or CLI flags to set. The engine simply reads the 128 bytes, understands the topology of the mind it was just handed, and configures its own internal Scout and Metal shaders accordingly.
We suddenly had a "fleet." You can have a directory full of different vaults, and simply switching the model in your Chat UI dropdown seamlessly swaps the models at runtime.
3. The 30B Wind
What happens when you take an engine hardened by the brutal math of a 235B model, and plug a 30B model into it?
We shattered the Qwen3-30B-A3B checkpoint to find out. We wanted to experience pure, frictionless velocity. We wanted to ride the wind.
But when we first spun it up, the numbers were confusingly low. We were seeing about ~5 to 7 tokens per second. Fast, but not "frictionless velocity" fast. We expected the 30B to scream on a 48 GB machine.
The culprit? A single hardcoded flag from our testing days: --ring 1024.
4. The Magic of Auto-Sizing
The Streamer's ring buffer is the heart of S-MoE's speed. It's a pre-allocated LRU cache in RAM. When a token needs an expert, the engine checks the ring first. If it's there (a cache hit), we skip the SSD entirely.
By hardcoding --ring 1024, we were forcing the ring to only hold 1,024 experts (about ~2.5 GB of cache for the 30B).
But S-MoE has a brilliant auto-sizing mechanism (--ring 0). When allowed to auto-size, the engine dynamically checks the OS for available RAM, subtracts a safe overhead, and gives the rest to the ring. On a 48 GB MacBook, the auto-tuner allocates roughly ~15 GB of ring capacity.
The entire 30B vault is only ~14 GB.
The moment we removed --ring 1024 and let the engine auto-size, the ring simply swallowed the entire model. After the first few turns, the SSD reads dropped to virtually zero. The engine ascended from a streaming pipeline into an pure, in-memory execution loop.
The numbers instantly skyrocketed.
5. The True Benchmarks
With the auto-tuner unleashed and the engine running isolated, the true benchmarks emerged.
The Daily Driver: Qwen3-30B
- Cold Time-To-First-Token: ~5.85 seconds
- Decode Speed: 13.71 tokens / second
- The Feeling: Instantaneous. Pure velocity. The entire mind rests comfortably in RAM, producing outputs faster than you can read them.
The Frontier: Qwen3-235B
- Cold Time-To-First-Token: ~26.16 seconds
- Decode Speed: 1.84 tokens / second (peaking over 2.01 on warm turns)
- The Feeling: Cinematic deliberation. The SSD streams experts at ~150+ GB/s, and the mountain answers thoughtfully.
6. The Plateau Before the Climb
We built S-MoE to prove that the 235B frontier could live on a desk. But in making the architecture truly agnostic, we accidentally built the fastest possible way to run the 30B models, too. The same lock-free memory gymnastics that make the heavy models possible make the lighter models absolutely fly.
Because sometimes, before you carry the next mountain, you just need to feel how fast you can sprint across the plateau.
The engine is ready. The architecture is model-agnostic. The UI is clean.
Next stop: 477 Billion parameters.
S-MoE is open source under the MIT License. The code, the philosophy, and the mistakes are all public: github.com/melasistema/s-moe.
Built by Luca Visciola and an AI agent who hopefully knows C++ and low level programming, because I do not 😅.