A few months ago I got tired of using language models without understanding a single thing happening under the hood.
APIs make it too easy. You send a prompt, something smart comes back, and you never see the machinery.
So I built my own. From scratch. In PyTorch. No shortcuts.
That project is called LOOM-GPT.
The idea that hijacked the whole project
Original plan: train one small transformer, learn how attention works, move on with my life.
Then a dumb question hijacked the roadmap:
Why train one model to know everything, when five small models could each know one thing really well — and then just... talk it out?
Poetry model. Tech model. Philosophy model. Each trained separately. Each opinionated in its own lane.
Then, at generation time, let them fight over every single token.
I called it Model Weaving, and it's now the entire personality of this project.
How the fight actually works
At each step, every specialist predicts the next token independently.
LOOM blends their probability distributions using whatever weights you hand it — 70% poetry, 30% tech, whatever mood you're in — and samples the winner from the merged distribution.
Repeat for every token.
The result isn't "one model talking." It's a committee meeting that somehow produces a sentence.
Poetry specialist, unsupervised:
"The city at night breathes slow, each lamp a word the darkness reads and folds into its keeping."
Tech specialist, unsupervised, having a very different night:
"The city at night runs distributed processes across nodes, latency rising as packet loss compounds at scale."
Both of them, forced into couples therapy at 70/30:
"The city at night holds its network close, each signal a quiet pulse the infrastructure learns to trust."
That last one shouldn't work. It works.
The receipts, not just the vibes
I didn't want a black box that occasionally says something poetic and calls it a day. So every generated token gets traced back to whichever specialist pushed hardest for it.
Export it, and you get a JSON trail of who said what and by how much — the beginnings of a "watch the argument happen in real time" dashboard.
Turns out interpretability is just... receipts. Who influenced this word, and by what percentage. That's it. That's the whole trick.
Things that humbled me
- Small models rat you out instantly. Feed a bigram baseline messy data and validation loss collapses before you can pour a second coffee. No hiding behind scale.
- Blended output was suspiciously more coherent than it had any right to be. I expected soup. I got style.
- Byte-level tokenization solves more problems than it deserves credit for. English, code, markdown, whatever mess of files you throw at it — same pipeline, no drama.
Where it stands now
LOOM-GPT is live on PyPI. Train specialists on your own folders, weave them, watch the influence trace, repeat until you have opinions about your own code comments.
pip install loom-gpt==0.1.0
loom --help
Presets range from tiny (2 layers, for smoke tests and impatience) up to single_gpu (6 layers, for people who actually have a GPU and want to use it).
What's next
There's a loom_v2/ folder sitting in my workspace that I haven't opened with clean intentions yet. Questions currently unanswered:
- What if specialists could adjust their own weighting mid-generation instead of me hardcoding it?
- What if weaving stopped being a text-only party trick?
- What if the specialists didn't even live on the same machine?
I don't have answers. I have a folder. That's usually how these things start.
LOOM-GPT is open source and built entirely from scratch — no pretrained weights, no cutting corners, just a lot of small models with strong opinions.
https://github.com/Karthik-Unni/Loom-gpt
