How A Multi Agent System Decides What To Remember And What To Trust

Leader ●4 ●53 ●178
calendar_today • schedule2 min read

Most of us have built an agent that works well for a week and then repeats a mistake it already made once. This is a writeup of how one self-learning multi agent setup handles that problem, because the interesting part is not the model choice, it is what happens between runs.

Memory Split By Kind, Not One Big Log

The common failure is treating memory as a single store of past conversation. Retrieval then returns whatever is textually nearest, which is usually not what the agent needed.

Splitting memory by kind fixes most of it. Skills hold reusable approaches to a class of problem. Tools catalog what can actually be run. Projects track ongoing work so nothing gets rebuilt from scratch. Memories hold short term events with their outcomes, while experience holds what stays true indefinitely. Rewards keep behavioral rules separate from observations. Questions, ideas, cross references and people each get their own type as well.

The practical payoff is that retrieval knows which shelf to look on. Searching for how to deploy code returns entries about pushing to production and release process even when none of them contain the word deploy, because the lookup is semantic and scoped to a type rather than scanning one undifferentiated pile.

A Pattern Has To Prove Itself

The part worth copying is how a learned pattern gets promoted. When a review finds a cluster of similar situations with similar outcomes, the pattern is stored as pending, not as guidance. It needs five separate confirming observations from later work before agents are allowed to follow it, and three contradictions deactivate it automatically.

That threshold is what prevents an agent from generalizing off one lucky run. Rules a human writes are permanent and outrank everything the system inferred on its own, which keeps the ordering sane: your constraints first, learned habits second, and nothing gets promoted quietly.

Pipelines That Review Their Own Work

Three pipelines run under a coordinating agent. A self learning pipeline brainstorms improvements, then switches to a critical pass that scores its own ideas and discards the weak ones before anything is implemented. A coding pipeline plans, writes, reviews its own output, fixes what it finds and verifies before marking work complete, with different models available at different steps. A research pipeline explores a topic, verifies findings against sources, cross references them and files the results where the other agents can retrieve them.

The separation matters more than the number of pipelines. Generation and criticism happening in the same pass tends to produce agreeable self review, which is worth nothing.

What This Means If You Are Building One

Three things transfer to almost any agent project. Store memory by type so retrieval has structure to work with. Require repeated confirmation before anything learned becomes a rule, and let contradictions retire it. Keep human authored rules above learned ones, permanently.

If you want the full picture of how the memory bank, the rewards engine and the pipelines fit together, the system writeup is here and the whole thing is free software.

What is the promotion rule in your own setup, if you have one? Most projects I have seen either trust the first success or never learn at all, and both extremes cost you.

🔥 Join developers growing publicly
Share your knowledge, build in public, and grow your developer presence with a global community.

More Posts

I’m a Senior Dev and I’ve Forgotten How to Think Without a Prompt

Karol Modelski - Mar 19

How AI Is Changing Web Development, and What Developers Still Need to Understand

Daniel Decker - Nov 10

AAIF Hits 10 Projects. Its CTO on What to Adopt Now and the Security Gaps Left Open

Tom Smithverified - Oct 23

Make for Small Business: What It Is, What It Costs, and When to Use It

floxolab - Oct 23

A Textile Mill's Dyehouse Ramp Curve: Get One Step Wrong and the Sheet Belongs to the Master

aiomniu - Oct 18
chevron_left
5.3k Points • 235 Badges
United States • t.co/5LlztlB5C5
118Posts
23Comments
46Connections
Our AI Apps are a self expanding AI SaaS ecosystem used to create the custom web application of your... Show more

Commenters (This Week)

1 comment

Contribute meaningful comments to climb the leaderboard and earn badges!