Git Was Never Built for a Thousand Agents. Here Is How OpenAI Rebuilt the Server Behind It.

Question

At the end of last year, the Git servers behind OpenAI's main repository started to buckle: fetches and pushes were slow, and merges failed. OpenAI says reliability at times fell below 90%. For a team where every engineer now works alongside several Codex agents, that means a lot of people and machines sitting and waiting on version control.

The fix was not a new version control tool. Every engineer still types git push and git fetch, exactly as before. What OpenAI replaced was the thing on the other end of the wire: the server. They called the new one Pando. It now serves nearly a petabyte of Git data a day, it cut the median push time by nearly 5x, and it does most of its serving from cache.

I spent the day reading their write-up line by line, checking it against the Git protocol docs, and tracing a real fetch on my own laptop to see the bytes. This is what I found, in plain words, from the first idea to the parts that break in production. At the end there are five interview questions with short answers, because this design is going to show up in system design rounds.

First, five words you need

If you already know Git internals, skip this part. If not, these five words carry the whole article.

• Object. Git stores your code as objects. There are four kinds: a blob is the content of one file, a tree is a folder listing, a commit is a snapshot plus a message and a parent, and a tag is a named label. Each object is named by a hash of its contents, so an object can never change. Change one line in a file and you get a brand new blob.
• Ref. A ref (short for reference) is a name that points at a commit. main is a ref. feat/login is a ref. A branch is nothing more than a ref that moves forward when you commit.
• Packfile. Sending objects one by one would be slow, so Git squeezes many objects into one compressed file called a packfile, or just a pack.
• Push. You send a pack of new objects to the server and ask it to move a ref to your new commit.
• Fetch. You tell the server which commits you want and which you already have. The server works out the difference and sends it back as a pack.

That last sentence is the one to hold on to. On a normal Git server, every fetch makes the server compute a fresh pack, just for you.

Why the old shape stopped working

OpenAI describes two pressures, and both come from agents.

The first is many writers. One engineer with Codex now has several changes ready for CI at once. Each change is its own branch, and each push moves a different ref.

The second is many readers. In OpenAI's words, each push can set several jobs fetching the same commit. Dev environments, research jobs and CI all clone and fetch the same history, over and over.

So the server is squeezed from both sides. Lots of small writes to different branches, and lots of big reads of the same data.

The two textbook answers, and where each one hurts

Before building anything, OpenAI modelled the two usual ways to run one Git repository on many machines. I think this is the most useful part of the post for anyone learning distributed systems, because both designs are correct and both fail in a way you can predict.

Design 1: three-phase commit. Every replica (a server holding a complete copy of the repository) gets every push. To publish a push, a coordinator asks the replicas to lock the ref and check it still points where the client expects. Then it sends a pre-commit message. Once a majority (a quorum) agrees, it tells them all to apply the change. You get a beautiful guarantee: any replica shows the same view of every branch. You pay for it on every single push, because the push now waits for several disks and several network round trips. OpenAI's simulation found that this design's write latency climbs under load as the replica disks saturate.

Design 2: a shared write-ahead log. A write-ahead log (WAL) is a list of changes written down before they are applied. Here the list is one shared file, a manifest, in blob storage. Each push uploads its pack, then tries to add its entry to the manifest with a compare-and-swap. A compare-and-swap (CAS) is an update that only succeeds if the value has not changed since you last read it. If someone else got there first, you re-read and try again. Uploads run in parallel, but every push in the whole repository, even to unrelated branches, fights over that one manifest. And every replica still has to read and index each new pack before it can serve it. OpenAI found this design's read latency climbs under load, as replicas fall behind the log.

So one design is slow to write, the other is slow to read. Pando's answer was to question two things that both designs take for granted.

The two assumptions Pando dropped

OpenAI names them directly:

1. Repository-wide consistency. The idea that at any moment there is one true snapshot of every branch together.
2. A unique packfile for every client. The idea that each fetch deserves a pack computed just for that client.

Drop the first, and writes to different branches stop competing. Drop the second, and reads become file serving instead of computing. Everything else in Pando follows from these two decisions.

Idea one: a branch is just a pointer to a stored pack

When you push, Pando takes your pack and stores it as an immutable blob in a blob store. Immutable means it is written once and never changed. Then it writes a small record called a ref version into a separate metadata store. A ref version holds three things: the new tip commit, the pack that supplied it, and any earlier ref versions that pack depends on.

The branch itself, feat/login say, is now just a pointer to its latest ref version. That pointer is the only thing in the whole system that two pushes can fight over. OpenAI says it plainly: the only contended key is the ref pointer, and they guard it with an atomic compare-and-swap-like operation.

Look at the figure again. The packs sit wherever the blob store put them. Nothing is sorted, nothing is unpacked, nothing is merged into a big repository on disk. The order lives in the pointers, not in the storage. This is the same trick a database index plays: the index knows the order, the pointer knows the place.

Idea two: a push touches one key, not the whole repository

Here is a single push, in order:

1. Agent A runs git push for feat/login. Git builds a pack and starts sending it.
2. Pando streams the pack straight into blob storage and checks it on the way through, in a single pass. It never unpacks it into a local repository.
3. In one multi-key transaction, Pando writes the new ref version and swaps the branch pointer to it.
4. The swap succeeds.
5. Pando tells the agent the push is accepted. Because the pointer moved inside the same transaction, the agent can fetch its own commit straight away. OpenAI calls this read-your-own-write consistency.

Now Agent B pushes to the same branch, built on the old tip. At step 3 its swap fails, because the tip has already moved. That collision is the only kind Pando has. A push to fix/cache-ttl and a push to docs/setup never wait for each other, and never wait for a majority vote. OpenAI says the whole write needs a constant number of metadata reads and writes, whatever the size of the push. For very large pushes, only some commits are written synchronously and the rest is scheduled as background work in the same transaction.

To add more write capacity, you add Pando servers, metadata store capacity, or blob stores. None of them hold a repository copy, so there is nothing to keep in sync.

The price: branches can disagree about time

This is the trade-off, and OpenAI does not hide it. Pando gives each single branch a strict order of updates. It does not give you one moment in time across all branches.

Their own example: suppose foo and bar are both updated while a clone is negotiating. One client may see foo at version 1 and bar at version 2. Another client may see foo at version 2 and bar at version 1. Both are valid.

The name for this is causal consistency. To fetch a branch, you get exactly the history of the writer who produced it, nothing less. What you do not get is a global clock. OpenAI's bet is that people and agents care about the order of updates to one branch, and almost never about the global order across all of them. For a monorepo full of independent feature branches, that bet looks right to me. For a system where two refs must always move together, it would not be.

Idea three: tell the client less, so packs can be shared

There is a second, sneakier trade-off on the push side.

When a normal Git server receives a push, it first advertises every ref it has. OpenAI compares this to a poker player showing all of its cards. The client uses that list to leave out objects the server already has, which keeps the push small.

I wanted to see how big that list really is, so I traced a real fetch from my laptop against the public Git source repository on GitHub (Git 2.39.5, protocol version 2, GIT_TRACE_PACKET=1). The server listed 1,013 refs, one branch and 1,012 tags, and those lines came to 109,809 bytes. That is about 108 bytes per ref. OpenAI says that with millions of branches, the ref advertisement alone can exceed 100 MiB. My numbers agree with theirs: at roughly 108 bytes a line, 100 MiB is a bit under a million refs.

Pando does something different. When you push a new feature branch, it advertises only one trunk branch as the base. Your pack then depends on trunk and nothing else. Anyone who already has trunk can use your exact pack, unchanged. Smaller advertisement, and a pack that can be reused by every reader.

The cost is that pushes get bigger. If you push to the same branch many times, or stack one feature branch on another, Git has to resend everything outside trunk. OpenAI measured this in their monorepo: the median pack produced this way was under 20 KiB, and the 99th percentile was under 3 MiB. Small enough that they called it a sensible trade.

Idea four: a fetch gets links, not a freshly built pack

On a fetch, your Git still sends its wants and haves. In my trace, the request was exactly what the figure shows: command=fetch, deepen 1 (because I asked for a shallow fetch), one want line with the commit hash, and done.

A normal server would now walk the object graph and build a pack just for me. Pando instead looks up my haves and wants in the commit mappings it recorded on earlier pushes, turns them into a set of ref versions, and picks the stored packs that cover what I am missing.

Then it uses a Git feature that very few people know about, called packfile URIs. The Git protocol only lets a server send one packfile inline in the response. With packfile URIs, the server can also send a list of links, each one a 40-character pack hash and a URL, and the client downloads every link before it checks the result. This only exists in protocol version 2. Pando uses it for everything and sends the inline packfile empty.

One honest detail from my own run: GitHub's server did not advertise packfile-uris in its capabilities, so my Git did not ask for links, and the reply came back as a normal packfile. So the right half of the figure is the format from the Git docs and OpenAI's description, not a capture. Also note that on stock Git the client only accepts these links if you set fetch.uriprotocols, which is empty by default. OpenAI ships its own Git build to its machines, and has contributed packfile URI fixes upstream.

Idea five: the links point at a cache that almost never misses

Those URLs point at a pack delivery service called Express, which caches the blob store in each region. Requests for a pack are routed to Express servers with consistent hashing, a way of spreading keys over servers so that each pack always lands on the same server, and adding a server only moves a small share of the keys. Because the same pack always goes to the same place, it stays hot. OpenAI reports cache hit rates of over 99%.

The numbers that follow from this are the ones that made me stop and reread:

• Clones and fetches fill 10 Gbps network links, on standard cloud virtual machines.
• Pando has served more than 57 terabytes of packs in a single hour, with more than 99% of those bytes from local disk cache.
• It delivers a petabyte of Git data per day with a handful of CPU cores.

A handful of cores for a petabyte a day is only possible because the servers are not computing packs. They are mostly picking files and handing out links. If you want the general version of this idea, store the finished answer once and serve it to many readers, I wrote a free lesson on it: server-side caching.

Idea six: do not make a client download ten thousand tiny packs

There is a problem hiding in idea one. If every push is its own pack, a client that is one day behind on trunk would need, in OpenAI's words, tens of thousands of packfiles to catch up. That is a lot of links and a lot of round trips.

So Pando borrows from LSM trees (log-structured merge trees), a storage design where small files are merged in the background into bigger ones, level by level. In Pando, each push makes an "L0" pack, the smallest unit it can send. Background workers group L0 packs into covers that span a run of ref versions, and the covers grow geometrically, level after level. OpenAI's monorepo goes up to L4 today. When a cover is full, a worker repacks it into one packfile, which also compresses better.

The covers work like a skip list, a structure where higher levels let you jump over many items at once. By scanning each level, Pando can choose a set of packs for a client in logarithmic time. OpenAI states the fetch planning cost as O(log Δ) metadata reads, where Δ is how many ref versions apart the client is from what it wants.

One detail I did not expect: packs are kept at every level and never garbage collected. Old L0 packs still serve clients that are only a little behind, and reading one object from a small L0 pack tends to need less data than reading it from a heavily compressed L4.

In the figure I drew four packs per level so it fits on one screen. OpenAI does not give the actual factor, only that it is geometric.

A worked example, start to finish

Let me put it all together with one small story. The branch names here are mine, the steps are OpenAI's.

1. A Codex agent fixes a bug on a new branch, fix/cache-ttl, and pushes. Pando advertises only trunk, so the agent's Git builds a pack containing just the new commits on top of trunk. Most packs made this way are under 20 KiB.
2. Pando streams the pack into blob storage, validates it in one pass, and in one transaction writes a ref version and swaps the fix/cache-ttl pointer. Nothing else in the repository is locked.
3. At the same moment, a hundred other agents push to a hundred other branches. None of them wait for this one.
4. Three CI jobs fetch fix/cache-ttl. They already have trunk. Pando maps their haves and wants to ref versions and finds that the agent's original pack is all they need.
5. Each CI job gets an empty inline packfile and one link. The link goes to Express, which routes it by consistent hashing, so all three requests hit the same cached copy.
6. Overnight, compaction folds that pack, and thousands like it, into bigger covers. A dev box that has not fetched since yesterday gets a few large packs instead of tens of thousands of small ones.

Nothing in that story requires a server to hold the repository, unpack anything, or build a pack for one client.

What breaks in production

The design is elegant, but OpenAI is honest that the hard part was the edges. Here is what their post says went wrong or needed care:

• Downloading reused packs became the critical path. Because clients now download and index packs directly, OpenAI hit edge cases in concurrent downloads, cleanup after interrupted transfers, and resuming downloads that had already finished. They fixed these in Git and sent the fixes upstream.
• You could not tell where a slow fetch spent its time. They added tracing for packfile URI downloads and byte counts for pack generation, so a slow fetch can be split into "making packs" and "moving bytes".
• Stacked branches cost more bandwidth. Pushing a branch on top of another branch resends everything outside trunk. OpenAI added "explicit haves" in its Git build, so a server can show a branch's current tip separately from the history the client may leave out. That lets a client use --force-with-lease to check a feature branch has not moved while still building its pack against trunk alone.
• Race conditions in negotiation and pack planning. They built deterministic simulation tests to catch them, plus a conformance suite that checks Pando against standard Git behaviour, and they had agents write Lean and TLA+ models to prove properties of the design.
• Moving live traffic. Pando first ran as a read-through and write-through proxy in front of the old system, so both stayed in sync and traffic moved over with no downtime. Clients moved in phases through an internal wrapper around the Git CLI.

And there are two limits that are simply the price of the design. A reader looking at several branches can see them at different points in time. And if your workflow really needs several refs to move together atomically, per-branch ordering will not give you that.

What I take away from this

Three lessons, and none of them are specific to Git.

1. Find the single contended key. Pando's whole write path shrinks to one compare-and-swap on one pointer per branch. When you design a system, ask which key two writers really must fight over, and make everything else independent.
2. Weaken the guarantee nobody uses. A global snapshot of every branch was expensive and, for this workload, almost nobody needed it. Causal consistency per branch was enough.
3. Serve stored answers, do not compute fresh ones. Reusing packs turned a CPU problem into a caching problem, and caching problems scale with disks and network, which are cheap.

OpenAI built this in Rust, went from kick-off to production in 10 weeks, and started serving production traffic in April. They say they are exploring ways to share the technology. Until then, the post and the Git protocol docs are enough to understand every moving part.

Five interview questions, with short answers

1. Why does a three-phase commit design get slow under heavy push traffic?
Every push waits for a majority of replicas to lock, pre-commit and commit the ref. That puts replica disks and several network round trips on every write, and refs are updated one at a time. As traffic grows, the disks saturate and write latency climbs.

2. In Pando, which two pushes can conflict, and how is the conflict resolved?
Only two pushes to the same branch at the same time. The branch pointer is updated with a compare-and-swap inside a transaction. The first push wins; the second finds that the tip moved, and its swap fails.

3. What consistency does Pando give, and what does it give up?
Each branch has a strict order of updates, and you can read your own push immediately. It gives up one point-in-time view across all branches: two clients can see two branches at different versions in opposite combinations.

4. Why does Pando advertise only the trunk branch on a push?
With millions of branches, the full ref list can exceed 100 MiB. Advertising only trunk keeps that list small and makes the client's pack depend on trunk alone, so any reader with trunk can reuse the same pack. The cost is somewhat larger pushes for stacked or repeated branch pushes.

5. How does a client that is a day behind avoid downloading tens of thousands of packs?
Pando compacts packs the way an LSM tree does. Each push is an L0 pack, background workers group them into larger covers level by level, up to L4 today, and the covers act like a skip list so Pando can choose a small set of packs in logarithmic time.

Sources

• OpenAI, "Scaling Git at OpenAI": https://openai.com/index/scaling-git-at-openai/ (every figure, number and quote about Pando in this article)
• Git documentation, protocol version 2, the packfile-uris argument and section: https://git-scm.com/docs/gitprotocol-v2
• Git documentation, Packfile URIs design note: https://github.com/git/git/blob/master/Documentation/technical/packfile-uri.adoc
• OpenAI's Git build: https://github.com/openai/git and its public patch tracker: https://oai-git.openai.chatgpt.site/
• My own trace: GIT_TRACE_PACKET=1 git -c protocol.version=2 fetch --depth=1 https://github.com/git/git.git master with Git 2.39.5 on 3 Nov 2026. It listed 1,013 refs in 109,809 bytes, the server did not advertise packfile-uris, and the request lines in figure 3 are copied from that trace.