What We Learned Building an AI Software Factory
The Foxl team ships dozens of releases a month from an AI software factory where 15 to 20 coding agents work at once. Writing the code was the easy part; knowing it was right was much harder. This is what we learned building the checks.

On this page
The Foxl team ships dozens of releases a month. In October there were 66 of them, 22 stable and 44 beta. Every release carries the desktop app for Windows, macOS and Linux, and every stable release also carries the iOS and Android apps and the web app. Each time, all of it has to be checked to work.
In the same month 2,140 pull requests were opened in our repository and 2,040 were merged, close to 70 opened a day. Most of that code was written by coding agents, not by people: 15 to 20 Claude Code sessions running at once on one Mac, each on its own branch. The contribution graph of the GitHub account those sessions commit as shows the change at a glance. In each of the two years before, it counted a few hundred contributions. This year it is past 30,000.

Development where agents plan the work, write the code, test it and review it has come to be called an AI software factory. People decide what to build and make the last call on what ships. Ours runs that way. But once we handed the code to agents in March, the hard part turned out not to be writing it. It was knowing whether the code the agents wrote was actually right, and whether it held up on all five platforms.
This post is what we learned building the checks for that. The short version: every rule we only wrote down for the agents was broken. Most lasted a few days. What held were checks placed in the exact path a mistake had come through, and the next mistake almost always came in through a path that had no check yet.
From one session to twenty
For the first four months we kept one session open all day, and a person read every diff as it came. Nearly everything went straight to main; the repository has a single pull request from all of June. The limit was human attention. One session was about all anyone could follow properly.
In July we pointed Foxl Code, our cloud coding agent, at Foxl's own repository. Each task got a cloud VM of its own, and on July 22 that setup shipped five production releases in one day. Those VMs have no macOS, though, so anything that needed Xcode or the iPhone simulator still had to come back to the Mac.
In August we changed how the work was done: a git worktree for every session and a pull request for every change. Sessions stopped writing over each other's work, so several could run at once, and pull requests went from 598 in August to 946 in September and 2,140 in October.

As the numbers grew, new problems showed up. Sessions edited the same files at the same time, and the Mac ran short of memory. Above all, the first merge train, on September 24, showed that a green CI run does not mean a pull request is right. More on that below.
Where we can check decides where we work
Checking five platforms starts with having machines that can run them. We use three kinds, and each one can check different things.

Cloud VMs are good for isolated Linux work, and they can be started straight from a phone. When a session needs the desktop app on a fresh Windows or Linux install, or an Android emulator, it gets a throwaway EC2 instance that we call a lane box. And only the Mac can check all of the rest: Xcode, the iPhone simulator, a real Chrome and the desktop app on macOS.
So most sessions ended up on one Mac. It is also a computer we use every day, which means every session shares one machine's memory and works right next to real data.
Starting agents is the solved part
There are plenty of tools for running many agents at once. Every one we looked at makes parallel runs easy, and some let agents talk to each other. Where they differ is where the agents run, and how much of the checking of their results they take on.
Claude Code and Codex
These two are the agents themselves. Claude Code runs background sessions that keep going after the terminal closes, each in its own worktree (claude --bg --worktree), and sessions can message each other or start subagents. There are cloud sessions too, started from the web or a phone. Codex comes as a CLI, a desktop app and IDE extensions, and Codex Web runs tasks in Linux containers in OpenAI's cloud.
Orca

Orca is a desktop app for macOS, Windows and Linux. It runs CLI agents such as Claude Code or Codex in parallel, each in its own worktree, and when you hand it a big task, a coordinator agent splits the work among workers and brings their results back, which is the scene in the screenshot above. Agents can also run on an SSH host or on a cloud VM per worktree, and failing checks and review comments go back to the agent to fix.
herdr

herdr is a terminal runtime in a single Rust binary. It owns the terminals rather than wrapping the agents, so it works the same with Claude Code, Codex or OpenCode. The terminals keep running in the background when an SSH connection drops, every pane shows whether its agent is working, blocked or idle, and agents can prompt each other through its CLI and socket API.
Kiro Crew

Kiro Crew is a workspace that runs on your own hardware, a Mac, a container or a remote machine, and keeps working between conversations. Sessions, memory and task checkpoints survive restarts, and corrections pile up as lessons that change what it does next. Its crew board, above, shows each item the conductor handed out next to what the worker reported and whether the conductor accepted the result against the bar.
Foxl Code

Foxl Code is the cloud product we mentioned earlier, in private beta for invited accounts. You describe a task from a browser or your phone, and an agent such as Claude Code works on it in a VM of its own. A larger ask is split into independent subtasks that start at the same time, one pull request each; the board above is one request seconds after it split into three. The reviewer reads the repository's MERGE-REVIEW.md from the default branch, never from the pull request, so a change cannot rewrite the bar it is graded against. Each pull request is still checked on its own, though: to test several of them merged together, you ask the chat for a merge train.
Side by side:
| Tool | Where the agents run | Strengths | Limits |
|---|---|---|---|
| Claude Code | Your machine, or a cloud session started from the web or a phone | Background sessions in their own worktrees that list and message each other; subagents | Its messages reach only Claude Code sessions |
| Codex | Your machine, or OpenAI's cloud | One agent in a CLI, a desktop app, IDEs and the cloud | Cloud tasks run in a Linux container, so Xcode work stays local |
| Orca | Your machine, an SSH host, or a cloud VM per worktree | Any CLI agent; a coordinator that splits a task among workers; a phone app; failing checks sent back | Nothing in its description tests several pull requests merged together |
| herdr | Your machine and SSH hosts | Sits under any agent: detached terminals, working, blocked or idle per pane, agents prompting each other | Owns the terminals, not the work: it adds no review or merge step of its own |
| Kiro Crew | Your Mac, a container, or a remote machine | Sessions, memory and checkpoints that survive restarts; lessons from corrections; a conductor that accepts work against a bar | No channels between workers yet |
| Foxl Code | A VM per task, in Foxl's cloud | Start from your phone; parallel subtasks, one pull request each; a reviewer bound to the repository's merge bar | Private beta; no macOS; pull requests are tested together only in a merge train you ask for |
We built our factory on Claude Code's own sessions. It grew out of a single Claude Code session, and the sessions on this Mac could already list and message each other, right next to Xcode and the simulators.
Two things stay the same whichever tool runs the agents. Agents on one machine have to share its memory, and someone has to decide whether what they did is right. The rest of this post is about those two.
Twenty sessions, one Mac's memory
On the afternoon of October 17 nineteen sessions were open. A web build takes about 3.5 GB of memory, and within ten minutes four sessions asked for one. Two ran TypeScript checks, and one booted an iPhone simulator without asking. Eight sessions had changes open in the same CSS file. That afternoon we told the sessions to use less memory, wait their turn, and ask before using it.
The same day, one session became the coordinator. Every other session reads a shared file of who owns which files when it starts, sends the coordinator one line with its task and the files it expects to touch, and gets a scope back. A session that needs a file another session owns sends that session the change to apply. Before anything memory-hungry, such as a build, a test suite or a simulator, it sends HEAVY? with its estimated memory and minutes, and waits for GO. The coordinator reads the Mac's memory pressure before every grant.
Below is the queue this post's first build waited in. A model benchmark and simulator runs were already in line, and the build started only after waiting 50 minutes.

The queue's rules live in one shell script, heavy.sh, and they have changed more than once. On October 28 they were too strict: ten jobs waited up to 16 minutes while the Mac had 43% of its memory free, and we took the queue out. On November 2 they were too loose. About 19 sessions were open, several of them running two to five subagents, and the gate counted neither the subagents nor a dev server that had been up for six hours. The Mac ran out of memory, and when the harness cleaned up background shells, the coordinator's watchers died with them.
Today's gate measures the Mac's memory every time it runs. With room, a job starts at once; without it, the job is held; and no more than four heavy jobs run at the same time. The check in front of heavy work is roughly this:
# LOCKS is a directory every session shares; run/<pid> marks a heavy run.
level=$(sysctl -n kern.memorystatus_vm_pressure_level) # 1 normal, 2 warn, 4 critical
free=$(memory_pressure -Q | awk -F: '/free percentage/ {print int($2)}')
live=0
for f in "$LOCKS"/run/*; do
[ -e "$f" ] && kill -0 "${f##*/}" 2>/dev/null && live=$((live + 1))
done
hold() {
[ -e "$LOCKS/STOP" ] || [ "$level" -ge 4 ] ||
{ [ "$level" -ge 2 ] && [ "$live" -ge 2 ]; } ||
[ "$free" -lt 25 ] || { [ "$free" -lt 30 ] && [ "$live" -ge 1 ]; } ||
[ "$live" -ge 4 ]
}
# Wait while hold() is true or older tickets fill the free slots (2 at warn, 4 otherwise).
# Then write run/$$, run "$@" as a child, and remove run/$$ when it exits.Waiting jobs take first-come tickets. Once, with about 30 waiting, whichever job polled first when a slot freed up won, and one job waited 35 minutes behind later arrivals. There are machine-wide limits too, six subagents and one simulator at most, but those two are still only written rules that the gate does not enforce.
Rules that are only written down break
Every incident in this post left a rule behind, written down with its date. Three of them show what happens to a rule that exists only in writing.
The first is real data. Agent sessions launched from the copy of the Foxl desktop app we actually use inherit the app's environment, including a variable that points at the app's database. On October 19 a test suite run from such a shell wrote 305 fake conversations into the real Foxl database. The rule to clear that variable went in the same day, and from October 24 heavy.sh cleared it itself. The mistake kept coming back anyway: a check run directly with npm, one-off scripts, tests that bundle the whole desktop server, each time through a path the wrapper did not cover. Now the desktop server refuses to open real data when it runs under a test runner (foxl/server/paths.ts). Only one-off scripts still rely on the written rule.
The second is a script that spends real money. check-remote-input.mjs sits among dozens of check- scripts that read the repository and fail on what they find, but it starts real tasks on the production Foxl Code service. On September 23 a sweep over every check- script started one. The rule then was to run checks one at a time, by name. On October 18 the script slipped into a list of checks picked with grep, and the charge for its three tasks took our test account below zero. Now it does nothing without a --live flag, and a test in pr-validate keeps it that way.
The third is the push lock. Pushes went through a lock that a session took by creating a directory. On November 2 one session tried to take it by hand, inside a batch of parallel tool calls. Creating the directory failed, because another session held the lock, but the rest of the batch ran anyway: the session pushed, then deleted the other session's lock. A script that takes the lock and releases only its own went in that morning, but a session confined to its own worktree could not use it and kept taking the lock by hand, one step at a time. The same mistake happened again on November 4.
None of these checks ended its kind of mistake. Each one blocked the path the last mistake came through, and so far the next one has always found another.
What a green check proves
All of our CI runs on self-hosted runners, and every pull request has to pass a gate called pr-validate. The question is what that green check proves. Green means the pull request passed the tests on its own, and nothing more.
The first merge train, for v0.7.27 on September 24, showed the gap. Fifty-four pull requests were each green on their own CI and still held four blockers. One lost the last characters typed through an input method; another broke replying from the iPhone lock screen. The same day the Mac froze outright while 19 reviewer agents ran at once. Since then reviewers run at most seven at a time, but that, too, is only a written rule.
So now a train is built from the merged tree of every pull request that is ready, and the merge decision is made on that tree. Reviewer agents each take a group of related pull requests, with a brief that includes this:
Your job is to REFUTE each PR in your group, not to confirm it. Assume every PR body overclaims until you have reproduced its claim. A green pr-validate on the PR proves little: it ran without the other PRs, and it cannot see UX or cost.
A reviewer answers MERGE, MERGE_WITH_FIXES with the patch written out, HOLD, or CLOSE. Fixes land as their own commits on top of the pull request, and a pull request that cannot be fixed is left out by rebuilding the train without it.
Cognition's “Don't Build Multi-Agents” argues against splitting one task among agents that cannot see each other's work, and for handing it to one agent that holds the whole context. We do give each session a whole task. One problem remains: agents are overconfident about their own work. That is why the reviewers are told to refute.
A test the agent writes alongside its feature is especially weak evidence. The test and the code come from the same understanding of the problem, so when the agent misreads a vendor's API or what was asked, both carry the same mistake. Here is what each kind of check can see:

So a new test counts only if it fails on the tree before the fix, and the final gate is CI on the merged train. For anything that happens outside the repository, the evidence is in that outside system. After a CLI version bump, the remote Codex and OpenCode terminals in Foxl Code took no typing for ten days while every gate was green. A release workflow once reported success while TestFlight had no build. And on November 3 we found that 34 of 43 screenshot harnesses had been serving pages without the real fonts, so reviewers had been judging letterforms the shipped app never draws.
Ask the destination
A stable release is one workflow run. That one run builds and ships the Windows, Linux, macOS, iOS and Android apps, the web apps and the relay server. We don't take the workflow's word that it succeeded: the checks after a release ask each destination directly.
A script waits until all five platforms have their files before it publishes the release. The Homebrew cask is compared with the published release, because it once served a week-old build for four releases while the workflow that updates it looked fine. The deployed web apps are compared with the release version, because a Worker will answer 200 with a stale bundle. TestFlight is asked through Apple's API.
If you build something similar
Here is what worked for us:
- Give every agent its own git worktree. On APFS,
cp -Rccopiesnode_modulesas a copy-on-write clone that takes almost no disk. Never symlink onenode_modulesinto several worktrees. - Make one agent the coordinator, and keep shared state in plain files every agent can read: who owns which files, who is running what, what is frozen.
- Gate memory-heavy work on measured memory, and count subagents and dev servers along with builds.
- Read refs by SHA rather than by name, because other sessions keep moving
origin/*.
Before building a merge train, you can map its conflicts without a checkout by folding each pull request onto main in turn. In bash:
# Each head as a local ref first: git fetch origin '+refs/pull/*/head:refs/pr/*'
prs=(101 102 103)
acc=$(git rev-parse origin/main)
for pr in "${prs[@]}"; do
out=$(git merge-tree --write-tree --name-only "$acc" "refs/pr/$pr"); code=$?
if [ "$code" -eq 0 ]; then
acc=$(git commit-tree "${out%%$'\n'*}" -p "$acc" -p "refs/pr/$pr" -m "train: #$pr")
elif [ "$code" -eq 1 ]; then
echo "#$pr conflicts in:"; echo "$out" | sed -n '2,/^$/p'
else
echo "#$pr: merge-tree failed ($code)"; exit "$code"
fi
doneA file that every pull request appends to, such as a changelog, will show up in nearly every conflict unless you give it a union merge for this command: git -c core.attributesFile=<file>, with CHANGELOG.md merge=union in that file.
What the people do
In this setup we don't write code, and we rarely read a diff. An ask is usually one sentence and a screenshot. What we read is what comes back: the screenshots in each pull request, each train's verdicts, and the report at the end of every release, which lists what shipped, what each check answered and the screenshots from every pull request in it.
Here is one morning of it. On November 3 at 04:59 UTC a pull request came up with 20 screenshots of a new Edit window on the card that posts a review. Its words were in another font, the text box was the browser's own, and three buttons ran down the card one under another. At 05:04 we sent the coordinator two lines: have the author redesign the window in the design system, take the pull request back to draft, and bring it up again. It was a draft twelve seconds later. At 09:30 it came back on the app's own draft editor with 296 screenshots, each taken after the app's fonts had loaded, and it merged at 11:50. The other font was not the pull request's doing: its screenshot harness never served the app's fonts, and checking why, that day, is how we found the 34 harnesses mentioned above.


When something goes wrong, we write the rule that would have caught it into the CLAUDE.md file for that area, with the date and the incident, and then look for the point where a script could enforce it. Agents make reversible decisions from the code and write down what they decided. Anything irreversible, and anything involving money, waits for a person. A pull request that makes Foxl less convenient than the products people compare it with is held, even when its code is right.
The part we haven't solved
The reviewers are agents, but the criteria they apply were written by people, and people still read their verdicts. That reading is now the slowest step between a pull request and a release. So we are writing a criterion for each kind of claim, one that another agent can check without a person: for a logic fix, a test that fails before the fix and passes on the merged train; for a vendor's API, a capture from the real service; for anything a person sees, screenshots of the real build in the real fonts; for a cost, measured request counts; for a release, each destination's answer. Anything irreversible, and anything involving money, stays with people.
We don't know yet how to tell when a criterion is good enough. The plan is to grade reviewers the way tests are graded: count the blockers each one finds that CI missed, and how often a person overturns its verdict. Once people stop overturning a reviewer on one kind of claim, that criterion no longer needs a person. We don't have either number yet.
References and further reading
- Claude Code changelogReference
- CodexReference
- OrcaReference
- herdrReference
- Kiro CrewReference
- Kiro Crew RFC indexReference
- Foxl CodeReference
- Cognition, Don’t Build Multi-AgentsReference