Skip to main contentSkip to content
Deep diveCoding agents / Multi-agent / Claude Code / Continuous integration / Verification / Release engineering

What We Learned Building an AI Software Factory

The Foxl team ships dozens of releases a month from an AI software factory where 15 to 20 coding agents work at once. Writing the code was the easy part; knowing it was right was much harder. This is what we learned building the checks.

Foxl TeamUpdated 19 min read

A hand-drawn cardboard parcel with its four flaps open and a large check mark on its side; the box passed its check and holds nothing.
On this page

The Foxl team ships dozens of releases a month. In October there were 66 of them, 22 stable and 44 beta. Every release carries the desktop app for Windows, macOS and Linux, and every stable release also carries the iOS and Android apps and the web app. Each time, all of it has to be checked to work.

In the same month 2,140 pull requests were opened in our repository and 2,040 were merged, close to 70 opened a day. Most of that code was written by coding agents, not by people: 15 to 20 Claude Code sessions running at once on one Mac, each on its own branch. The contribution graph of the GitHub account those sessions commit as shows the change at a glance. In each of the two years before, it counted a few hundred contributions. This year it is past 30,000.

A GitHub profile's 2026 contribution graph from late March on, headed 33,951 contributions in 2026: from April to August most days are light green squares, and dark green squares crowd together from late September into November.

Development where agents plan the work, write the code, test it and review it has come to be called an AI software factory. People decide what to build and make the last call on what ships. Ours runs that way. But once we handed the code to agents in March, the hard part turned out not to be writing it. It was knowing whether the code the agents wrote was actually right, and whether it held up on all five platforms.

This post is what we learned building the checks for that. The short version: every rule we only wrote down for the agents was broken. Most lasted a few days. What held were checks placed in the exact path a mistake had come through, and the next mistake almost always came in through a path that had no check yet.

From one session to twenty

For the first four months we kept one session open all day, and a person read every diff as it came. Nearly everything went straight to main; the repository has a single pull request from all of June. The limit was human attention. One session was about all anyone could follow properly.

In July we pointed Foxl Code, our cloud coding agent, at Foxl's own repository. Each task got a cloud VM of its own, and on July 22 that setup shipped five production releases in one day. Those VMs have no macOS, though, so anything that needed Xcode or the iPhone simulator still had to come back to the Mac.

In August we changed how the work was done: a git worktree for every session and a pull request for every change. Sessions stopped writing over each other's work, so several could run at once, and pull requests went from 598 in August to 946 in September and 2,140 in October.

Commits to main per month in 2026: between 260 and 435 a month from March to July, 1,204 in August when work moved to pull requests, 1,985 in September with the first merge train, 6,400 in October after a coordinator session took over, and 4,969 in the first eleven days of November. Notes beside the bars also mark the five releases on July 22 and running out of memory on November 2.

As the numbers grew, new problems showed up. Sessions edited the same files at the same time, and the Mac ran short of memory. Above all, the first merge train, on September 24, showed that a green CI run does not mean a pull request is right. More on that below.

Where we can check decides where we work

Checking five platforms starts with having machines that can run them. We use three kinds, and each one can check different things.

A matrix of six kinds of check against three machines. All three run unit tests and web builds. Only the lane box runs the desktop app on a clean Windows or Linux install. The lane box and the Mac run an Android emulator. Only the Mac has Xcode, the iPhone simulator and the desktop app on macOS with a real browser. Only the cloud VM can be started from a phone.

Cloud VMs are good for isolated Linux work, and they can be started straight from a phone. When a session needs the desktop app on a fresh Windows or Linux install, or an Android emulator, it gets a throwaway EC2 instance that we call a lane box. And only the Mac can check all of the rest: Xcode, the iPhone simulator, a real Chrome and the desktop app on macOS.

So most sessions ended up on one Mac. It is also a computer we use every day, which means every session shares one machine's memory and works right next to real data.

Starting agents is the solved part

There are plenty of tools for running many agents at once. Every one we looked at makes parallel runs easy, and some let agents talk to each other. Where they differ is where the agents run, and how much of the checking of their results they take on.

Claude Code and Codex

These two are the agents themselves. Claude Code runs background sessions that keep going after the terminal closes, each in its own worktree (claude --bg --worktree), and sessions can message each other or start subagents. There are cloud sessions too, started from the web or a phone. Codex comes as a CLI, a desktop app and IDE extensions, and Codex Web runs tasks in Linux containers in OpenAI's cloud.

Orca

Orca's desktop app: a sidebar of projects and worktrees on two Macs, a coordinator agent's chat in the middle that started two workers and reports what each one did, the workers' own panes on the right, and Orca's phone app over the corner showing the same conversation.

Orca is a desktop app for macOS, Windows and Linux. It runs CLI agents such as Claude Code or Codex in parallel, each in its own worktree, and when you hand it a big task, a coordinator agent splits the work among workers and brings their results back, which is the scene in the screenshot above. Agents can also run on an SSH host or on a cloud VM per worktree, and failing checks and review comments go back to the agent to fix.

herdr

herdr in a terminal window: a sidebar with two workspaces and an agents list that shows one agent idle and Claude working, and two panes, one with an agent's notes on herdr's own code and one running Claude Code.

herdr is a terminal runtime in a single Rust binary. It owns the terminals rather than wrapping the agents, so it works the same with Claude Code, Codex or OpenCode. The terminals keep running in the background when an SSH connection drops, every pane shows whether its agent is working, blocked or idle, and agents can prompt each other through its CLI and socket API.

Kiro Crew

Kiro Crew's crew board: a conductor's goal and its work items grouped under needs a decision, working and finished, each with what its worker reported and the conductor's state; one item is flagged as having no acceptance criteria, and the finished one reads verified against the bar, accepted.

Kiro Crew is a workspace that runs on your own hardware, a Mac, a container or a remote machine, and keeps working between conversations. Sessions, memory and task checkpoints survive restarts, and corrections pile up as lessons that change what it does next. Its crew board, above, shows each item the conductor handed out next to what the worker reported and whether the conductor accepted the result against the bar.

Foxl Code

Foxl Code's Work board in a browser: Needs you and Done are empty, and In flight holds three tasks on foxl-ai/acme-web, each with Claude Code as its agent, one waiting to start and two with the agent working.

Foxl Code is the cloud product we mentioned earlier, in private beta for invited accounts. You describe a task from a browser or your phone, and an agent such as Claude Code works on it in a VM of its own. A larger ask is split into independent subtasks that start at the same time, one pull request each; the board above is one request seconds after it split into three. The reviewer reads the repository's MERGE-REVIEW.md from the default branch, never from the pull request, so a change cannot rewrite the bar it is graded against. Each pull request is still checked on its own, though: to test several of them merged together, you ask the chat for a merge train.

Side by side:

ToolWhere the agents runStrengthsLimits
Claude CodeYour machine, or a cloud session started from the web or a phoneBackground sessions in their own worktrees that list and message each other; subagentsIts messages reach only Claude Code sessions
CodexYour machine, or OpenAI's cloudOne agent in a CLI, a desktop app, IDEs and the cloudCloud tasks run in a Linux container, so Xcode work stays local
OrcaYour machine, an SSH host, or a cloud VM per worktreeAny CLI agent; a coordinator that splits a task among workers; a phone app; failing checks sent backNothing in its description tests several pull requests merged together
herdrYour machine and SSH hostsSits under any agent: detached terminals, working, blocked or idle per pane, agents prompting each otherOwns the terminals, not the work: it adds no review or merge step of its own
Kiro CrewYour Mac, a container, or a remote machineSessions, memory and checkpoints that survive restarts; lessons from corrections; a conductor that accepts work against a barNo channels between workers yet
Foxl CodeA VM per task, in Foxl's cloudStart from your phone; parallel subtasks, one pull request each; a reviewer bound to the repository's merge barPrivate beta; no macOS; pull requests are tested together only in a merge train you ask for

We built our factory on Claude Code's own sessions. It grew out of a single Claude Code session, and the sessions on this Mac could already list and message each other, right next to Xcode and the simulators.

Two things stay the same whichever tool runs the agents. Agents on one machine have to share its memory, and someone has to decide whether what they did is right. The rest of this post is about those two.

Twenty sessions, one Mac's memory

On the afternoon of October 17 nineteen sessions were open. A web build takes about 3.5 GB of memory, and within ten minutes four sessions asked for one. Two ran TypeScript checks, and one booted an iPhone simulator without asking. Eight sessions had changes open in the same CSS file. That afternoon we told the sessions to use less memory, wait their turn, and ask before using it.

The same day, one session became the coordinator. Every other session reads a shared file of who owns which files when it starts, sends the coordinator one line with its task and the files it expects to touch, and gets a scope back. A session that needs a file another session owns sends that session the change to apply. Before anything memory-hungry, such as a build, a test suite or a simulator, it sends HEAVY? with its estimated memory and minutes, and waits for GO. The coordinator reads the Mac's memory pressure before every grant.

Below is the queue this post's first build waited in. A model benchmark and simulator runs were already in line, and the build started only after waiting 50 minutes.

A timeline from 05:00 to 06:30 UTC with one row per step of the shared heavy-work queue: a Notes model benchmark from 05:03 to 05:41, a simulator run, a native build and a simulator run, a screenshot check that stopped the minute it opened and a simulator timing run between 05:41 and 06:07, then this post's two site builds from 06:08 to 06:25, queued since 05:18, with its early dependency copy at 05:57.

The queue's rules live in one shell script, heavy.sh, and they have changed more than once. On October 28 they were too strict: ten jobs waited up to 16 minutes while the Mac had 43% of its memory free, and we took the queue out. On November 2 they were too loose. About 19 sessions were open, several of them running two to five subagents, and the gate counted neither the subagents nor a dev server that had been up for six hours. The Mac ran out of memory, and when the harness cleaned up background shells, the coordinator's watchers died with them.

Today's gate measures the Mac's memory every time it runs. With room, a job starts at once; without it, the job is held; and no more than four heavy jobs run at the same time. The check in front of heavy work is roughly this:

# LOCKS is a directory every session shares; run/<pid> marks a heavy run.
level=$(sysctl -n kern.memorystatus_vm_pressure_level)  # 1 normal, 2 warn, 4 critical
free=$(memory_pressure -Q | awk -F: '/free percentage/ {print int($2)}')
live=0
for f in "$LOCKS"/run/*; do
  [ -e "$f" ] && kill -0 "${f##*/}" 2>/dev/null && live=$((live + 1))
done

hold() {
  [ -e "$LOCKS/STOP" ] || [ "$level" -ge 4 ] ||
  { [ "$level" -ge 2 ] && [ "$live" -ge 2 ]; } ||
  [ "$free" -lt 25 ] || { [ "$free" -lt 30 ] && [ "$live" -ge 1 ]; } ||
  [ "$live" -ge 4 ]
}
# Wait while hold() is true or older tickets fill the free slots (2 at warn, 4 otherwise).
# Then write run/$$, run "$@" as a child, and remove run/$$ when it exits.

Waiting jobs take first-come tickets. Once, with about 30 waiting, whichever job polled first when a slot freed up won, and one job waited 35 minutes behind later arrivals. There are machine-wide limits too, six subagents and one simulator at most, but those two are still only written rules that the gate does not enforce.

Rules that are only written down break

Every incident in this post left a rule behind, written down with its date. Three of them show what happens to a rule that exists only in writing.

The first is real data. Agent sessions launched from the copy of the Foxl desktop app we actually use inherit the app's environment, including a variable that points at the app's database. On October 19 a test suite run from such a shell wrote 305 fake conversations into the real Foxl database. The rule to clear that variable went in the same day, and from October 24 heavy.sh cleared it itself. The mistake kept coming back anyway: a check run directly with npm, one-off scripts, tests that bundle the whole desktop server, each time through a path the wrapper did not cover. Now the desktop server refuses to open real data when it runs under a test runner (foxl/server/paths.ts). Only one-off scripts still rely on the written rule.

The second is a script that spends real money. check-remote-input.mjs sits among dozens of check- scripts that read the repository and fail on what they find, but it starts real tasks on the production Foxl Code service. On September 23 a sweep over every check- script started one. The rule then was to run checks one at a time, by name. On October 18 the script slipped into a list of checks picked with grep, and the charge for its three tasks took our test account below zero. Now it does nothing without a --live flag, and a test in pr-validate keeps it that way.

The third is the push lock. Pushes went through a lock that a session took by creating a directory. On November 2 one session tried to take it by hand, inside a batch of parallel tool calls. Creating the directory failed, because another session held the lock, but the rest of the batch ran anyway: the session pushed, then deleted the other session's lock. A script that takes the lock and releases only its own went in that morning, but a session confined to its own worktree could not use it and kept taking the lock by hand, one step at a time. The same mistake happened again on November 4.

None of these checks ended its kind of mistake. Each one blocked the path the last mistake came through, and so far the next one has always found another.

What a green check proves

All of our CI runs on self-hosted runners, and every pull request has to pass a gate called pr-validate. The question is what that green check proves. Green means the pull request passed the tests on its own, and nothing more.

The first merge train, for v0.7.27 on September 24, showed the gap. Fifty-four pull requests were each green on their own CI and still held four blockers. One lost the last characters typed through an input method; another broke replying from the iPhone lock screen. The same day the Mac froze outright while 19 reviewer agents ran at once. Since then reviewers run at most seven at a time, but that, too, is only a written rule.

So now a train is built from the merged tree of every pull request that is ready, and the merge decision is made on that tree. Reviewer agents each take a group of related pull requests, with a brief that includes this:

Your job is to REFUTE each PR in your group, not to confirm it. Assume every PR body overclaims until you have reproduced its claim. A green pr-validate on the PR proves little: it ran without the other PRs, and it cannot see UX or cost.

A reviewer answers MERGE, MERGE_WITH_FIXES with the patch written out, HOLD, or CLOSE. Fixes land as their own commits on top of the pull request, and a pull request that cannot be fixed is left out by rebuilding the train without it.

Cognition's “Don't Build Multi-Agents” argues against splitting one task among agents that cannot see each other's work, and for handing it to one agent that holds the whole context. We do give each session a whole task. One problem remains: agents are overconfident about their own work. That is why the reviewers are told to refute.

A test the agent writes alongside its feature is especially weak evidence. The test and the code come from the same understanding of the problem, so when the agent misreads a vendor's API or what was asked, both carry the same mistake. Here is what each kind of check can see:

A matrix of six kinds of check against five kinds of failure. A typecheck, a test written in the same commit, a test that fails on the tree before the fix and CI on the merged train each catch only some of the cases where the code is wrong; the last two also expose some tests that test nothing, and CI on the merged train catches changes that break each other. A reviewer who did not write the change catches the code, the test and those interactions, and sometimes an outside system behaving differently. Only asking the destination shows whether a release landed.

So a new test counts only if it fails on the tree before the fix, and the final gate is CI on the merged train. For anything that happens outside the repository, the evidence is in that outside system. After a CLI version bump, the remote Codex and OpenCode terminals in Foxl Code took no typing for ten days while every gate was green. A release workflow once reported success while TestFlight had no build. And on November 3 we found that 34 of 43 screenshot harnesses had been serving pages without the real fonts, so reviewers had been judging letterforms the shipped app never draws.

Ask the destination

A stable release is one workflow run. That one run builds and ships the Windows, Linux, macOS, iOS and Android apps, the web apps and the relay server. We don't take the workflow's word that it succeeded: the checks after a release ask each destination directly.

A script waits until all five platforms have their files before it publishes the release. The Homebrew cask is compared with the published release, because it once served a week-old build for four releases while the workflow that updates it looked fine. The deployed web apps are compared with the release version, because a Worker will answer 200 with a stale bundle. TestFlight is asked through Apple's API.

If you build something similar

Here is what worked for us:

  • Give every agent its own git worktree. On APFS, cp -Rc copies node_modules as a copy-on-write clone that takes almost no disk. Never symlink one node_modules into several worktrees.
  • Make one agent the coordinator, and keep shared state in plain files every agent can read: who owns which files, who is running what, what is frozen.
  • Gate memory-heavy work on measured memory, and count subagents and dev servers along with builds.
  • Read refs by SHA rather than by name, because other sessions keep moving origin/*.

Before building a merge train, you can map its conflicts without a checkout by folding each pull request onto main in turn. In bash:

# Each head as a local ref first: git fetch origin '+refs/pull/*/head:refs/pr/*'
prs=(101 102 103)
acc=$(git rev-parse origin/main)
for pr in "${prs[@]}"; do
  out=$(git merge-tree --write-tree --name-only "$acc" "refs/pr/$pr"); code=$?
  if [ "$code" -eq 0 ]; then
    acc=$(git commit-tree "${out%%$'\n'*}" -p "$acc" -p "refs/pr/$pr" -m "train: #$pr")
  elif [ "$code" -eq 1 ]; then
    echo "#$pr conflicts in:"; echo "$out" | sed -n '2,/^$/p'
  else
    echo "#$pr: merge-tree failed ($code)"; exit "$code"
  fi
done

A file that every pull request appends to, such as a changelog, will show up in nearly every conflict unless you give it a union merge for this command: git -c core.attributesFile=<file>, with CHANGELOG.md merge=union in that file.

What the people do

In this setup we don't write code, and we rarely read a diff. An ask is usually one sentence and a screenshot. What we read is what comes back: the screenshots in each pull request, each train's verdicts, and the report at the end of every release, which lists what shipped, what each check answered and the screenshots from every pull request in it.

Here is one morning of it. On November 3 at 04:59 UTC a pull request came up with 20 screenshots of a new Edit window on the card that posts a review. Its words were in another font, the text box was the browser's own, and three buttons ran down the card one under another. At 05:04 we sent the coordinator two lines: have the author redesign the window in the design system, take the pull request back to draft, and bring it up again. It was a draft twelve seconds later. At 09:30 it came back on the app's own draft editor with 296 screenshots, each taken after the app's fonts had loaded, and it merged at 11:50. The other font was not the pull request's doing: its screenshot harness never served the app's fonts, and checking why, that day, is how we found the 34 harnesses mentioned above.

The pull request on GitHub as it first came up: its Screenshots section, tables of the Edit window on a Telegram message and on a review card; the review card is in a fallback font, with the browser's own text box and Post, Done and Not now stacked down the card.The same pull request on GitHub when it came back: The real bundle: the Edit window, a note that the app's fonts were checked before every shot, and a table of desktop shots in four looks, the review card's Edit window among them.

When something goes wrong, we write the rule that would have caught it into the CLAUDE.md file for that area, with the date and the incident, and then look for the point where a script could enforce it. Agents make reversible decisions from the code and write down what they decided. Anything irreversible, and anything involving money, waits for a person. A pull request that makes Foxl less convenient than the products people compare it with is held, even when its code is right.

The part we haven't solved

The reviewers are agents, but the criteria they apply were written by people, and people still read their verdicts. That reading is now the slowest step between a pull request and a release. So we are writing a criterion for each kind of claim, one that another agent can check without a person: for a logic fix, a test that fails before the fix and passes on the merged train; for a vendor's API, a capture from the real service; for anything a person sees, screenshots of the real build in the real fonts; for a cost, measured request counts; for a release, each destination's answer. Anything irreversible, and anything involving money, stays with people.

We don't know yet how to tell when a criterion is good enough. The plan is to grade reviewers the way tests are graded: count the blockers each one finds that CI missed, and how often a person overturns its verdict. Once people stop overturning a reviewer on one kind of claim, that criterion no longer needs a person. We don't have either number yet.

References and further reading

  1. Claude Code changelogReference
  2. CodexReference
  3. OrcaReference
  4. herdrReference
  5. Kiro CrewReference
  6. Kiro Crew RFC indexReference
  7. Foxl CodeReference
  8. Cognition, Don’t Build Multi-AgentsReference