A Citation Is Not a Measurement

Foxl ships roughly once a day. The machinery around that is not small. Our pull-request validation runs nine jobs; the browser job alone has 170 steps across four shards, and a separate job runs another 122 checks that read our own source and fail on what it says. A release fans out to five platforms and refuses to publish until every one of them has landed.

This post is about the times all of it was green and the product was broken anyway, and about the one shape those failures share. In most of them a signal we owned agreed with us and the system on the other side of a boundary did not. A workflow run concluded success while Apple held no build. A job reported skipped in one second while a package manager served an app from the previous week. A test asserted that a security policy was correct while the app could not start under it.

The most expensive one is the exception, and it is worth stating up front: nothing about it was green. It was red, in the product, in front of users, for fifteen releases, and four separate investigations read that redness and each landed on the wrong suspect. The cause was a vendor behaviour we had quoted accurately from the vendor's own documentation, which was true only under a condition that documentation does not mention.

We have written before about finding failures behind a green board. This is the follow-up we owed it: not the individual bugs, but what a gate can and cannot see, and what we changed once that became the question.

The sentence that cost fifteen releases

Foxl Code runs a coding agent on a remote machine and gives you a real terminal into it. You watch the agent work, and you can type. The typing is the part that matters here: from v0.6.0 through v0.7.2, every keystroke sent to every remote terminal was delivered to a machine that did not have that terminal in it, and answered with a message saying the session was gone.

The remote machines are AWS Bedrock AgentCore runtimes. Each session gets its own micro virtual machine, and you address a session by putting its id in a header. Our own architecture notes explained the design in one sentence: secondary calls carrying the session header are routed to the same container instance. That sentence was not invented. It is AWS's documented behaviour, and here it is, still live on the page as this is published:

MicroVM stickiness: Amazon Bedrock AgentCore uses the session header to
route requests to the same microVM instance. Clients must capture the
session ID returned in the response and include it in all subsequent
requests to ensure session affinity. Without a consistent session ID,
each request may be routed to a new microVM, which may result in
additional latency due to cold starts.

Read that as a promise and you get our design. The only stated failure mode is not sending a consistent session id, which we always did. There is no sentence about what happens when a second request arrives while the first one is still open, and that is the whole problem: an interactive terminal holds its first request open for its entire life, because that open request is how the terminal's output reaches your browser.

We measured it against the production runtime. Three probes, one session id, three different conditions:

With nothing in flight, three sequential calls landed on the same machine three times out of three. With the terminal's own output stream open, every call landed somewhere else: 6f63dab6 became 96d93c3d, 74dc937c became a0b90d74. Once that stream had closed, calls went back to the original machine and could see its checkout.

The machine that answers a concurrent call is not a broken machine. It is a perfectly healthy one that has never heard of your session. Eighteen seconds after a first call had cloned a repository, a second call on the same session id landed elsewhere and cloned it again, into an empty workspace. So the terminal's input route reported the session as missing on every keystroke, and the route that fetches the current screen reported that the session was still starting, forever, because both were talking to a stranger.

In a real browser: 14 frames of terminal output in the first 3 seconds, then zero for the next 148, while the machine itself kept posting snapshots of a screen nobody could reach every 10 seconds.

Four correct fixes, none of which could work

This is the part worth sitting with. Between v0.6.0 and v0.7.2 we shipped four separate fixes for the symptom. A guard against typing before the session was ready. A durable queue so keystrokes could not be dropped while the machine booted. The removal of an overlay that covered the pane. A latch that was supposed to offer you a reconnect button.

Every one of them was a real bug, correctly identified and correctly fixed. Not one of them could have worked, because all four sat downstream of a transport that could not deliver. Two of the investigations ended by writing down that the question was still open: why does the container never register its session? It always registered its session. The probe was reaching a different container.

One of those fixes is in our own blog. The twelfth item in the v0.6.28 gotchas post describes a pane whose failure verdict could never change its mind, and the fix for it. That was accurate as far as it went, and it went nowhere: two days after we published it we found out why none of the preceding work had helped. Correcting a published post is cheaper than leaving a reader with the impression that the terminal was fixed in v0.6.28. It was fixed in v0.7.3, by a different change entirely.

The fix is to stop making the routing decision at all. The container is the client now. It reaches out from inside the machine and long-polls for keystrokes and resizes, scoped to a per-task credential, draining the queue that already existed. There is no second call to route, so there is nothing left to route wrongly.

The rule we wrote down next to it is narrower than "do not trust vendor docs", because that rule is useless. It is this: when a design rests on a documented guarantee, measure the guarantee in the shape your code uses it, and write the condition down beside the citation. Sequentially, concurrently, while streaming, after close. Our documentation traced the claim to a primary source, which is the rule we already had, and the primary source was right. What nobody had established was the scope. The experiment that settled fifteen releases of failure was two calls and one log read.

The tell was two answers that could not both be true

Four investigations read that redness as a client bug, a status oracle, an overlay, a latch. The signal that should have redirected all four was sitting in the symptom the whole time. About one session, at the same moment, three routes reported three things: the screen route said the session was starting, the input route said the session was gone, and the output stream was healthy and delivering. Those cannot all be true of one session.

A disagreement between two routes about one subject is not a bug in either route. It is evidence that they are not looking at the same object. When you get one, stop reading the code that produced the answers and go and ask the thing itself which one it is. That is now the first item on our own debugging checklist, ahead of reading any client code, because four rounds of reading client code is what it replaced.

A flag that defaults to off, and a release that shipped nothing

v0.6.5 was published as a complete five-platform release. Apple held no v0.6.5 build at all. TestFlight was empty.

The iOS release workflow takes an input called upload. It defaults to false. Without it the workflow builds the app, archives it, exports a signed installable package, uploads nothing, and concludes success, with a run title naming the right version. iOS is the one platform that attaches no file to a GitHub release, because its artifact goes to Apple instead. So our five-platform gate had nothing to read except that green conclusion, and it read it.

The sequence is worth spelling out, because no single step in it is a mistake. The release ran, the iOS leg died on an unrelated bug, a human re-dispatched it by hand from our own runbook, the runbook did not mention the flag, the re-run went green, and the gate printed ok ios: TestFlight upload succeeded.

Three changes, all of which are needed and none of which is sufficient alone. The workflow now asks Apple whether the build arrived before its own run is allowed to go green. Its run title carries [upload] or [NO UPLOAD], so a machine that cannot reach Apple can still tell the difference. And the release gate refuses a build-only run, and separately refuses a run that could not ask.

The generalisation is the cheap part to state and the expensive part to believe: a run conclusion is a statement about a run, not about the world. If a workflow's purpose is to put something somewhere, the only evidence that it worked is asking the destination.

A condition that quietly became unsatisfiable

Our macOS package manager tap served a week-old build across four consecutive releases. Every gate was green for all four.

The workflow that updates it was triggered on a successful desktop build whose triggering event was a push. That was true when it was written. Then the platform builds moved to being dispatched by hand rather than run on push, which was a good change for other reasons, and the condition became one that could never again be satisfied. The workflow did not break. It ran, evaluated its condition, and reported skipped in about one second, over and over. A run list full of one-second skips looks exactly like a pipeline with nothing to do.

The repair has three parts, and the important one is the middle. The trigger accepts any successful build. The workflow no longer infers intent from the shape of the run that woke it up: it asks whether the latest published release carries an installer whose digest differs from the one the tap is serving, which is the actual question. And a gate asserts that all of that still holds, because this failure was invisible for a month and nothing in the repository could see it.

Verified while writing this, which is the only reason it appears here as fixed rather than as fixed in principle: the tap names 0.7.18 and the latest published release is v0.7.18.

Never ask a deployed file how big it is

A gate we wrote checked that a deployed asset was the right size by comparing the content-length header against the expected number of bytes. It had only ever run against a local development server. Against the real site it does not merely flake. It fails unconditionally, because the header is not there.

Measured again today, against production, for this post:

200 text/plain  HEAD cl=null  GET cl=null  enc=br    bytes=1836  /robots.txt
200 image/png   HEAD cl=null  GET cl=6529  enc=null  bytes=6529  /apple-touch-icon.png

A HEAD request carries no length at all, on any type we tried, including a PNG that nothing recompresses. And because a modern runtime asks for compression by default, a GET of anything compressible has no length either: the server is streaming it and does not know the final size when the headers go out. So the comparison was 0 === N every single time.

The reading that is true in every case is to fetch the thing and count the bytes you received. These files are kilobytes. There was never a reason to use a proxy for a measurement this cheap, and the proxy was chosen because it looked more efficient.

Our own documentation compiled into the product

This one is our favourite, because the boundary that got crossed is one nobody thinks of as a boundary.

The styling system we use scans source files as text. It does not parse them, does not care whether a fragment is inside a string, a comment, or a sentence in a Markdown file. Anything that looks like a style utility becomes a style rule in the shipped bundle.

Our version of that system changed one piece of syntax between major versions: the older spelling for referring to a raw variable does not produce a usable rule in the newer one. It compiles. It emits a declaration whose value is the bare name of a variable, which every browser discards, so the element silently falls back to its default size. The sidebar we hit this with rendered at full container width and pushed the layout off screen.

We fixed the components. Then we went looking for why the dead rule was still in the shipped stylesheet, and found that the only remaining sources of it were an architecture document, a to-do list, and one comment in a browser test. Three files with no build role whatsoever, each of which happened to spell out the broken syntax while warning against it. A scratch script left next to the app re-added another one the same way.

So the note in our documentation now says to describe that shape and never spell it, and it does not spell it. A warning written in the same vocabulary as the mistake is an instance of the mistake when the tool reads text rather than code.

The file that grew until CI stopped reading it

The same class of problem, in the same week, on the other side of the stack.

Our validation workflow carries its own history in prose. Every step explains what it measured and why, so that the next person to touch it does not re-derive a decision from scratch. That file grew. On 2026-09-11 it stopped running, and the failure looked like this: the run failed in about zero seconds, with no jobs, no check runs, and no log to fetch.

That is the exact shape of a run blocked by an exhausted account allowance, which our own notes tell you to check first, so the first thing we did was check the wrong thing. The one tell that distinguishes them is the run's name: a healthy run is named after the workflow, and a file the platform cannot accept produces a run named after its path.

Two consecutive commits differing only by a comment bracketed it: 511,296 bytes ran and 512,505 bytes did not. That interval rules out the interpretation of the documented limit that everybody reaches for first, since the binary reading of "512 KB" is 524,288 and we were already well past it. The number is 512,000. Neither a workflow linter nor a YAML parser has an opinion about it, so nothing local catches it.

The assertion now rides inside a check that already parsed that file for other reasons, specifically because adding a new step would have cost the bytes it was guarding. It fails over the ceiling and warns within two kilobytes of it. Run right now, on the commit that adds this post:

WARN pr-validate.yml is 510291 bytes, within 1709 of the 512000-byte ceiling

We are 1,709 bytes from turning our own continuous integration off again, and we know it, which is the entire difference between this and last time.

The bill was logs, and the unit was time

A storage allowance alert arrived. The obvious suspects were build artifacts, so we inventoried them: 226 records, 59 live, 272 MB in total. The metered figure said we were holding about 1.67 GB continuously.

Both numbers were right. The artifact inventory cannot see log archives at all, and log archives were 84% of the holding. Measured directly by downloading them: our validation workflow averages 1.16 MB of logs per run, everything else in the repository averages under 90 KB, and with 870 of those runs in the window that one workflow was about two thirds of the entire bill.

Two things about that meter that changed how we read it.

The unit is gigabyte-hours, so the number is an integral, not a snapshot. An allowance of half a gigabyte is 360 gigabyte-hours across a month. Deleting something today refunds nothing that has already accrued; it only lowers the rate from here. A retention window is therefore a direct multiplier on the bill, and the daily rate is the only figure that reproduces between two readings.

The retention setting is not retroactive. Lowering the cap does not touch anything that already exists, because existing logs keep the window they were created under. We lowered it and reclaimed nothing, then deleted 1,427 runs' logs by hand and reclaimed everything. A day inside the new window still served its archive perfectly, which is how we found out the earlier cleanup had a cutoff of its own.

The driver was not our release cadence, which is what we assumed. It was a burst: a normal day is 16 to 35 validation runs, and three consecutive days that month were 238, 118 and 188, because a batch of parallel work landed at once. About 226 MB of logs in one day against 30 on a quiet one. So a retention window has to be chosen to survive a spike inside it rather than to fit the median, which is why ours is four days and not the friendlier seven that sits exactly on the line.

The daily figure has been effectively zero since the change. It is actually lower than the retained bytes predict, and we have not accounted for the last factor of that, so it is written down as a residual rather than as a win.

You can watch a shell start and still not be able to type

One more, because it is the cheapest possible illustration of why an artifact has to be measured on the machine it ships to.

We changed the runtime bundled inside the desktop app, from Node to Bun, to get a build of SQLite that supports full-text search. Conversation search became about three times faster.

It also killed the terminal on every Mac build for four releases, and the failure was completely silent: the library that reads a terminal does it through one particular stream interface, which under the new runtime on macOS delivered zero bytes and never fired its exit callback. Nothing threw. Nothing logged. The function that reports whether terminals are available returned true, and spawning one returned a real process id. We replaced that one read with a different mechanism, and it works on macOS and Linux.

On Windows the terminal renders and cannot be typed into, and no version of that fix can reach it, because Windows reads a terminal through a different operating-system mechanism entirely. Both runtimes deliver 314 bytes of PowerShell banner and prompt, the same count in the same run, so the shell really is up and idle and the reading path works. Then a typed command produces nothing. Retyping produces nothing. A 20,000 line burst produces zero characters. The process has to be killed at 60 seconds. Under Node, every check passes in 10.7 seconds.

Neither half of that was reachable from our own code, and neither was in any release note. It took a real terminal on a real machine per platform, which is now a job that runs on all three whenever that pinned version moves.

So Windows ships the old runtime and falls back to a slower search. That is a worse product than fast search and a much better one than a terminal nobody can type into, and it is written down as a trade with the measurement attached rather than as a to-do.

What we actually changed

Not a new test framework. Five rules, each of which exists because something specific got past us.

Trace every external constant to a primary source, at the constant. Endpoint paths, parameter spellings, header names, field names, status codes, enum values: the URL goes in a comment beside the value. Nobody downstream can tell a fact from a plausible guess otherwise.

Then measure the guarantee in the shape your code uses it. This is the rule the terminal bought, and it is the one we did not have. A citation establishes that a behaviour exists. It says nothing about scope, and scope is where a design lives: sequential or concurrent, streaming or settled, first call or fifth.

If it can be executed, execute it once, for real, before calling it done. One handshake against a live endpoint settles what an hour of reading cannot. This replaced a habit of reasoning carefully about an unexecuted path and concluding it was fine.

An unverified contract must fail loudly. Never fall back, never log and continue. The worst outcome available is not a crash. It is a wrong header producing an error response, a pipeline that logs it and carries on, and then a recording that runs, a meter that bills, a screen that stays green, and no transcript. Silence behind a healthy looking surface is the most expensive thing we know how to build.

A gate may not assert a spelling. Four guards went red on correct code in a single release because they pinned the text of an implementation instead of the property it protected, and one of them, satisfied the cheap way, would have argued us into replacing a constant-time security comparison with a weaker one. A guard whose cheapest repair makes the code worse is worse than no guard.

And one rule about how we report

The rule that took longest to accept is not about code at all.

Writing "unverified" next to a claim does not discharge the obligation to verify it. A disclosure is honest, and it is still not a measurement. A report that ends in three unconfirmed strings has handed the verification to the reader, and the reader is the one person who was relying on it already being done.

The practical version of that: leaving something out is often the right call, and it has to be stated as a choice. Four guessed integrations are not twice the value of two measured ones. They are twice the surface that can fail quietly. So this post names what is still outstanding rather than ending on the fixes.

Windows still shows a warning the first time somebody runs our installer, because a certificate the operating system trusts without being told costs money and has to accumulate reputation. Windows still ships the older bundled runtime and the slower search. The desktop app still hands its own page a startup credential in a way that a stricter window policy would have to permit, so that policy is one directive weaker than we want it. And the number our storage meter reports is lower than the bytes we hold can explain.

Each of those is a decision with a measurement attached, which is the only state we now consider acceptable for something that is not finished. The version of this list we would have written six months ago would have been shorter, and every item missing from it would have been missing because a gate was green.