In chapter 19 I told the story of our first trains: a small script that lands pull requests one by one, without a model token. It held for a week. From 1 to 3 November it broke in four ways I had not seen coming, and I rebuilt it. This chapter shows the new version with drawings of trains, because that is how I see it in my head.
One note before we start. Our products go by code names here: Product A, Product B, Product C, and the platform. What matters is the mechanism, not the names.
Four moments of the weekend
Green PRs that never arrived
On 1 November, PRs had been waiting for hours, every check green, and nothing carried them. The trains of the time had no order: several fought over the same repository, one PR had two, and some had been running for two days on a blocked PR, hiding the ready ones behind it.
A red main that blocks everyone
When a repository's main branch turns red, every open PR turns red with it, even the ones that have nothing to do with it. On the evening of 2 November, on the platform, three PRs were red for the same reason: a weekly automatic update had pushed a size budget over the limit on main. None of the three could pass, and re-running them was pointless.
The repair our own tool refused
The session that fixed main tried to open its PR. The opening tool refused it: it touched the same files as the PRs it was about to unblock. On a normal day that is a good rule, it keeps two PRs from stepping on each other. Here it blocked the one PR that could get everything moving again.
A session that died without a sound
On 2 November around 9:45, Montréal time, one of our sessions stopped dead on the model's weekly usage limit. Nothing rang. Its record stayed open for hours, the next handoff said no session was running, and its cost was never recorded.
The rails
The new version fits in one picture. Each product has its rail. On each rail, one ordered queue: the oldest PR in front. At most three PRs are active per repository; the others wait outside the window. The station is main.
- queued
- leaving, CI green
- arrived on main
- A session opens its PR, then stops. The opening tool puts the PR in its product's queue, behind the older ones.
- At most three PRs are active per repository. The others wait, and come in as one arrives.
- Every three minutes the conductor looks at the head PR: up to date with main, checks green. It approves it and puts it in the merge queue.
- The merge queue runs the checks again on the merged result. The PR arrives on main as a single commit, and its branch is deleted.
- The next one leaves. The three rails run at the same time, and none waits for another.
The conductor does not judge the code. Review happens before, by the session that wrote the PR. The conductor does the platform work: wait, update, approve, enqueue. Above all, it never stays stuck behind a PR that will not leave.
The worst cases
This is where the new version changes everything. Before, a red PR at the head stopped the train, and everything behind it waited. Now the red PR is taken off the rail, put on a siding with its reason written down, and the next one leaves.
A red main has its own rule. If main is red, a green PR proves nothing: its checks ran on a broken base. The rail is frozen. Only one train may pass, the repair train, a PR whose title starts with fix(main). It goes ahead of everyone, outside the cap, and since 3 November the opening tool no longer refuses it when it touches the files of the PRs it unblocks: it notes it, and those PRs will update behind it.
- queued
- leaving, CI green
- arrived on main
- red, on the siding
- fix(main) repair train
- stacked on the PR ahead
- Product A: PR 2 is red. The conductor moves it to the siding with its reason, the red check and its link, and PR 3 leaves. PR 2 comes back into the queue as soon as a new commit arrives.
- Product B: main is red, the rail is frozen. PRs 5 and 6 wait, even green, and the conductor opens an issue so the red has an owner.
- Only the repair train leaves: the fix(main) PR goes first, outside the cap, even if it touches the files of the waiting PRs.
- main turns green again. The frozen PRs come back into the queue by themselves, update onto the repair and leave in order.
- Product C: PR 8 is built on PR 7's branch. Before putting PR 7 in the merge queue, the conductor retargets PR 8 to main, so deleting the branch at merge does not close it.
The last case comes from a real loss. Before this rule, two stacked PRs closed on their own: their base had merged, its branch had been deleted, and GitHub closes a PR whose base no longer exists. The work was there, the PR was not.
Who does what, and what costs
I work with agents. A pilot starts the sessions: one per product, one for the platform, one for security. Each session writes the code, runs the checks, opens its PR, writes its handoff and stops. Those two are models: they cost tokens.
Everything else is a script. The queue and the conductor run as soon as a PR waits. The controller, which chases duplicate trains and heads stuck for an hour, and the watchdog, which reads and writes its alerts, run every fifteen minutes. None of them costs a model token. Two GitHub Apps open, approve and enqueue. No human account works in the background.
- script, zero tokens
- model, costs tokens
- The pilot starts a session. The session writes, tests, opens its PR, writes its handoff and stops. It never waits for CI.
- From there on, no model at all: the queue files the PR, the conductor carries it, the two apps approve it and enqueue it.
- Every fifteen minutes the watchdog reads the state of the repositories and writes an alert line: a red PR with no owner, a main red for thirty minutes, a frozen rail with green PRs, a dead session.
- A model is woken only to repair what the watchdog flagged.
- When the session ends, its cost is read from its transcript, not estimated.
The dead session of 2 November left its guard behind. A session now records, when it starts, its process and its transcript. The watchdog calls it dead when the process is gone or when the transcript ends on an error such as a usage limit, and stalled when it has been silent for more than an hour. It is closed cleanly, and its cost is read from what it really did.
What the logs say
Each queue writes one line per event. Between 1 November at 17:43 (Montréal time), when the per-product queues started, and midnight on 3 November, a little over 30 hours, the trains brought 267 PRs to main. They set 196 aside, each time with a reason:
| Reason | Times |
|---|---|
| a red check | 141 |
| left the merge queue | 15 |
| a required check that never ran | 13 |
| a conflict | 12 |
| main red, rail frozen | 10 |
| already carried by another train | 4 |
| a train over 24 hours | 1 |
A PR set aside is not a lost PR. It comes back when a new commit is pushed, or when main turns green again. What this table shows is 196 times when, before, the whole rail would have stopped.
The watchdog wrote 359 alert lines since the evening of 1 November. The most frequent: 122 times a red or conflicting PR with no owner, 56 times a main red for more than thirty minutes, 38 times a frozen rail with green PRs waiting. None of them cost a token. They only tell a session where to look.
Over the same 30 hours, 49 model sessions were recorded. Until 3 November their token cost was an estimate typed by the session itself; since then it is measured. So I give no “tokens saved”: I have no honest measure of what a session waiting on CI would have cost. What I know is that none waits any more.
What is left
The frozen rail says main is red, it does not repair it: a session still has to write the fix(main). The cap of three PRs per repository sometimes holds a good PR behind two red ones, and the answer is still human: land them or close them. And the watchdog warns, it does not decide. That is on purpose: a script that costs nothing can watch all the time, but repairing is still somebody's job.
And the repair train itself can be left standing at the station. On the morning of 3 November, on Product A, the fix(main) PR stayed frozen with the others for about two hours: the conductor recognises the repair train by a label, and the opening tool had not set it. It had to be added by hand. The next guard: the tool sets the label as soon as the title starts with fix(main), and a test checks it.
A firefighter, and the real bottleneck
On the evening of 3 November, I added a firefighter. Its rule is simple. A repository with its three PRs open, at least one of them red, is on fire: nobody can open a new PR there while the red one stays. Every five minutes, a script goes over each repository. When it finds a fire, it puts a label on the red PR, leaves a comment, sends me a notification and writes the brief for the session that will put it out. A fire comes before any new work. If nobody takes it within thirty minutes, it becomes urgent.
Before saying whether it helps, I wanted to know where we started from. I went back over the fourteen days before it in GitHub: 38 repositories, 3,871 PRs. With the three-PR rule, our repositories spent 1,487 hours blocked, 11.6% of their time. 27 of the 38 were blocked at least once.
On its first evening it found three fires. The first one left the fire state in two minutes, and the real repair landed about twenty minutes after the alarm. The second went out by itself after thirty minutes, without anyone taking it. The third was taken after twenty-five minutes, and a session was still working on it when I wrote this.
What it has gained us, I do not know yet. Three fires in one evening is not a measurement. I set myself a minimum before comparing: seven days and ten fires. All I have is an anecdote, and a modest one: twenty minutes for the first fire is about as long as an ordinary block before it. Half of them already lasted less than a quarter of an hour.
There was also a false alarm on my side. During the first night, the firefighter's log went quiet, and I thought its alarm was broken. It was running, it just had nothing new to say. It now writes the time of its last pass, and the next session sees at once whether it has really stopped.
The chouf
The next day, I looked at what those firefighters cost. Each one had used between 110,000 and 135,000 tokens, and a good part of it did not go into putting the fire out: the session stayed awake to wait for the CI to pass and for the train to take the PRs. A firefighter waiting outside the station still costs something.
So I split the two roles. The one who watches we call the chouf. In Arabic dialect, chouf means "look"; the chouf is the one who keeps watch. It is the script that passes every five minutes. It has no AI, it costs nothing, and it does one thing: it says when it is time to wake the firefighter.
How this was made
- Model
- GPT Image 2.5 Sunburst · Runway terms of use (output owned by the customer)
- Engine
- Runway API (gpt_image_2_5_sunburst), text to image with 4 reference pictures (three pale covers of the series for the style, the earlier version of the picture for the scene and the characters); quality medium, ratio 1920:1088; 1 take(s), the best kept
- Machine
- Runway cloud
- Through
- Runway
- Prompt
Use the first three reference pictures only for their style: a light, airy hand-drawn illustration on white paper, lots of unpainted white paper left around and inside the scene, soft pale transparent watercolor washes (pale blue, pale indigo, peach, light orange, cream), a light bright sky or none, delicate thin pen-and-ink linework of varying weight with some broken lines, gentle muted pastel colour, a few small touches of stronger colour only on the characters and one lamp. No dark night, no heavy saturated shadows, no photographic detail, no digital smoothness. Absolutely not a cartoon: no big eyes, no cel shading, no flat vector fills, no glossy gradients, no clothes on the animals. Birds are small, drawn naturalistically with a few ink strokes. No people, no text, no letters, no numbers, no logos, no watermark. The last reference picture shows the scene and the character to keep exactly (the same small orange-brown kestrel perched on the back of the same weathered wooden folding chair at the edge of the tracks, the same lantern on the ground, the same station with lit windows far away): redraw it in the light pastel style of the first three, a pale winter dusk, snow left as white paper.
- Size
- 1920×1088
- Time
- not recorded
- Cost
- not recorded
- Sovkora run
- not recorded
- Note
- Hand-drawn ink and watercolor on white paper, light and pastel like chapters 1 to 20, redrawn after a cartoon look and a darker one (krizaka-internal#2455). No text, no logo; looked at in full size before use.
The rule we are writing is simple. The chouf wakes the firefighter at most once an hour, for all fires at once, and only when a fire has lasted more than a quarter of an hour: many reds go out by themselves before that. The firefighter wakes with a light model and a token cap, and moves to the strongest model only when the same PR is red a second time. It repairs, puts the PR on the train, and goes back to sleep. It no longer waits for the landing: the train and the chouf take care of that. If it turns red again, the next wake deals with it.
How this was made
- Model
- GPT Image 2.5 Sunburst · Runway terms of use (output owned by the customer)
- Engine
- Runway API (gpt_image_2_5_sunburst), text to image with 4 reference pictures (three pale covers of the series for the style, the earlier version of the picture for the scene and the characters); quality medium, ratio 1920:1088; 2 take(s), the best kept
- Machine
- Runway cloud
- Through
- Runway
- Prompt
Use the first three reference pictures only for their style: a light, airy hand-drawn illustration on white paper, lots of unpainted white paper left around and inside the scene, soft pale transparent watercolor washes (pale blue, pale indigo, peach, light orange, cream), a light bright sky or none, delicate thin pen-and-ink linework of varying weight with some broken lines, gentle muted pastel colour, a few small touches of stronger colour only on the characters and one lamp. No dark night, no heavy saturated shadows, no photographic detail, no digital smoothness. Absolutely not a cartoon: no big eyes, no cel shading, no flat vector fills, no glossy gradients, no clothes on the animals. Birds are small, drawn naturalistically with a few ink strokes. No people, no text, no letters, no numbers, no logos, no watermark. The last reference picture shows the scene and the character to keep exactly (the same small red duck with round glasses on the siding, the same little bucket at its feet, the same thin jet of water from a hose coil toward a small fire and steam on the roof of the same red wagon): redraw it in the light pastel style of the first three, a pale winter dusk, snow left as white paper.
- Size
- 1920×1088
- Time
- not recorded
- Cost
- not recorded
- Sovkora run
- not recorded
- Note
- Hand-drawn ink and watercolor on white paper, light and pastel like chapters 1 to 20, redrawn after a cartoon look and a darker one (krizaka-internal#2455). No text, no logo; looked at in full size before use.
The first two firefighters have already gone back to sleep that way. The rest, the hourly wake and the cap, is in the queue: I will say what it changes once I have measured it.
The real bottleneck
When I looked for where those 1,487 hours came from, I expected a CI that was too slow. That is not it. A red PR turns green again in twenty-six minutes, median. The blocked time comes from somewhere else.
The bottleneck is not the PR that is red right now. It is the forgotten red: a PR that will never leave, but keeps its place among the three. Nobody is in charge of it, so it sits there for days, and then someone closes it. The firefighter is quick on a fresh fire. For the one that has been smouldering for a week, it does nothing.
A second figure surprised me. On top of those blocked hours, our repositories spent 1,178 hours with their three PRs open, all green, waiting for the train to take them. That is not a fire, but it is a place taken too.
I keep what is measured apart from what is inferred. The outcome of each PR, how long it had been red, who opened it: that is measured. That 28% of the blocked hours come from one failure shared by several PRs, like a red main or a broken shared CI, I infer: the same check was red on two PRs at the same moment. And GitHub only keeps the checks of a PR's last commit, so some figures are lower bounds. The blocked hours are solid.
We will go after the forgotten red, but not right away: other subjects come first. The idea is written down. A PR red for more than a day gets fixed that day, or closed with one line that says why. Meanwhile, the firefighter stays.
