Skip to the content
Krizaka Blog

19

All aboard: how our pull requests catch the train

Oct 28–30, 20268 min read

What I faced, 140 open PRs and sessions waiting on CI; What we built, trains that land, 5 PRs per repository at most, a guard per failure

A watercolour canal at dawn under a pale sun: brick houses on the left, street lamps along the quay on the right, and a small train crossing the stone bridge, reflected in the still water

This week I understood that my agents were not short of speed. They were short of a platform. Every session finished its work, opened a pull request, then waited for CI to pass, for the merge queue to accept it, for the branch to be up to date. While it waited, the session kept costing, and the pile kept growing.

This chapter tells the small system that got things moving again. I call it the train, because that is exactly what it is: wagons leaving one behind the other, never overtaking.

The problem: a pile

We work with five lanes: one session per application (Yomizaka, Akinaza, Orazaka), one for the platform, one for security. Each opens its PRs. Thursday evening there were about a hundred open PRs across the organisation. At 2 a.m. on Friday there were 140, and an hour later Akinaza's repository carried 24 of them alone.

The bottleneck was not the code. It was the capacity of CI and of the merge queue: one PR passes, another waits behind it, and if the branch moves in between, everything starts over. A session that waits is a session burning budget to watch a circle turn.

The train

The train is a small script: under eighty lines of shell, with no model token at all. It takes PRs that have already been read, in order, and for each one:

  1. WaitThe previous PR is merged. Not before.
  2. UpdateThe branch catches up with main. If GitHub changed its base, it follows.
  3. ApproveOnce per version of the branch, with my account.
  4. EnqueueThe PR enters the merge queue.
  5. Stop at the first redA red check, a conflict, an ejection: the train stops and says why. It never merges around a failure.

The important point: the review happens before, by the session that wrote the PR, not by the train. The train judges nothing. It does the platform work, the kind an agent does badly because waiting for it is expensive.

A pull request's journey, from its branch to production: the session writes and reviews, the train only carries.
  1. Branch (one session)A worktree from main, one issue, the test first. The session reads its own diff before it opens the PR.
  2. Checks (CI on the PR)Lint, tests, build, code analysis. A red check stops everything here.
  3. The train (no model token)It waits for the previous PR, updates the branch, approves once per version, enqueues. It stops at the first red.
  4. Merge queue (CI again)The checks run once more, on the PR merged with what is ahead of it in the queue.
  5. main (one commit)The PR becomes one commit on main, and its branch is deleted.
  6. Deploy (then checked)The site is built and deployed; a job then checks that the live site carries that commit.

What broke, and the guard that came out of it

The train did not work the first time. Each failure became a rule, and each rule has its test.

  • duplicates

    What happened
    The same train started four times by different sessions.
    The guard
    One train per PR The script refuses to start if a live train already carries that PR.
  • approvals

    What happened
    An approval every 90 seconds: GitHub rate-limited us.
    The guard
    Approve once per version The train remembers the version it already approved.
  • ejection

    What happened
    A PR ejected from the queue was put back seven times.
    The guard
    Stop at the first ejection An ejection is a signal, not an incident to replay.
  • quota

    What happened
    Twenty-seven trains polling the API every 90 seconds emptied the shared quota.
    The guard
    REST, every 3 minutes Reads go through the REST API; only enqueueing stays a mutation.
  • silence

    What happened
    A rewrite of the script made every train spin in place for an hour, without an error.
    The guard
    A test of the reading functions The state of a PR is now tested before each change of the script.
  • version bot

    What happened
    A bot opened one update PR per merge: 116 duplicates.
    The guard
    One version PR per repository A new organisation rule joins the open bulk PR; it does not open another.
  • timeout

    What happened
    The merge queue's timeout was shorter than CI: ejections for nothing.
    The guard
    A timeout longer than CI The queue no longer judges a healthy CI too slow.
  • skipped check

    What happened
    The name of a skipped job was not evaluated: a PR stayed stuck on a phantom check.
    The guard
    Literal names Required checks carry their exact name.

Every row of that table is a real incident from this week. I keep them together because their common shape is what matters: a failure is not over when it is repaired, it is over when it cannot come back without a test turning red.

The rules that came out of it

The train also changed our working rules, which matter more than the script:

  • At most 5 open PRs per repository. At the limit, nobody opens a new one: we merge first.
  • Finish before starting. A session's first job is its open PRs, oldest first: land them, or close them saying why.
  • One bulk PR per repository at a time. A new organisation rule joins the open bulk PR instead of opening another.
  • A landing order. Two PRs are stacked only when the first is already in the queue.

Rules that live in a document get forgotten, so each one became a guard that refuses, with its test:

  1. CommitNo co-author lineA commit or a PR body that names an AI as co-author is refused before it leaves the machine: our app is the only author.
  2. OpenFive PRs per repositoryOpening a sixth is refused, version bots included. At the limit, we land first.
  3. SweepNo orphanA script lists the open PRs that no train carries: green, it gets a train; red, it goes back to its lane by name.
  4. RedA stale red is refreshed onceA red on a branch behind main is often already fixed there: the train updates the branch once, then judges the new run.
What stops a train, or never lets it leave: four guards written this week, each with its test.

A pile of open PRs tells me one thing: the order of the work was wrong. The train relieves the symptom, the rule of five treats the cause.

What it changes

The figures I can give are the ones I can read back. I counted them from the GitHub API: for each hour, every pull request of the organisation that was created before it and not yet closed. Akinaza's repository went from 24 open PRs at 3 a.m. on Friday to 4 by the evening. Across the organisation, the pile went from 140 to 50 as I write, and 317 PRs were merged on Friday.

Open pull requests across the organisation, every six hours from Thursday morning to Friday night: the pile peaks at 140, then the trains bring it down.

Counted from the GitHub API on 30 October: pull requests created before each hour and not closed yet, in the repositories the organisation holds today; Montréal time.

Bar chart, 8 (Thu 8 a.m. – Fri 10:30 p.m.). Open PRs: from 119 to 50, lowest 50 (Fri 10:30 p.m.), highest 140 (Fri 2 a.m.), down 58%.

Open pull requests across the organisation, every six hours from Thursday morning to Friday night: the pile peaks at 140, then the trains bring it down. — data
ItemOpen PRs
Thu 8 a.m.119
Thu 2 p.m.89
Thu 8 p.m.107
Fri 2 a.m.140
Fri 8 a.m.77
Fri 2 p.m.66
Fri 8 p.m.66
Fri 10:30 p.m.50
Open pull requests across the organisation, every six hours from Thursday morning to Friday night: the pile peaks at 140, then the trains bring it down.

The peak at night is not a mystery: that is when the sessions finish their work and open their PRs, while the trains still wait for CI. The fall in the morning is the trains, landing one PR after the other.

I give no "time saved": I did not measure it, and I would rather stay quiet than invent a number. What I see is that sessions no longer spend their time waiting, and that we stopped stepping on each other.

What is left

The train depends on CI, and CI is still slow: the next chapter tells how it grew, and how we measured it. The ceiling of 5 PRs per repository is a rule that a human or an agent can forget: we still check it by hand at each checkpoint. And when the API quota is empty, everything stops, which has already happened once.

My motto, and economy mode

This week I also put one sentence at the head of all our agents' rules: "The flow keeps running, the deadline is ours: it may move, the quality never does." The flow is the trains and the small PRs. The deadline is ours to set, and we say when it moves. The quality is not trimmed to hold a date: no skipped hook, no weakened test, no "good enough" render.

When my Claude usage reached 75 % of the week on Friday, we went into economy mode: two sessions at most, a cheaper model for repository chores, no paid renders. The trains kept working without spending a single model token. Economy mode lasted a few hours, then I decided to carry on. That was my choice, and my budget.

Frequently asked questions

Does a train replace review?

No. Review happens before, by the session that writes the PR. The train does not judge the code: it waits, updates, approves and enqueues.

Why not GitHub's auto-merge?

We use it too. The train covers what it does not: waiting for the previous PR, bringing the branch up to date after a base change, and stopping at the first problem while saying so.

Why 5 PRs per repository?

Because 24 open PRs on one repository means 24 branches rebasing on each other. Five is what we manage to land in a day.

Chapter 19

Results

open PRs across the organisation at the peak of the pile, Friday 2 a.m.

140

The problem: a pile

open PRs on the busiest repository, before and after

24 → 4

What it changes

version-update PRs closed as duplicates

116

What broke, and the guard that came out of it