Skip to the content
Krizaka Blog

20

How our automation grew: from one heavy CI to workflows by concern

Oct 9–31, 20268 min read

What I faced, A CI growing heavier, the same for everything; What we built, workflows by concern, a speed budget, a guard for every bug

A long vaulted hall in watercolour, stone arches repeating to the far end, lamps hanging under the roof beams and a small red duck in the middle of the aisle

Three weeks ago, krizaka.com's CI took a minute and a half. Today, a pull request waits a little over ten minutes to learn whether it can go. That is no accident: every added minute is a check we wanted, often after a bug. This chapter tells how it grew, what it taught us, and how we put it back in order.

It follows straight on from Tchou-tchou. The trains land pull requests one by one, but a train never runs faster than the CI it waits for.

A CI that grows

At first, krizaka.com had its own CI: one file, a few steps, lint, tests, build. Then, on 21 October, the site moved onto the organisation's shared workflow, the one our Node applications share. On the 22nd, a duplicate-code check with a ratchet. On the 26th, SonarCloud's analysis with test coverage. On the 28th, the merge queue. Each of these had a reason, and each cost minutes.

The median duration of a successful CI run on a krizaka.com pull request, by week: it grows as checks are added.

Counted from the GitHub API on 31 October: the last 1,000 runs of krizaka.com's CI workflow since 9 October, pull requests that succeeded only; weeks from Monday to Sunday, the last one up to 31 October.

Bar chart, 4 (Week of 6 Oct – Week of 27 Oct). Minutes: from 1.4 to 10.6, lowest 1.4 (Week of 6 Oct), highest 10.6 (Week of 27 Oct), up 657%.

The median duration of a successful CI run on a krizaka.com pull request, by week: it grows as checks are added. — data
ItemMinutes
Week of 6 Oct1.4
Week of 13 Oct2.1
Week of 20 Oct6.1
Week of 27 Oct10.6
The median duration of a successful CI run on a krizaka.com pull request, by week: it grows as checks are added.

At the median, a successful run went from 1.4 minutes to 10.6. I will not present this chart as a win. It shows a CI getting slower, week after week. What we gained does not show in the bar: it is what the CI now stops before production. But the cost is real, and it is what forced us to measure.

A pull request's path, today

Here is what krizaka.com's CI does when a pull request opens. The site's file is forty-five lines: it calls the organisation's shared workflow and tells it what to run. Everything else lives in one place, for every Node repository.

The graph of krizaka.com's CI on a pull request, redrawn: what runs, in what order, and what is skipped when nothing concerns it.
  1. Changed files (a few seconds)A first job looks at what the PR touches. A PR that only changes documentation skips all the heavy work.
  2. Plan (the matrix)It reads the repository's scripts and turns them into a list of tasks to run in parallel.
  3. Verify (in parallel)Lint, types, tests and build, each in its own job. Alongside, two checks that wait for no one: “a fix needs a guard” and duplicate code.
  4. Journeys (in shards)The end-to-end tests, cut into several pieces that run at the same time, only when the interface changes.
  5. Image and analysis (after green)The container image is built and started, and SonarCloud analyses the code. Both wait for verify to be green: no point paying for the longest jobs on a PR that is already red.
  6. CI gate (one check)A last job reads all the others and gives one answer. It is the only check required on main.

Two choices matter more than the rest. First, skip what serves nothing: a PR that does not touch the interface does not run the journeys, and a documentation PR runs neither the tests nor the image. Second, one required check, the gate. Before, each repository required a list of job names, and the slightest rename blocked PRs on a check that no longer existed.

The loop: a bug, a guard

The rule that made the CI grow is this one: a bug means a guard was missing. We do not only fix it, we ask why nothing stopped it earlier, and we add what would have. This week, the loop produced these checks:

  1. A fix needs a guardA PR whose title starts with fix must add or change a test, a rule or an audit. Otherwise the job is red. You can be exempted, but you have to write why.
  2. No co-author lineA commit that names an AI as co-author is refused. The author is our application.
  3. Five PRs per repositoryOpening a sixth pull request on a repository is refused, bots included.
  4. No orphansA script lists the open PRs no train carries. Green, it gets a train; red, it goes back to its lane.
  5. Search checkedEvery site of the family passes the same check of the files search engines read, in CI and then after each deployment, on the live site.
  6. Green before heavyThe image and the analysis only start after a green verify.

Each of these guards has its test. That is what sets them apart from a rule written in a document: a rule gets forgotten, a guard refuses.

Measure before you cut

On 30 October, I asked that we stop adding and measure. One full week, read from the GitHub API, for the organisation's forty active repositories: 16,003 workflow runs in seven days. We set a simple budget: at the 90th percentile, a PR must get its answer in 10 minutes, the merge queue in 15. The result: 23 repository and event pairs out of 41 were over.

The measurement found bugs first, not slowness. The worst: since 25 October, the build job ran again the tests the test job had just run. On Akinaza, that cost almost nine minutes on every PR and every pass through the queue. Nobody had seen it, because the result was green.

Workflows by concern

The decision is called ADR-018. Instead of one big workflow doing everything for everyone, each workflow has one concern, and its name says it: ci-node, ui-visual, mobile-e2e, deps-automerge, release-changelog.

Before

  • One big workflow per language, doing everything
  • The same work on every PR, whatever the change
  • A list of required checks copied into every repository
  • No measure of the time

After

  • One workflow per concern: ci, ui, mobile, deps, release, seo
  • Path filters: only what the PR concerns runs
  • One profile per repository, and a single required gate
  • A speed budget, measured every week
Before and after: one workflow that does everything, then workflows that each have a concern, a moment and a budget.

Each repository declares a profile (a library, a web application, a mobile application, a Java service, a site) and gets the set of workflows that goes with it. Akinaza is both a web and a mobile application: it composes the two. A test refuses an active repository without a profile.

What we got wrong

I would rather say it, because these mistakes are what shaped the rules.

  • Computed job names. The name of a verify job was an expression. When the job was skipped, GitHub showed the raw expression instead of the name, and a PR stayed blocked on a check that would never come. Names are now literal, and a test checks it.
  • A queue timeout too short. The merge queue gave up on a PR after thirty minutes, while some CI runs took longer. Healthy PRs were ejected. The timeout is now longer than the slowest CI.
  • Stale reds. A red check on a branch behind main had often already been fixed on main. We re-ran by hand. The train now updates the branch once before judging.
  • One rebuild per release. Every product release forced a full rebuild of krizaka.com: 287 in one week, over thirty hours of machines waiting.
  • krizaka.com's CI is too slow. At the 90th percentile, a PR still waits almost seventeen minutes, above our own budget. The container image, with its start-up test, takes almost eight. It is not solved.

What I take from it

A CI that grows is not a failure. It is the sign that we learn from our bugs. The failure would be letting it grow without measuring it. What I take from these three weeks fits in three ideas: every bug leaves a guard, every guard has a concern and a moment, and every minute is measured against a budget.

What comes next is already written in the ADR: a health scorecard for every repository, proving the rules are applied without an agent having to check them by hand.

Frequently asked questions

Why is krizaka.com's CI slower than before?

Because it does much more: end-to-end tests, the container image started, code analysis, the duplicate check. The speed budget is there to keep these checks without letting the time drift.

Why a single required check?

Because a list of job names to require breaks as soon as a name changes. The gate reads every other job and gives one answer: that is the one we require.

One workflow per concern, isn't that too many files?

A few more files, but each is short, has a name that says what it does, and only runs when it is concerned. The file name is not part of the check's name, so renaming a workflow breaks no protection.

Chapter 20

Results

minutes, the median successful CI of a krizaka.com PR, first and last week

1.4 → 10.6

A CI that grows

workflow runs in seven days across the organisation

16,003

Measure before you cut

repository and event pairs over the speed budget

23 of 41

Measure before you cut