For a few weeks, people who build software have been asking me the same question. How do you run your CI, with so many repositories and agents opening pull requests all day?
I answer it here, once, with the real files. Everything I show is in public repositories of the krizaka organisation. You can go and check every line.
Why one shared CI
At first, each repository had its own CI file. We copied the neighbour's and adapted it a little. A fix in one repository never reached the others.
In the chapter on agents, I talked about a “build pack” for the CI, not shipped yet. Part of it has shipped. The krizaka/.github repository holds reusable workflows, one per kind of project: node-ci.yml, java-ci.yml, python-ci.yml, expo-ci.yml, maven.yml, mobile-e2e.yml. Each repository calls one, and says only what is its own.
Here is krizaka.com's, in krizaka-com/.github/workflows/ci.yml (shortened):
jobs:
ci:
uses: krizaka/.github/.github/workflows/node-ci.yml@1cb2e1b6f572ba712083cb6f05fd041370c5ea0b # v1.5.0
permissions:
contents: read
with:
duplication: true # jscpd + dup-ratchet.json: duplicated lines may only go down
next-cache-paths: .next/cache
container: true
container-smoke: scripts/ci-container-smoke.shThe workflow is pinned by its SHA, not by a tag. A tag can move, a SHA cannot. The # v1.5.0 comment says which version it is, and Dependabot proposes the update, in a PR that goes through the same CI as everything else.
Orazaka has its own pipeline, component.yml, in the krizaka/orazaka repository. An Orazaka component is never built alone: the CI clones the whole workspace, puts the tested repository in its place, and first builds its dependencies from source.
The full build pack, where each repository would have one line, still does not exist. Today it is one call and a few options. That is already much better than copies.
The path of a pull request
Here is what happens, for krizaka.com, between the moment a PR opens and the moment the change is live.
- Pull request (opened by an agent)The agent pushes its branch, opens the PR and arms auto-merge. It merges nothing itself.
- changes (docs only?)A first job looks at the files touched. A PR that only changes documentation skips the heavy jobs. If this job breaks, everything runs anyway.
- verify (lint, ratchets, tests)Install with a cache, lint, the ratchets, the type check, the tests with their coverage, then the build. It is the first gate: the image and SonarCloud wait for it.
- container (after verify)When verify is green, the Docker image is built, scanned by Trivy (a fixable high or critical finding fails the job) and given an SBOM. Then it is started, and a real browser checks the pages, from 360 to 1440 pixels, in dark and in light. No image is built for nothing when lint or the tests fail. A PR's image is never published.
- CodeQL (a separate check, required, parallel to everything)On by default on GitHub's side and enforced by an organisation rule. It reads the source, needs neither the build nor the tests, and starts as soon as the PR opens, outside our workflow. It blocks a serious security alert.
- SonarCloud (after the tests, inside the pipeline)An analysis run in our CI, after verify, with the test coverage; we wait for its quality gate. This is the case for krizaka.com, Akinaza and Yomizaka. The other repositories keep automatic analysis for now.
- CI gate (one verdict)A job reads the result of verify, container, SonarCloud and the workflow's other jobs, and gives a verdict. That is the one main requires, along with CodeQL, which stays a separate check.
- Squash on main (protection decides)When everything is green, GitHub merges. Branch protection decides, not the agent.
- release-please (a release PR)A bot keeps a release PR up to date. When I merge it, it creates the vX.Y.Z tag and the release.
- production (moves, deploys)The production branch moves forward to the tag, never forced, and DigitalOcean deploys from production.
One thing to read in this drawing: after verify, the image and SonarCloud run side by side, and CodeQL runs from the start, parallel to everything. The arrows give the order of the story, not the order of execution. The list below gives the detail.
The most important point is box 7. A PR may skip jobs, legitimately: a documentation PR does not need a Docker build. But a required check that never shows up blocks the PR forever. And a check that turns green because a job was skipped by mistake is worse.
So one job, CI gate, always runs, and it decides. An excerpt, in .github/.github/workflows/node-ci.yml:
gate:
name: CI gate
needs: [changes, ui-changes, docker-changes, plan, verify, e2e, duplication, container, sonar]
if: ${{ always() }}And the rule it applies, in the same file:
# The heavy jobs run unless the gate answered exactly `false`: a skip without that answer is a bug, not a pass.
if [ "$CODE" != "false" ]; then
[ "$VERIFY" = success ] || { echo "::error::verify must run when the gate did not answer false"; fail=1; }
fiA skipped job is accepted only if the first job said, exactly, “nothing to do”. The rest of the time, skipping is failing.
What main requires
On the main branch of our repositories, GitHub refuses a merge until some checks are green. For krizaka.com, Akinaza or Yomizaka, they are:
ci / CI gate, the pipeline's verdict;SonarCloud Code Analysis, required in 25 repositories since 25 October;- for the two mobile apps,
mobile-e2e / gate, more on that below.
CodeQL goes through another door: code scanning is on by default on GitHub's side, and an organisation rule enforces it. It is a separate check, required to merge, and it runs in parallel with the rest. It blocks a PR that brings an error, or a security alert of high or critical level.
SonarCloud has been on the Team plan since 26 October. We keep the default quality gate, which mostly judges new code. For krizaka.com, Akinaza and Yomizaka, the analysis now runs in our CI, after the tests, with coverage: it is a job of the workflow, and its result goes into the CI gate. The GitHub app still posts the SonarCloud Code Analysis check, the same as before. For the other repositories it is still automatic analysis, which does not read coverage: the “80 % on new code” condition is not evaluated there.
OpenSSF Scorecard runs too, every week and when a branch's protection changes. It is not required, on purpose: it never runs on a PR, so it could not be. It tells us whether our practices hold, it blocks nothing.
The protection applies to administrators too, so to me. History stays linear, and a review conversation must be resolved before the merge. No approval is required. There is one person reviewing. I did not want a second account or a bot that approves as soon as the CI is green: that would be an empty signature. I do the review. The right to merge comes from the CI.
The ratchets: the debt only goes down
A ratchet is the part that lets a wheel turn one way only. We have three, and they all work the same way. A file in the repository keeps a count. The CI counts again. If the count goes up, it refuses.
- lint-ratchet counts interface debts: a colour written by hand instead of a token, a
fetchin a client-side effect, a direct TanStack import instead of our facade. The tool iskrizaka-ratchet, from the@krizaka/configpackage. - sg-ratchet counts the violations of our ast-grep rules, rule by rule.
- dup-ratchet counts duplicated lines, with jscpd. For krizaka.com, the count is zero.
The heart of sg-ratchet fits in a few lines, in krizaka-com/scripts/sg-ratchet.mjs:
for (const id of [...ruleIds].sort()) {
const current = found[id] ?? 0;
if (!(id in recorded)) {
unrecorded.push(id);
next[id] = current;
continue;
}
if (current > recorded[id]) increased.push(id);
if (current < recorded[id]) decreased.push(id);
next[id] = Math.min(recorded[id], current);
}The last line is the rule. When we fix violations, we record the new count, and Math.min means it can never be raised that way. A rule without a recorded count fails the CI too: a rule cannot be slipped in quietly.
Why not fix everything at once? Because a new rule often finds dozens of cases. Fixing them all in the same PR makes a huge PR nobody reviews well. With a ratchet, the rule protects new code right away, and the existing debt goes down little by little.
On krizaka.com, the count of hand-written tables and forms shows it well. When the counter arrived, on 12 October, it found 20. Two days later, four changes had brought it to 11, 8, 3, then 2, and it has stayed at 2 since. Nobody can add a third.
A bug becomes a rule
This is the habit that changed our way of working the most. When we find a bug, we ask whether it is alone, or part of a family. If it is a family, we write a rule that refuses it, with its tests.
The first example comes from Akinaza. A privacy test checked that a webhook did not leak a client's phone number. It searched the payload for “514”. But that payload holds random ids, and about one time in ten, a UUID contains “514”. The test failed at random. We saw it in Akinaza: five failures in a single CI run, which passed when run again.
We could have run it again and moved on. We wrote rules/no-bare-digits-on-serialized.yml:
id: no-bare-digits-on-serialized
language: Tsx
severity: error
rule:
pattern: expect(JSON.stringify($$$A)).not.toMatch($R)
constraints:
R:
regex: "^/(?:[^/]*\\|)?[0-9]+(?:\\|[^/]*)?/[a-z]*$"Each rule has its tests, valid cases and refused cases, in rule-tests/no-bare-digits-on-serialized-test.yml:
valid:
- |
expect(JSON.stringify(m)).not.toMatch(/\bIA\b|intelligen/i);
invalid:
- |
expect(JSON.stringify(body)).not.toMatch(/42/);The first case passes: the pattern looks for words, not a run of digits. The second is refused.
A few days later, the same mistake came back in another form: a test looked for “4821” in an encrypted text, and an encrypted text is random bytes too. The issue for it says: fix the class, not the instance. The tests now check the whole plain text, and a second rule, no-bare-digits-on-random-bytes, refuses the pattern.
akinaza
- The bug
- “514” found by chance in a UUID.
- The rule
- no-bare-digits-on-serialized An ast-grep rule, with its valid and refused cases.
akinaza
- The bug
- A server page read a value from a client module: an invoice's state chip had lost its colour.
- The rule
- The client boundary test A server file imports only components and types from a “use client” module.
krizaka-ui
- The bug
- The build grouped files, and each group had its own router: some components ignored the app's.
- The rule
- A guard in tsup, and its test The router is always the package's entry, never a copy.
The third one is interesting because no test found it. A krizaka-ui PR went 6 kB over the package's size limit. Looking for why, we saw that each group of the build carried its own copy of the router context. The fix is a small plugin, in krizaka-ui/packages/ui/tsup.config.ts:
const oneRouter: NonNullable<Options["esbuildPlugins"]>[number] = {
name: "krizaka-one-router",
setup(build) {
build.onResolve({ filter: /^\.\.\/router$/ }, (args) =>
args.importer.includes("/src/router/") ? undefined : { path: "@krizaka/ui/router", external: true },
);
},
};A test checks that this plugin is plugged in. If someone removes it one day, the CI will say so.
A fixed bug is one bug less. A rule is its whole family less, in every repository that shares the rule.
- A random failure (akinaza#341)A privacy test looks for “514” in a payload full of random ids: five failures in one CI run.
- Alone, or a family? (the question)Searching a run of digits in random text fails by chance. That is a family, not one test.
- A rule (ast-grep)no-bare-digits-on-serialized refuses the pattern, with its valid and refused cases.
- The ratchet (a count recorded)A rule without a recorded count fails the CI: it cannot be slipped in quietly.
- The family comes back (akinaza#358)“4821” in an encrypted text: a second rule, no-bare-digits-on-random-bytes. The class is fixed, not the instance.
Mobile: Maestro in CI
For the web, this was already in place. For mobile, until this week, an agent opened the emulator and checked by hand. That cost a model session every time, and proved nothing the next day.
One issue changed that. The mobile-e2e.yml workflow runs everything in GitHub Actions:
- Throwaway backend (a fresh PostgreSQL)An empty PostgreSQL database and the built web app, on the CI machine. Nothing leaves the machine.
- Demo persona (seeded data)A demo account is seeded. For Yomizaka, it is Éric. The database is new on every run, so the data always starts from zero.
- Release APK (JavaScript bundled)An APK in the release variant, signed with the debug key, with the JavaScript inside. No Metro server.
- Emulator (Android, KVM)An Android emulator starts on the CI machine.
- Maestro flows (captures, video)Maestro plays the app's journeys. The captures, the video and the JUnit report are always kept.
- mobile-e2e / gate (required on main)A verdict that always shows up. A failing flow fails it.
The workflow's header says it itself, in .github/.github/workflows/mobile-e2e.yml:
# gate ALWAYS reports: `mobile-e2e / gate`. It is never skipped to get green: a failed or cancelled flow run fails it.The adoption rule is 3 green runs in a row before the check becomes required. Not before: a required check that fails at random blocks everyone, and people end up going around it. After three green runs, mobile-e2e / gate became required on main for Yomizaka and for Akinaza. An unstable flow is fixed, or set aside with an issue. It is never deleted.
On the web, it is Playwright. For krizaka.com, the CI starts the production Docker image and a real browser visits it, in krizaka-com/scripts/ci-container-smoke.sh:
# 360–1440 px, dark and light: no page scrolls sideways (a real browser on the real container).
npx playwright install --with-deps chromium
node scripts/check-overflow.mjs --base http://localhost:3000In krizaka-ui, the components' captures are compared pixel by pixel, inside the Playwright Docker image where the references were made. Same fonts, same Chromium, or every comparison would be wrong.
Releases and production
Our libraries, Krizaka UI and Krizaka TypeScript, use changesets. Each PR that changes a package adds a small file that says what changed, and at which level: patch, minor or major. On main, an action keeps a “chore: version packages” PR up to date. When we merge it, the packages go to npm, with their provenance.
I have to be precise on one point. Provenance is there, but npm's trusted publishing, the kind that needs no token, is not yet. npm refused our first attempt, because the packages had no trusted publisher declared yet. For now, an automation token does the publishing.
For krizaka.com, we set up a release train on 25 October, in krizaka-com#390. It is the end of the first drawing. release-please keeps a release PR. When I merge it, the tag is created, and a workflow moves the production branch forward to it. In krizaka-com/scripts/promote-production.mjs:
if (tagIsAncestor) return { action: "noop", reason: `${BRANCH} already contains ${tag}: never moved backwards` };
if (productionIsAncestor) return { action: "forward", reason: `${BRANCH} fast-forwarded to ${tag}` };
return { action: "refuse", reason: `${BRANCH} has diverged from main: it is never forced, look at it` };Three cases, no more. The branch is already up to date, it moves forward, or it has diverged, and then we stop and look. Never --force. The site deploys from production, not from main, in krizaka-com/deploy/digitalocean/app.yaml:
services:
- name: web
github:
repo: krizaka/krizaka-com
branch: production
deploy_on_push: trueSo merging on main puts nothing live. The release does, and I always know which version is running.
What broke this week
I do not want to give the impression that all this works the first time. Here is what broke, this week alone.
A race in turbo, in krizaka-ts. The release failed with Failed to resolve entry for package "@krizaka/db". Each package's check ended with a build, which deleted dist/ while another package's tests imported it. We first built everything before checking (krizaka-ts#79), then fixed the cause in turbo.json (krizaka-ts#75), with a test that guards the configuration:
"check": { "dependsOn": ["^build", "^check"], "outputs": ["dist/**"] }The fonts test before its build, in krizaka-ui. After a version PR, the release failed: the test of packages/fonts looked for dist/index.js, which did not exist yet. The test depended on the other packages' build, not on its own. krizaka-ui#702 adds the missing dependency.
pnpm refusing install scripts. Recent versions of pnpm refuse to run the install scripts of a dependency that was not allowed, with ERR_PNPM_IGNORED_BUILDS. The development build of orazaka-web-client had been broken since the day before. The fix lists the three allowed dependencies, in pnpm-workspace.yaml (orazaka-web-client#283):
allowBuilds:
"@ast-grep/cli": true
"@parcel/watcher": true
unrs-resolver: truenpm on a pnpm project. The deployment ran npm install on the web client's pnpm tree. npm crashed, on every deployment, with a message that said nothing useful. The build command became pnpm run build (orazaka#412).
Two repositories waiting for each other. To move to Krizaka UI 3.55, Orazaka's workspace had to regenerate its lockfile. Its CI failed against the web client, which still declared the old version. And the web client's PR needed the new lockfile. Each waited for the other. We got out by making the web client self-contained: it is no longer part of the npm workspace, it installs from its own pnpm-lock.yaml (orazaka#415). The price is that it now takes two shared packages from the registry, no longer from their source.
None of these failures was settled by running the CI again. Each one was fixed where it started.
Agents in the loop
Most of our PRs are opened by agents. An agent takes an issue, works on a branch, builds and tests before pushing, opens the PR and arms auto-merge. Then it is done.
Branch protection decides when the PR is merged. Not the agent, and not me clicking at the right moment. If the CI is red, nothing happens. Dependabot's updates follow the same path, in .github/.github/workflows/dependabot-automerge.yml:
- name: Enable auto-merge (not a major)
if: steps.meta.outputs.update-type != 'version-update:semver-major'
run: gh pr merge --auto --squash "$PR_URL"A major update stays manual.
An unstable test is a bug. We do not run it again blindly. The issue about a browser test racing a page transition says it plainly: start with a test that fails every time, by repeating it twenty times locally, and never make the check optional or retry it blindly. It is the same idea as the rules above: we fix the family.
An agent does not watch the CI run. Waiting for a CI run to finish is, for an agent, a model session running for nothing. GitHub knows how to wait, for free. The agent pushes, arms the merge, and stops. If the CI fails, we read the report and the captures the workflow kept. Nobody opens an emulator to check by hand.
We also stopped using Copilot's coding agent, after a cost spike. I give no figure here. Issues now go through our other agent sessions, with the same CI and the same rules.
Why CodeQL stands apart, and why Sonar and the image come after verify
I got this wrong in the first version of this chapter: I drew a queue, verify, container, Sonar, CodeQL. It was not true. verify and container ran at the same time, and so did CodeQL and SonarCloud. Here is what I changed, and why.
- CodeQL stayed in parallel, and required. It reads the source. It needs neither the build nor the tests, so it gives the fastest answer. Putting it behind verify would have made it wait for nothing. The only thing to do was to check that it is required: it is, by a rule, in Akinaza, Orazaka and Yomizaka.
- Sonar moved into the pipeline, after the tests. It is the only way to read coverage: automatic analysis cannot import it, so the “80 % on new code” condition was never evaluated. The
sonarjob waits for verify, takes the coverage reports from the tests, and we wait for SonarCloud's verdict. It is the issue we opened for it. It runs on krizaka.com, Akinaza and Yomizaka. For each repository I switched automatic analysis off (the two cannot run together), then re-ran the CI at once, ready to switch it back on if the CI was red. It was green, and all three stayed. The other repositories will follow one by one. Orazaka first needs Java coverage added to the shared parent. - The image waits for verify, and it is scanned. Before, it was built even when lint failed, which cost time for nothing. Now it comes after verify. Trivy looks for vulnerabilities and the job fails on a fixable high or critical finding; an accepted exception is written in a file, with its reason and an end date. An SBOM is kept on every run.
- Images are published only from main, after the merge. That did not change: a PR's image is never published, and the deploy builds from the
productionbranch. Only code that passed every required check ships.
On the first day, the image scan found real things. A critical flaw in the Next.js version of krizaka.com's image, fixed and released since. And high-severity flaws in npm itself, which ships with the Node image although the application never uses it in production: we removed it from the runtime images of Akinaza, Yomizaka and krizaka.com.
The krizaka/.github repository also has its own CI gate since this change. Its test suites only ran when their files changed, so none could be required, and two PRs had been merged with a red test. A job that always runs now decides which ones apply, and that is the one required.
What is not solved
There are still holes, and I would rather name them.
Coverage does not reach SonarCloud everywhere yet. It does for krizaka.com, Akinaza and Yomizaka. The other repositories, Orazaka included, keep automatic analysis, which reads the code but not our tests. The 80 % coverage threshold on new code is still to be required, after a week in the green.
The iOS test has not started. The mobile workflow only does Android. iOS is one line in its header, nothing more.
DNS is code, but applying it is still by hand. Our domains are now described in a repository, and the CI runs a read-only plan on every pull request and every day, to catch any drift. Changing the live records is still a manual step.
The web's Playwright tests are not required checks of their own. On krizaka.com, they count through the CI gate. Making them required checks in their own right is the next step planned in #241.
And the one-line build pack, I am still waiting for it.
What I take from it
A CI is not a file. It is a series of small decisions we no longer have to make again: one verdict, counts that do not go up, a rule after each family of bugs, a production branch that is never forced.
With agents, it matters even more. An agent works fast and does not get tired, but it only knows what we wrote for it. The CI is the place where what we wrote is checked, on every PR, for everyone. Me included.
