A while ago, I asked a friend who works in IT to run an experiment with no bias. He asked three models, Claude, Gemini and ChatGPT, to look at our GitHub organisation and give an unvarnished audit. What he told me was flattering. Too flattering for me to repeat without proof.
At Krizaka, a claim has to lead to a measurement, a run or a line of code. I don't have the outputs of his experiment, so I can't quote them. So I ran it again myself, on 24 October, with a prompt written in advance and published, and an answer published as is.
Why I ran it again
A compliment someone passes on can't be checked. A raw output can. You can reread the prompt, reread the answer, and follow every link.
I also wanted a prompt that doesn't push towards praise. It doesn't ask what is impressive. It asks for an audit, strengths and weaknesses, and says the weaknesses matter as much as the strengths. It also asks for a source for every point, and to say when the model couldn't open something rather than guess.
The full prompt is in the appendix.
The protocol
Two models got exactly the same prompt.
Claude Opus 5.5, from Anthropic, with two tools, web search and page reading. It started with no context: an empty folder, no project settings, no tool server, no memory. All it knew about Krizaka was what it read on the web during the audit.
Gemini 3.8 Flash, from Google, in the Antigravity editor, the same evening. Its access was not the same: it read the repositories with GitHub's command-line tool, on my machine, and queried SonarCloud's public API. In all, 143 steps, 69 of them commands (59 calls to GitHub, 10 to SonarCloud).
Claude Opus 5.5
- Web search and page reading, nothing else
- Pages summarised first by Claude Haiku 5.5
- An empty folder: no settings, no memory
- 39 turns, 5 minutes
Gemini 3.8 Flash
- GitHub's command-line tool, on my machine, with my sign-in
- SonarCloud's public API
- 143 steps, 69 of them commands
I also have to say what didn't run.
- ChatGPT. We have no OpenAI key. No ChatGPT audit either.
So the title only names Claude and Gemini. My friend's experiment with three models happened, but I'm not reproducing it, and I'm not quoting a word of it.
One technical detail matters when you read the answer: the page-reading tool first has the pages summarised by a smaller model, Claude Haiku 5.5. So the main model read a lot of summaries. It says so itself at the top of its answer, and adds that it checked the points that mattered most against the source code.
What Claude found solid
Quoted word for word. The full answer is in the appendix.
On the written governance:
"The written governance contract is unusually detailed, and CI enforces parts of it (ArchUnit, ratchets, docs-freshness check)."
On CI:
"The CI hygiene is good: SHA pinning, least-privilege permissions, red-main alerting, nightly builds and Scorecard."
On documentation and releases:
"Decisions are well documented, and the authors write candid self-audits."
"There are real Maven Central and npm releases, with signing and release-please."
Its overall judgement, in one sentence:
"The design and documentation are above average for a small organisation, and the CI is good."
"Above average for a small organisation." That's it. The model didn't write that it rarely sees this, or that it beats enterprise teams. So I don't write it either.
The guardrails it mentions exist, and you can read them. The ast-grep rules such as no-ambient-clock (no implicit clock in domain code) and no-enum-string-compare (no status compared to a string). The ArchUnit rules in CodeRules, including assertNoStandardStreams (no System.out in production), and their counterparts in GovernanceRules.
What Claude found weak
This is the longest part of its answer, and the most useful.
On review:
"The bus factor is one human. Review is effectively AI approving AI-authored code."
On the code itself:
"The implementation is ordinary-to-uneven and further behind than the documents suggest."
On one specific file of Orazaka's job service:
"
JobListener.javais the weakest file I read. It is 920 lines (874 lines of code) with 16 constructor parameters."
And a real defect, in that same file: a five-second timeout for any prompt that starts with TIMEOUT, with no guard.
"Any user can shorten their own job's timeout this way."
It also points out that our written rules and our code don't always agree (hand-written SQL despite the rule, a -SNAPSHOT version despite the ban), that two of our own audits contradict each other (one says don't launch, the other counts no blocker), that workspace builds follow the main branch of sibling repositories and so can't be reproduced, and that @krizaka/ui changes version too fast for a major number to still mean much.
Its conclusion:
"The biggest structural risk is not technical: one person and a set of agents produce more code, ADRs and issues than anyone can independently review."
What Gemini found
Gemini wrote a longer audit with more figures, because it read more code and SonarCloud's measures. Quoted word for word.
On what holds:
“The platform architecture cleanly decouples domain logic from framework baggage. […] Architectural boundaries are automatically verified on every build via ArchUnit.”
“quality-controls.md explicitly catalogs gaps, copy-paste drift, and technical debt instead of hiding them.”
“CI workflows consistently employ full-SHA pinning, OpenSSF Scorecard scans […], reproducible Maven build timestamps […], and binary compatibility checks via japicmp-maven-plugin.”
On what does not, its list is longer than its strengths:
“Infrastructure as Code Repository is Completely Empty.”
“Copy-Pasted Scripts and Configuration Drift Across 12 Repositories.”
“The "UTC Day" Date Slice Bug Across 8 Repositories.”
“While web frontends use strict rules, mobile applications […] lint with bare typescript-eslint, bypassing @krizaka/config and ast-grep quality ratchets entirely.”
“SonarCloud quality gates display "Passed / OK" on repositories with dozens or hundreds of open issues […], creating a false visual impression of a clean codebase.”
And its overall view:
“The core platform libraries […] receive top-tier engineering attention, while downstream products and tooling […] contain critical gaps.”
Where they agree, where they differ
Both models say the same thing on the essentials: the documentation and the CI are solid, and the gap between what is written and what runs is the real risk. Claude says it about review (“one human”), Gemini about breadth (“visibly stretched across 43 repositories”).
They differ in tone and in material. Claude stays sober (“above average for a small organisation”) and found a defect Gemini did not see, the TIMEOUT prefix. Gemini is warmer about the Java core, but brings more checkable facts: SonarCloud's figures, the copied scripts, the UTC day bug across several repositories. Part of that gap comes from access: Gemini read more files, and measures Claude did not have.
What I checked myself
An AI audit is a list of leads, not a verdict. I took the claims that can be checked with a number.
Size of the organisation
- What the model says
- 43 public repositories
- What I measured
- 43, from the GitHub API
JobListener.java
- What the model says
- 920 lines
- What I measured
- 920 lines on main
The TIMEOUT prefix
- What the model says
- 5 seconds, no guard
- What I measured
- True Lines 696 to 702, no property protects it.
Versions of @krizaka/ui
- What the model says
- about 83
- What I measured
- 83 on npm
Open issues in krizaka-ui
- What the model says
- 128
- What I measured
- 128
krizaka-ui on SonarCloud (Gemini)
- What the model says
- 172 bugs, 73 vulnerabilities
- What I measured
- 172 and 73 And the quality gate says “OK”: Gemini is right, it only judges new code.
A demo app's password (Gemini)
- What the model says
- “Blocker”, in production
- What I measured
- Overstated A local-development address, already refused in production. Removed anyway.
Concatenated SQL (Gemini)
- What the model says
- “Dynamic SQL”
- What I measured
- Overstated Constant fragments, the values bound as parameters: no injection, but the shape fools the scanner.
The empty infra (Gemini)
- What the model says
- No .tf file
- What I measured
- True when it read it The first stack landed the same evening, with a read-only plan (krizaka-infra#8).
The UTC day (Gemini)
- What the model says
- 8 repositories
- What I measured
- True Fixed in Yomizaka (#311), a rule now refuses the pattern; the other repositories follow.
Both models' figures were right. Two of Gemini's claims are stronger than the facts, and the table says so. The TIMEOUT prefix defect is real. There was already an issue to split that file, but it only covered the simulation hook, which is properly behind a setting. I added what the model found: a line our own issue had missed. I fixed it the same evening: a prompt no longer chooses its own timeout; only the test simulation can, behind its setting (orazaka-job-service#72).
I didn't check the rest. The claim about jobs that keep running after their timeout, for instance, the model itself says it didn't follow all the way. It stays a lead.
- The claim (Claude)A prompt that starts with TIMEOUT gets a five-second timeout, with no guard.
- Measured (lines 696 to 702)True on main: no property protects it.
- Our own issue (orazaka-job-service#50)The split of the file was planned, but only covered the simulation hook. The line was added to it.
- The fix (orazaka-job-service#72)A prompt no longer chooses its own timeout; only the test simulation can, behind its setting.
- What it teaches (no rule saw it)Our guardrails catch what we wrote a rule for. This line broke none.
The limits of the exercise
I want to be clear about what this audit doesn't prove.
- A model can flatter. The prompt tries to prevent it, but there's no guarantee. A sentence like "above average" is worth what an opinion is worth.
- They did not have the same access. Claude read only the public web. Gemini read the repositories with GitHub's tool on my machine, so with my sign-in. It is not a like-for-like comparison, and their answers should be read with that in mind.
- Claude only reads what's public. It saw neither our branch protection settings (the page needs a sign-in), nor what runs in production, nor anything not pushed.
- It read a lot of summaries. Pages go through a smaller model before they reach it. What it says about a file it didn't open itself is less certain.
- It's a single run. Run again tomorrow, the same prompt will give a different answer. I haven't measured that variation.
It's not a proof. It's an outside reading, published with its sources, that anyone can run again.
What I take from it
Mostly the list of weaknesses. Compliments are nice, but they do not say what to fix.
First, the way of working. A good part of the code is written by agents. What holds the quality is not the agents: it is the checks that say no early, the rules, the types, the architecture tests. A sentence I wrote down a while ago sums it up:
“To build effectively with AI, you have to structure your automations to detect errors as early and as fast as possible.”
Then, where that idea stops. Our guardrails catch what we wrote a rule for, not the rest. The TIMEOUT line broke no rule. And a rule only looks where it is told to: in Yomizaka, a date computed in UTC instead of the Québec day slipped past no-ambient-clock, because the rule did not read that folder. We had to write no-utc-day-slice after the bug (yomizaka#311). Our own inventory, docs/quality-controls.md (.github#232), says it in one sentence: “The rules do not look where the bug was.”
Last, review. The model is right: there is one person to review, and part of the approvals goes through an agent I delegated decisions to. The automated checks are real. A second pair of human eyes is still missing. That is the point I keep in mind, and another tool will not settle it.
Appendix: the prompt and the raw output
The audit ran on 24 October 2026, from 23:53 to 23:58 UTC, with Claude Code 2.1.308 in headless mode, the model claude-opus-5-5, and only the WebSearch and WebFetch tools. The published files are untouched:
- the prompt;
- Claude's full answer, uncut;
- the run details: the command, the models, the tokens, the cost, and what couldn't run;
- Gemini's full answer, with one redaction, the password of a local address masked;
- the metadata of Gemini's run, read from Antigravity: the model, the number of steps and every command it ran.
The prompt, as sent:
You are reviewing a software organisation you have never seen before. Use your web tools to read it yourself; do not rely on prior knowledge of it.
Target: the public GitHub organisation https://github.com/krizaka
Please:
1. Look at the organisation page and the list of its public repositories. Open the repositories you judge most important (for example their README, a few source files, the CI workflows, any architecture or decision documents, and recent pull requests or commits).
2. Give an unvarnished technical audit of what you actually read: the architecture, code quality, testing, documentation, and the engineering setup (CI, automated checks, release process, governance).
3. List the strengths and the weaknesses. Weaknesses are as important as strengths: name risks, gaps, inconsistencies, things that look unfinished, and anything a careful reviewer would worry about.
4. For every point, say which repository, file or page it is based on (a URL), so a reader can check it. If you could not open something, say so instead of guessing.
5. End with a short overall assessment and the three changes you would make first.
Be specific and neutral. Do not flatter and do not soften criticism. If something is ordinary, say it is ordinary.The tokens, as the tool counted them:
- Claude Opus 5.5: 30 input tokens, 47,862 written to the cache and 416,449 read from it, 14,027 output, so $0.61.
- Claude Haiku 5.5, for the page summaries: 481,596 input tokens and 79,082 output, so $0.09.
