On 17 November, the sixteen probes Grafana runs against our sites and our dev apps were all answering. This chapter is about how they are written, what happens when one of them fails, and what we left to DigitalOcean. It is written for people who run production: it is about rules and windows, but no stack name, no internal address and no exact threshold.
Krizaka has no customer yet. There is no on-call team: alerts come to me. So the monitoring had to fit in a free tier and bother me only for a real outage.
First, what a visitor gets
The host's console tells me a container is running. It does not tell me that the domain name, the certificate, the CDN and the page all answer together for someone arriving from outside. The first signal I wanted was a visitor's: a request that leaves from somewhere else, and its result, probe_success.
The probes are Synthetic Monitoring's, the probe service of Grafana Cloud. There are 16: eleven public pages and health routes (the company site and its health route, its French home, a public data file, the blog, the library sites and the product sites), the health routes of the three dev apps, and the payment webhooks of two of them. Each one leaves from two Canadian locations, Montreal and Calgary. It is a plain GET, with no added header, no cookie and no secret: the probe sees what an anonymous visitor sees, and it carries nothing worth stealing.
The payment webhooks needed something else. A healthy webhook refuses a call that is not signed. So the probe sends it an empty POST with no signature, and the only accepted status is 400. A 200 would mean it accepted an unsigned request, a 5xx that it is down. The same probe checks that it answers and that it refuses an unsigned call. It does not prove that a real event gets through: with a wrong signing secret, the webhook refuses everything and the probe stays green.
The probes and the rules, written as code
The list of probes is a file of our infrastructure as code, not a configuration clicked together in the UI. The script has two commands. plan, the default, reads the stack and says what would change, writing nothing. apply creates or updates the checks, the contact point, the rule and the routing; a second apply changes nothing. Only the checks whose name starts with our prefix are ours, the others are never touched.
plan earned its keep on day one. On 7 November, the code declared eleven checks and the stack was running thirteen: two health routes under old code names, retired since, were still being probed. A configuration made in the UI drifts silently; a configuration as code drifts too, but plan says so.
The frequency has a ceiling. The free tier counts executions: 100,000 API tests a month (Grafana's pricing page, read on 17 November 2026). Checks × locations × runs a month has to stay under it, and a test fails if a new probe would push us over.
- 1Two locations: Montreal and Calgary, public probes of Grafana Cloud, outside our host.
- 2The sites and the dev apps: a GET for the pages and the health routes, an unsigned POST for the webhooks.
- 3The Sovkora engine: it pushes its six pilot measures over OTLP, with no label that names a customer.
- 4The metrics: the probe results and the engine measures, in the stack's Prometheus.
- 5The rules: they read the metrics; no data does not fire an alert.
- 6One contact point: an e-mail, grouped by alert and by check.
- 7DigitalOcean's alerts: deployment, domain, restarts and memory, written in each app's spec.
- 8Me: two sources of alerts, never the same rule in both places.
The rule “Site public injoignable” (public site unreachable) is the only one about the sites. For each check and each location it takes the maximum of probe_success over a sliding window, then keeps the highest value. It fires when that value is below 1, which means every location fails. We assume a single location failing is the network between a probe and us, not a site down. No data leaves the rule in its normal state, an evaluation error puts it in error. The flip side: if the probes stop writing, the rule stays green, and no other rule says so yet. It carries the label severity=page.
The routing is as simple as it gets. The root policy sends everything without a route to the contact point, grouped by alert name and by check, with a long repeat interval. The contact point is an e-mail. The script creates it without marking its provenance, which leaves it editable in the UI, and apply never rewrites an existing address: I can change who is notified without a pull request. While setting it up, we found that our stack only e-mailed members of its organisation.
The Sovkora engine has rules of its own, six, written by a second script. It reuses the first script's client, folder and routing without ever changing them: each object of the stack has one owning script. The engine exports its measures over OTLP: a histogram of the delay before a run's first step, a counter of ended runs by outcome, a histogram of durations, a gauge of stuck runs, and two counters, tokens and provider cost. No label names a customer.
| Rule | What it reads | Why it is written this way |
|---|---|---|
| Slow run start | the p95 of the delay before the first step | a histogram quantile: it shows the slow tail a mean dilutes |
| Failed runs | the share of ended runs that failed | only counted above a small number of runs: below, a share is noise |
| Runs too long | the p95 of the duration | same reason as the start |
| A run stuck | the gauge of runs in progress for too long | one is enough; this group is evaluated more often than the sites' rule |
| Tokens running away | the tokens of the last hour | an agent loop shows here before it shows on the bill |
| Cost of the day | the provider cost over 24 hours | compared with the dev's daily budget, the one in the app's spec |
All six carry severity=page and the same team, so the root policy sends them to the same contact point as the probes. Their thresholds are marked in the code as starting values, to be tuned from what dev shows. For the engine too, no data does not fire an alert: an idle dev has no runs. Next to them, one last alert watches the Grafana bill itself.
DigitalOcean on one side, Grafana on the other, nothing twice
On 17 November, I set the rule in one sentence: monitoring lives in DigitalOcean and in Grafana, nowhere else, with no duplicate.
DigitalOcean keeps what the host knows better than anyone. Each app spec carries its alerts, as code: a failed deployment and a failed domain, and for the services, on top, the restart count and memory use. The spec is applied by our deploy tool, and a read-only check must report no difference between the file and what runs. Grafana keeps what the outside sees and what the engine does.
Before this rule, two other things did the same job: a scheduled GitHub workflow and a probe running on my Mac. We are removing them. Two alerts for one fact teach you to ignore both, and a probe on a laptop sleeps when the laptop sleeps. One outage can still arrive from both sides: a failed domain at DigitalOcean also makes the probes fail. One says the cause, the other what the visitor sees.
The drawing in words
Elements
- Visitor: Reads the company site, the blog and the library sites.
- Krizaka platform: How the org works: shared CI and standards, the infrastructure as code, the templates.
- Infrastructure as code (OpenTofu): One OpenTofu stack for every environment (ADR-023).
- Cloud hosting: The app and static site hosting, the managed databases.
- Dev apps: The products in their development environment, on the same host.
- Observability: A hosted Grafana stack in a Canadian region: what a visitor gets, what the engine does, who is told.
- Public sites: The company site, the blog, the library sites and the products sites.
- Founder: Sets the strategy and approves.
Relations
- Visitor → Public sites: reads
- Dev apps → Observability: OTLP, no tenant label
- Observability → Public sites: probes from outside
- Observability → Dev apps: probes from outside
- Cloud hosting → Public sites: hosts
- Cloud hosting → Dev apps: hosts
- Infrastructure as code → Observability: two scripts beside it, plan then apply: probes, rules, dashboards
- Infrastructure as code → Cloud hosting: provisions
- Observability → Founder: e-mails what is urgent
- Cloud hosting → Founder: e-mails deploy, domain, restart and memory alerts
- Founder → Observability: reads the surfaces
At the container level, every arrow has its direction. The probes write to the metrics, the rules and the reading surfaces read them, and only the rules talk to the contact point. Logs, traces and load tests are drawn as planned, not as done.
The drawing in words
Elements
- Synthetic probes (Synthetic Monitoring): One HTTP check per public page or health route, from two Canadian locations, no added header, no secret.
- Alert rules (Grafana Alerting): Unreachable site, six engine measures, the bill. No data fires no alert.
- Reading surfaces (Grafana dashboards): Seen from outside, the engine pilot measures: one row per probe, a legend under it.
- Load tests (k6): On dev only, thresholds taken from the SLOs.
- Sovkora engine (Java, OpenTelemetry): Exports its pilot measures: run start, ended runs, duration, stuck runs, tokens, cost.
- Payment webhooks (HTTPS): Refuse an unsigned call: 400 is the healthy answer.
- Public sites: The company site, the blog, the library sites and the products sites.
- Contact point (E-mail): One for everything urgent, grouped by alert and check.
- Logs and traces (Loki, Tempo): Waiting for the shared redaction and the processor agreement.
- Health routes (HTTPS): One per web app: answers when the app and what it needs answer.
- Metrics (Prometheus): Probe results and the engine measures.
- Founder: Sets the strategy and approves.
Relations
- Synthetic probes → Metrics: success, duration, status, certificate
- Alert rules → Metrics: evaluates
- Reading surfaces → Metrics: reads
- Alert rules → Contact point: severity page
- Reading surfaces → Logs and traces: will read, once they arrive
- Synthetic probes → Health routes: GET
- Synthetic probes → Payment webhooks: unsigned POST, expects 400
- Load tests → Health routes: gentle load, dev only
- Sovkora engine → Metrics: OTLP, no tenant label
- Synthetic probes → Public sites: GET, as a visitor would
- Contact point → Founder: e-mails
- Reading surfaces → Founder: shows the sites and the engine
The screens
I first built “krizaka.com · Seen from outside” with Grafana's Pages, the new editorial form Grafana announced on 19 October 2026. A page reads like a text, with a sentence and a table, and I prefer it to a grid of panels. But a Page cannot be kept as code today. The feature is in public preview, its API is an alpha version that holds only the catalogue, no export or schema is documented, and on 7 November the stack's service account could see none of them.
So the reading surface is a Grafana dashboard, written in the Page's style.
- 1The title and its sentence: what a visitor gets, in one line.
- 2The source line: the signal read and the repository file the dashboard comes from; the rule is that a change made in the UI is overwritten by the next apply, and the line says so.
- 3“Right now”: one sentence, every probe answering or a probe failing, see the table.
- 4One row per probe: the check's name, selected by the app and environment labels rather than a list of names.
- 5The columns: up now, availability against its target, failed runs, HTTP status, days before the certificate expires.
- 6The band: up or not, run after run.
- 7The footnote legend: what each column means, and what to do when it turns red.
Its data source is a variable picked by name, never by an id, and on 7 November it passed Grafana's linter in strict mode with no finding. The engine's dashboard, “Sovkora — engine pilot measures”, has six panels two by two, one per measure: the delay before the first step, ended runs by outcome, the duration, stuck runs, tokens per hour in and out, the cost over 24 hours. It comes from the same script as its rules: a renamed metric breaks the panel and the rule in the same change.
Where we are going
The sites' rule sees a clean outage, not a slow degradation, and says nothing about an objective. Google's SRE workbook offers better: alert on how fast an error budget is being spent.
Service level objectives will become data: one file per app, with the availability target and its 28-day window, the p95 latency target on a route, and the source signal. krizaka.com will start at 99.9 % over 28 days. The reading surface, the load tests' thresholds, the alert rules and the cockpit will all read that one file.
Alerts will burn an error budget. The plan follows the workbook: two multi-window rules per objective, a page at 14.4 times the budget rate over one hour and over five minutes, and a ticket at 6 times over six hours and over thirty minutes. The workbook makes the second one a page as well; we make it a ticket, because one person reads the alerts. One question is still open. On sites that still have few visitors, the error rate is made of probe runs, few per hour, and a single missed run already exceeds 14.4 times the budget rate over an hour: as is, burn rate would alert more often than today's rule, and a short window holds too few runs to say anything. The workbook has a section on low-traffic services; we will read it with our own numbers before writing the first rule.
The load tests are written with k6, in the app's repository, and aim at dev only: the script refuses any other origin. The scenarios stay gentle, at most 5 virtual users for at most two minutes, because dev is shared. The first run, on 7 November, against Sovkora dev's health route: 7,767 requests, none failed, a p95 of 62 ms. Pointed at production, the same script stopped before sending a request. A load test on production is a paid decision, and it has not been taken.
The service map will come from OpenTelemetry traces, with the W3C context propagated over HTTP and over the message broker. The links observed over seven days will sit next to the architecture generated from the code. A link observed but not modelled, or modelled but never observed, will be reported as drift.
In the cockpit, all of this will fit in one “Observabilité” block. It does not exist yet: the drawing comes from the plan, not from a screen.
- 1Every link: Grafana's home, the reading surfaces, the k6 project, the probes, the alert rules.
- 2Dev and prod: production shows “à créer” (to create) as long as it does not exist.
- 3One row per app: read live from the stack, server-side, with a read-only token.
- 4The headline numbers: up, availability against the target, p95 latency, error budget left, open alerts, the last load-test verdict; each one leads to its page.
- 5“indisponible”: an unreachable stack shows as unavailable, never as a made-up 100 %.
Why Grafana and not Datadog, and why the cloud
There is no written decision for this choice. What follows is our reasoning, rebuilt from what our code needs, and the facts read on the two vendors' official pages on 17 November 2026. I keep the two apart.
The facts first. Grafana Cloud offers a region in Canada, and our stack is there. Datadog's list of sites has sites in the United States, Germany, Japan, Australia and the United Kingdom, none in Canada. Grafana, Loki, Tempo, Mimir, Pyroscope and k6 are published under AGPLv3, and Prometheus under Apache 2.0. Datadog is a hosted service: its agent is open source, under Apache 2.0, but the platform does not install on your own machines.
The free tiers next. Grafana Cloud's includes 10,000 active series, 50 GB of logs and 50 GB of traces, kept 14 days, 50 GB of profiles, 100,000 Synthetic Monitoring API tests and 500 k6 virtual-user hours a month, and 3 active users; above that, the paid tier is billed by usage. Datadog's covers up to 5 hosts, with one day of metric retention.
One last fact, which is not a reason. Both accept OTLP straight from an application's SDK, and both recommend a collector or an agent in production. “No agent to run” holds for dev with either, and does not tell them apart.
Now our reasoning. We are a Quebec company; when our logs leave, redacted, they will stay in Canada, and that option exists with one and not the other. The region says where the data is stored, not who can access it: Grafana Labs remains a US company, and the data processing agreement will set the rest. Our PromQL queries and our dashboards as JSON work outside Grafana Labs: if we leave, Prometheus and Grafana run anywhere. The list of probes, though, is written for the Synthetic Monitoring API and would need translating. With a platform that only exists hosted, leaving would mean rewriting the dashboards, the rules and the probes; the OTLP instrumentation would leave as it is. And the free tier fits our size, a company with no customer, where the probes are what consumes, which the code knows and checks.
Why the cloud, when all of it is open source? Because a monitoring tool must not fall with what it watches. Installed at our host, it would go down with it. It would also be one more server to pay for, patch, back up and watch, so a monitor for the monitor. The hosted stack is outside our host, in Canada, and free at our size.
The catch is production. The free tier allows only one stack per account, and ours is the dev one. A separate production stack needs the paid tier or another account, and that decision is mine. Until it is taken, no production application data goes to Grafana; only the probes, which carry no customer data, already measure the public sites.
What we have not done yet
Grafana is not in our OpenTofu. The probes, the rules and the dashboards go through the API, from two scripts run by hand, plan then apply; neither runs in CI.
The apps send no logs and no traces yet. The engine is the first whose code exports metrics. The logs wait for the shared redaction of personal data, which must run before any export, and for the processor agreement with Grafana Labs, which must be signed before the first log.
The alert arrives by e-mail at one address. There is no on-call rotation and no paging on a phone, and the sites' only rule is still a raw threshold.
The engine's thresholds have never seen real traffic. On 7 November, k6's results did not reach Grafana, for lack of the token they need. The cockpit block is a plan.
If you do this for a living and one of these choices makes you wince, burn rate on low-traffic sites above all, write to me through the site's contact form.
What we measured, and what we did not
The state of the sixteen probes, the list of rules and the dashboards' names were read in Grafana on 17 November. The rest, the probes, the windows, the rules, the labels, the routing, comes from the code of the two scripts and the alerts of the specs, read the same day. The drift count and the k6 run date from 7 November. The facts about Grafana Cloud and Datadog come from their official pages, read on 17 November 2026; prices change, and those links are the reference. I give no availability figure: no 28-day window has been measured against a written objective yet. The drawings are made from the screen's description, the code and the plan, and are not to scale; the cockpit block exists only on paper.
