Nobody was told: why silence is the failure mode of alerting

For seven days a watchdog on our trading server was killed on timeout, thirty-eight times. It said nothing each time, because a process killed by the scheduler never reaches its own notifier. The scheduler then cleared the failed state when the next run started. Every dashboard we had read green for the whole week. The thing guarding the connection to the broker was falling over several times a day, and nothing anywhere knew.

We found it with a script whose only job is to ask the two questions an alert cannot. That script, and the rules that grew around it, are what this post is about. None of the rules is clever. Each one was paid for.

The short version

• Alerting can only report what someone thought to alarm on. Its failure mode is not noise. It is silence, and silence looks exactly like a quiet night.
• A dashboard shows the last state a job reported. A job that dies before reporting leaves green on the screen.
• Every process leaves a verdict every day: pass, fail, or skip with a reason. A missing verdict is displayed as a failure, never as a blank.
• Alarms rest on positive evidence. Absence of a signal is not a problem; it is the absence of a signal.
• The alert wire needs its own alarm. If nobody has been at the other end for two days, something has to say so through a different door.

flowchart LR
  J["Job fails"] --> A{"Did anyone<br/>think to alarm<br/>on this?"}
  A -->|yes| T["Alert sent"] --> R{"Did it reach<br/>anyone?"}
  A -->|no| S["Silence"]
  R -->|no| S
  R -->|yes| H["Someone looks"]
  S --> G["Dashboard stays green"]
  G --> D["Found days later,<br/>by its consequences"]

Where silence comes from

Three shapes, and we have had all three.

The job that dies before it can speak. A wrapper that reports failures at the end of a run never runs its ending if the scheduler kills it for overrunning. The watchdog above was this shape. So was a nightly data collector on the same server that failed two days running; the first real sign was the trading loop refusing to trade on half its data, forty-eight hours later.

The message that was never delivered. A sender that fails politely, swallowing the error and returning cleanly, produces the same silence as no sender at all. Two of three credential incidents in one night on our servers were this shape: the job ran, the alert was composed, and the send failed without a sound.

The alert that reached nobody. The wire worked, the messages landed, and the chat they landed in was one nobody had opened for five days. Thirty-one alerts marked "action needed" sat there. Nothing noticed, because the only component that did notice reported through the wire it was reporting on.

At the plant where I worked before, the same thing wore a different coat. A fault in January quietly stopped the scrap and rework figures landing. Nobody noticed for six months, because nothing alerted and the dashboards simply showed less. Less waste looked like good news.

Rule 1. Every process leaves a verdict, every day

The first fix was to stop asking "did anything alarm?" and start asking "is yesterday complete?". A single morning check walks every process we depend on and demands a fresh verdict from each: the collectors, the market-day loops, the backups, the self-tests.

Three outcomes, never two. Pass means the job ran and proved its result. Fail means it ran and did not. Skip means it could not honestly run, with a reason: a market holiday, an empty queue, a deliberate stand-down. A probe that could not run is not a pass, and a job that ran but reported nothing is a fail.

The display rule matters as much as the check: a missing verdict is shown as a failure. Before this, three collectors had failed on the same day and the health page showed all three as a neutral "scheduled, no record yet". A failure displayed as a silence is worse than no page at all, because the page is what tells you not to look.

flowchart TB
  P["Process ran today?"] --> V{"Verdict recorded?"}
  V -->|"pass"| OK["✓ proved its result"]
  V -->|"fail"| F["✗ ran, did not prove it"]
  V -->|"skip + reason"| SK["– could not honestly run"]
  V -->|"no verdict at all"| NV["✗ shown as a failure,<br/>never as a blank"]

Rule 2. Prove the write, not the intent

A collector that reports "90.9% collected" from its own memory is describing what it meant to do. Ours did exactly that for four days while the store took one row a day. Now every collector proves its write by reading it back from the store and comparing against what it was asked to fetch. If the read-back is short, the job exits non-zero and says by how much.

The same rule, one level up: a nightly job that pulls a day of data proves it holds that day for every name on the roster, by content, never by the existence of a file. A gate that checks file presence is a gate that passes on an empty file.

Rule 3. Alarms rest on positive evidence

This is the rule that stops the previous two from turning the channel into noise.

A long job that has not moved its counter in an hour looks stuck. Often it is not: a data puller that respects a rate limit sits at zero CPU for minutes at a time and is working perfectly. Alarming on "no movement" produced false stalls, and a false alarm is what gets a channel muted, after which the real one is missed too.

So a job with no measurable signal is shown as running, with no counter, and is never called stalled. The corollary is a duty: give every long job a real counter, names done or rows landed, so that a genuine freeze is detectable from evidence rather than inferred from absence. And never widen the quiet set to silence a noisy job. A job that pages too often is a job to fix.

Rule 4. Tiers are promises, not moods

Every automatic message we send opens with a status word, and the word is a promise about what the reader must do. Action needed means nothing retries and nothing recovers. Crashed means a job died and nothing restarts it. Warning means one step went wrong and the job carried on. Done and Skipped are on the record and ask for nothing.

The tier decides who is woken. Only the tiers that mean something is wrong reach the always-on agent that investigates; the rest are readable any time and wake nobody. Three properties hold this apart from a suppression. A message with a missing or unknown status always wakes someone, because "we have never seen this word" is not evidence that things are fine. Nothing is hidden: the quiet records still ride along as context, because they are often what explains the alarm. And a job that wants to be quiet has to say so at the source; the reader never has to guess.

Rule 5. The wire needs its own alarm

Once alerts reached an agent instead of a person, the last silence was the wire itself. So the wire is proved every day: a labelled test alarm goes in one end and the check confirms an agent picked it up at the other. The proof has three outcomes like everything else.

Then the dead man's switch. Two failed proofs in a row, not one, because a single miss can be a restart in progress. On the second, the check escalates through a different channel from the one it is checking, because a component that reports its own failure through the thing that failed reports nothing. That circularity was the whole defect behind the five silent days.

sequenceDiagram
  participant D as Daily proof
  participant W as Alert wire
  participant A as Agent session
  participant S as Dead man's switch
  participant O as Other channel
  D->>W: test alarm, labelled as a drill
  W->>A: typed into the session
  A-->>D: picked up (pass)
  Note over D,A: any other outcome is a fail, not a blank
  D->>S: verdict recorded
  S->>S: two fails in a row?
  S->>O: escalate, not through the wire

The sweep: asking what an alert cannot

The rules above make failures loud. The sweep is for the ones that are still quiet, and for the opposite error. Twice a day it reads the scheduler's own journal and the ledger of every alert sent, then cross-references them and asks:

1. What failed that nobody was told about? A failure in the journal with no alert beside it.
2. What alerted that had not actually failed? The false alarm, which erodes the channel from the other side.
3. What was raised but never delivered? An alert the ledger recorded as not sent.
4. Which job has no failure alert wired at all?
5. Has anyone been at the other end? The proof's verdicts, read as a streak.

The cross-reference is mechanical and tested. What it does not do is decide. It writes everything to one file and hands the file to the agent, and the judgement about what is strange is the agent's. Encoding "strange" as a rule is exactly the assumption that produced a green dashboard over a broken watchdog.

flowchart LR
  JL["Scheduler journal<br/>what actually ran and failed"] --> X["Cross-reference"]
  AL["Alert ledger<br/>what was sent, to whom, delivered or not"] --> X
  PR["Wire proofs<br/>was anyone there?"] --> X
  X --> Q1["Failed, nobody told"]
  X --> Q2["Alerted, had not failed"]
  X --> Q3["Raised, never delivered"]
  X --> Q4["No failure alert wired"]
  X --> Q5["Nobody answering"]
  Q1 & Q2 & Q3 & Q4 & Q5 --> F["One file → the agent decides"]

The first time it ran it found the watchdog. The first morning the alert wire itself went live, it found a calendar sync that had been failing every two hours since January, seven months, into a log file nobody read. We retired it that morning rather than repair it, which is its own lesson: half of what silence hides is not worth fixing, only worth knowing about.

What this costs, and what it buys

Five scheduled checks, a ledger, a forwarder and a sweep is not nothing. Most of it is small scripts with one job each, and the whole apparatus runs on the same server as the work it watches. What it buys is the difference between finding a failure by its consequences two days later and finding it within thirty seconds, with an agent already reading the logs.

The rule that generalises beyond our servers is the first one. Whatever you run, make every process leave a verdict every day, display a missing verdict as a failure, and then go looking for what still says nothing. The silence is where the incidents live.

What we do with this

At Pygocen this is the part of a build that clients rarely ask for and always end up needing: a job that runs unattended is only as trustworthy as the thing that notices when it stops. We wire it in from the first release, because retrofitting it after the first silent week is how it usually happens.

If you run scheduled jobs, a data pipeline, or an agent that works while you sleep, and you are not sure what it would take for it to fail quietly, that is a conversation worth having. Get in touch, or see how we build and run software end to end.