Your agents react

Part 1 was ship time: agents running the six-step SDLC. Part 2 was the org learning from itself. Both describe work you choose — you pick the project, you set the pace. Production doesn't ask. An alarm at 2am, a customer bug report, a disk that fills on a Saturday: reaction is the work that picks you.

Here's the claim this part defends: the remediation leg of an incident is just a ticket with a deadline. Not the whole incident — command, customer communications and the regulatory clock stay human work, and I'll come back to that. But the part engineers actually spend the night on reuses the machinery from Part 1, which means agents can climb every rung of the reaction ladder while humans keep the moments that carry authority.

But let me start with the story where our reaction system failed, because it's the most instructive thing we own.

The weekend the pager stayed silent

We did everything right, on paper. Synthetic checks that sign in and walk the core user journey every minute. Infrastructure alarms on the clusters. An incident platform with an on-call schedule, escalation paths and paging. Both dashboards green, both sides configured, every box ticked.

And for weeks, not a single alert got through. The webhook template passed the alert type through in a field where the receiving side accepts exactly two values — anything else is rejected at the door with a validation error. Checks fired, alerts were sent, and every one of them was silently dropped. We had even "tested" the path — with a hand-written request to the alert endpoint. That proved the endpoint and the routing worked. It proved nothing about what the monitoring tool actually sends, which is the only thing that matters.

We found out the way you always find out: a real production incident, one July weekend, that should have paged within two minutes — and didn't. It ran far longer than it had any right to.

Three lessons, each of which now lives in code:

1. Alerting configuration is code. Every check, alarm, schedule, escalation path and route is declared in the repo and applied by CI on merge. Dashboard edits drift; code doesn't. The same SDLC that ships the product ships the monitoring — reviewed, versioned, applied.
2. Test the path end-to-end, as the sender. A drill fires a real check failure and watches it arrive on a real phone. Anything less is testing your assumptions, not your system. The number we now track is embarrassingly basic and the only one that matters here: page delivery rate on a drill — it was effectively zero for weeks, and we don't get to call it fixed on any evidence weaker than a phone that actually rang.
3. Monitor the monitoring. A dead alert channel and a healthy quiet week render identically — the exact failure class Part 2's learning loops had: the system measured its activity, not its health. Silence is a signal you have to design for, because by default it looks like success.

The reaction ladder

When a signal does arrive, the work is a ladder with five rungs:

Agents can work every rung in 2026 — but support is the honest verb, because command never moves. What stays human: acknowledging the page, deciding severity, talking to customers, and writing to production, which sits behind the gate from Part 1. Rung by rung.

Detect — the QA agents from Part 1, pointed at production

Our detection layer is the same idea as the QA gate that clicks through every PR — agent-shaped checks that open a real browser, sign in and use the product — running continuously against production and demo. Under it, a couple of dozen infrastructure alarms. Above it, routing: production failures page a phone; staging noise goes to a Slack channel and pages nobody.

Two details that took real scars to learn. First, synthetic users are seeded fixtures, not real signups — a synthetic check must never send a customer SMS, trigger billing, or pollute analytics. Second, synthetic monitoring is metered, and the meters are not interchangeable. A single once-a-minute API check is 43,000 runs a month; we blew through our plan's allowance in early August by having eight of them. Price your paranoia: the cheapest check that carries the signal — a status assertion, not a browser journey — is the right one.

Triage — never trust a raw list of alerts

The alert channel lies to you. A snapshot from mid-August: sixteen firing urgent alerts, none resolved, the oldest six days old. Three of them screamed "disk above 80%" about volumes that were 4% full — pods we had deliberately stopped, whose missing metrics the alarm was configured to treat as breaching. Eight were "production deploy failed", including three for a release that had succeeded on its third attempt and left its earlier failures lit forever.

So the first triage rule our agents follow: snapshot before, diff after. Before any infrastructure work, list what's already firing; afterwards, attribute only what's new. A raw list attributes nothing. And the rule's mirror: a deliberate stop must carry its alarm — scaling something to zero on purpose and letting it page the on-call is a defect in the action, not in the alerting.

The investigation itself runs over the incident platform's MCP server — the agent pulls alert detail, linked incidents, escalation history, and answers "is this new, is it ours, is it real" before a human has found their glasses. For analytical questions it goes stats first, then list, then detail — never paginating through incidents to count them.

Diagnose and fix — the incident becomes a ticket

Once triaged, the incident stops being special. It becomes a Linear ticket and enters the Part 1 pipeline: an agent reproduces the failure in a browser, drafts a timeline from the logs, proposes the fix, and the PR faces the same gates as any feature — adversarial review if it's structural, the mandatory bug-hunt, the QA agent clicking through the repro. Incident code that skips review because "it's urgent" is how one outage becomes two.

The choreography matters more than the code. Our maintenance mode is one config file with three consumers: the app serves the wall, the monitors that can't pass a walled app deactivate, and the deploy pipeline skips the journey check that would false-alarm. The ordering is asymmetric — monitors down before the wall goes up, app answering before monitors come back, because reactivating a sign-in check against a walled app is a self-inflicted page two minutes later. Order-of-operations is exactly the thing agents never forget and humans at 2am always do.

The other inbox — bugs, requests, and the virtual sprint

Reaction isn't only alarms. Bugs arrive from teammates; requests arrive from customers; ideas arrive from the founders' group chat. In Part 1 the triage was one label: an agent can verify this, or a human is required.

Everything above this line runs today. What follows is the rung we're building now — described as a design, not a deployment, so you can tell which is which: nothing reaches human eyes before an agent has tried it.

• Auto-validation. A bug report lands and an agent immediately tries to reproduce it on staging. The ticket gains a verdict before anyone reads it: reproduced, here are the steps and the screenshot — or could not reproduce, here's exactly what I tried. Human triage stops being detective work and becomes reading verdicts.
• Authority follows the requester. Who filed the ticket determines what it may trigger. When Marton — VP Product — files a small, well-scoped feature, it can go straight into the sprint and an agent starts on it. An unknown reporter's ticket can trigger a sandboxed reproduction and nothing else. Two conditions make this safe rather than clever: the identity has to be authenticated by the tracker rather than typed into a field, and the mapping from identity to authority has to be a deterministic table, not a judgment the model makes while reading the ticket. A compromised account should widen the blast radius by exactly one person's authority — never to everything. (Part 4 shows why this is a security boundary, not a workflow nicety.)
• The virtual sprint. The end state: a backlog that organises itself. Validated bugs rank by measured impact, small features from trusted requesters schedule themselves, and the sprint becomes a standing negotiation between what arrived and the capacity you approve — from your phone, like everything else.

What's left for the human

Three things, and they're the right three.

The ack. The page goes to a person: primary has a minute to acknowledge before the backup gets woken too. But by the time you've opened the laptop, the agent has diffed the alert channel, pulled the logs, and drafted a timeline. You're not starting the investigation — you're reviewing it.

Command and communications. Someone has to decide this is a Major, decide whether customers hear about it and in what words, and hold the regulatory clock if one is running. No agent drafts a status-page update that goes out unread — a published maintenance window is permanent public history, and there is no delete endpoint for a sentence you regret.

The goal is not an org where nothing ever wakes you. It's an org where being woken is the last step of the machine's process — not the first step of yours.

And the write. Everything in this part reads production and fixes it through the pipeline. But sometimes something must change in production right now — and I'm on a beach, or asleep, or both. What happens then is a security question, and it deserves a whole part.

Part 4: Securing the agentic org — a free-fire staging zone, read-versus-write production, break-glass behind MFA, and why your ticket tracker is an attack surface.