Your agentic org self-improves

An agentic org has no institutional memory. That's the problem nobody warns you about after Part 1's six-step SDLC with agents doing every role — and left unfixed, it quietly caps everything else.

When an engineer hits a nasty gotcha — the migration tool that generates broken SQL, the CLI pin that can't talk to any deployed cluster — they remember it. They mention it at standup. It becomes tribal knowledge. When an agent hits the same gotcha, it burns twenty minutes solving it brilliantly… and then the session ends, and the knowledge dies in terminal scrollback. The next agent burns the same twenty minutes next week.

Human orgs fix this with retros. So we built one for a workforce of agents, and it has been running ever since — capturing what each session learned, batching those lessons, and turning the recurring ones into documentation every future agent reads. The compounding is the point: the gotcha that cost twenty minutes in week one is a line in an agent guidance file by week two, and nobody pays for it again.

Here is the whole loop, including the embarrassing parts — because the day we switched it on, it found five bugs in itself.

The principle: collect cheap, decide together

Everything below runs on one split I keep coming back to: collection is cheap and asynchronous; processing and implementation are collective and thought out.

Capture must cost nothing, or it stops happening — no ceremony, no meeting, no form. Judgment is expensive, so it runs on a cadence, in batches, where patterns are visible. And decisions about what to build stay human, because that's the part where taste and strategy live.

The corollary that keeps it safe: no LLM in these loops runs in CI, and no judgment runs on a cron. Scheduled jobs are deterministic — count things, compare dates, file tickets. All the thinking happens in agents that a ticket hands work to, and all the deciding happens with me.

Four layers, four frequencies

every merge

Capture

A hook reflects on the session transcript. One tagged what / why / evidence entry — or NO-ENTRY.

→ ticket comment · agent, async

weekly gate

Consolidate

5 pending entries, or one 14 days old, files a consolidation request. Clusters become small doc PRs.

→ doc PRs · agent, reviewed

every 2 weeks

Sprint retro

One report: metrics row, friction by tag, ticket ages — and an audit of the last retro's decisions.

→ max 2 investments · human decides

every 6th retro

Spring clean

The pruning pass: stale rules, tickets nobody actioned, lessons that stopped recurring.

→ deletions · proposed, then decided

12sessions backfilled

5lessons captured

8said NO-ENTRY

1batch gate tripped

5defects caught day one

Seen from the outside, the org has exactly three exits for anything that goes wrong, and every service and every scheduled check sits on one of them. A fault a customer can feel rings a phone. A boring, deterministic finding becomes a ticket an agent picks up. A lesson a session learned becomes a rule the next session reads. Nothing dead-ends in a person's inbox.

Every signal has exactly three exits

1. A phone ringsa customer can feel it
   1. Sentryreports its own failures
   2. Checklysynthetic journeys, minutely
   3. CloudWatchcluster and database alarms
   4. incident.ioroutes, groups, opens incident
   5. On-callack in a minute, else backup
2. An agent picks it upboring, deterministic, daily
   1. 22 detectorscount things, compare dates
   2. Linearone ticket each, needs-cursor
   3. Agent → PRa human merges
3. A rule gets writtenthe org remembers
   1. Merge hookreflects on the session
   2. Engineering logentry, or NO-ENTRY
   3. Consolidation5 pending, or one 14d old
   4. Doc PRa human decides

Loop 0: the maintenance detectors (already running)

Before the retro existed, we had the base layer: nineteen deterministic detectors on a daily heartbeat. Each one checks something boring — stale staging environments, docs drifting from code, dependency advisories, unpinned CI actions, ISO evidence going stale — and files findings as one Linear ticket per detector, updated in place. Labelled needs-cursor, an agent picks it up; the heartbeat itself never thinks. (More have joined since, and the list below is current — the reason the twentieth exists is further down, and is the most expensive thing in this article.)

Grouped by what they guard, they are the maintenance an agentic org needs and a human org does by habit and standup:

The 22 detectors, by what they guard

1. 1Production faults

   What the product reports about itself — the one signal that cannot be green while a customer flow is dead.

   • sentry-triageissues that are new, still recurring after a week, or hot and unassigneddailyfiles a ticket
2. 2Environment hygiene

   Agents create environments faster than anyone tears them down.

   • env-cleanupstaging environments nobody is usingdailyfiles + acts
   • capacity-healthpods running hot, and whether the reserved instances still match what runsweeklyfiles a ticket
3. 3Spend

   The axis nobody instrumented until an invoice did it for us — and the one where a green signal is most likely to be structurally green rather than healthily so.

   • cost-reviewdaily spend per service, per-log-group ingestion, retention, and credit coverage falling awayweeklyfiles a ticket
4. 4SDLC health

   Is the pipeline still doing what the process says it does?

   • repo-healtha metrics digest: merges, reverts, review lagweeklyfiles a ticket
   • qa-healthwhether the QA agent still finds defects, and how long its verdicts takeweeklyfiles a ticket
   • ci-hygieneunpinned actions, implicit permissions, rerun-heavy workflowsmonthlyfiles a ticket
   • board-hygienetickets the pipeline lost: shipped but parked, ready with no PR, in review after the mergeweeklyfiles a ticket
5. 5Security posture

   Drift is silent; a baseline is only a baseline if something diffs against it.

   • security-checkdependency advisoriesmonthlyfiles a ticket
   • cloud-posturethe live AWS and GitHub security baselinemonthlyfiles a ticket
   • credential-rotationcredentials past their rotation datemonthlyfiles a ticket
   • access-driftwho can push, merge and administer, versus the doc that says who shouldmonthlyfiles a ticket
   • biz-logic-securityauthorisation and validation tripwires in the business logicmonthlyfiles a ticket
6. 6Compliance evidence

   An audit is a question about the past; the evidence has to exist before it is asked.

   • evidence-collectre-collects live ISO evidence and files what it could not provemonthlyfiles + commits
   • iso27001-auditthe quarterly ISMS and risk-register review is duemonthlyfiles a ticket
   • restore-drillthe backup restore has been rehearsedmonthlyfiles a ticket
   • supplier-registersuppliers visible in the repo that the register does not namemonthlyfiles a ticket
   • assuric-syncdocs pending publication to the compliance platformweeklyfiles a ticket
7. 7The loop's own upkeep

   The retro machinery is code too, and it rots like any other.

   • engineering-logenough session lessons are pending to consolidateweeklyfiles a ticket
   • sprint-retrofiles the retro prep report and audits the last decisionsfortnightlyfiles a ticket
   • docs-freshnessdocs that reference moved or removed codeweeklyfiles a ticket
   • skill-refactoragent skills that grew oversized or malformedmonthlyfiles a ticket

The design lesson came from the first one. env-cleanup politely filed the same ticket every day; nobody acted; staging ran out of capacity to schedule anything. Reporting alone doesn't work — so that one detector earned the right to act (it dispatches the teardown pipeline, which has its own guards). Every other detector still just files tickets. A detector earns teeth by demonstrating that the ticket alone failed.

Loop 1: capture — every merge teaches the system

The capture layer. The moment a session runs the merge command, a hook spawns a background reflection over the session transcript: did anything in here cost time that a doc, skill, or process change would have prevented?

The reflection reads the transcript inside the same session that produced it, and emits only its own summary — no transcript text is stored, shipped anywhere, or attached to the ticket, and the repo's secret-scanning rules apply to what it writes just as they do to code. Its entire output is either one entry — what happened, why it cost time, evidence, proposed fix, plus one or two tags from a fixed vocabulary — or the literal string NO-ENTRY. A tiny script posts it as a comment on a standing Linear ticket. The model never touches Linear; the script never judges.

We backfilled it over the last two weeks of real sessions:

ENTRY    4e4ddce3   NO-ENTRY 0bc71d72   NO-ENTRY 5746fd9d
ENTRY    449ba2fa   NO-ENTRY 2dca2efe   NO-ENTRY bdd009f5
ENTRY    faf5c17e   NO-ENTRY ee1779b9   NO-ENTRY 9219e621
ENTRY    1f173552   NO-ENTRY 22edcb9b   NO-ENTRY 4c760709
backfill complete: 12 transcripts considered

Four lessons, eight NO-ENTRYs — plus a fifth the system had already captured about itself while we tested the capture prompt. The NO-ENTRYs are the feature. Most sessions have nothing durable to teach, and a reflection that admits it is what keeps the log worth reading. The five that survived were all real: a .gitignore append that silently fused two lines because the file lacked a trailing newline; a database migration tool whose generated enum migrations fail outright in Postgres; a CI run that a GitHub outage left simultaneously "already completed" and "already running".

Loop 2: the batch gate — nothing acts on one entry

A weekly detector counts pending entries. Five entries — or one that's been sitting two weeks — files a consolidation request and hands it to an agent: cluster the entries, dedupe against existing docs, and turn recurring lessons into small doc PRs against the repo's agent guidance. Normal review, normal merge.

Why batch? Because "three sessions hit the same friction" is a signal no single session can produce. Acting on individual entries is how you get doc churn nobody reads. The noise bar lives at the batch, not at capture.

And this is the part that compounds: in Part 1 I said we keep ~30 versioned skill files in the repo instead of vendor "memories". This loop is what writes them. Lessons from agent sessions become docs that every future agent session reads — the org's knowledge base maintains itself, one reviewed PR at a time.

Loop 3: the sprint retro — deciding what to build

Every two weeks, one report lands on a standing ticket. It opens with a metrics row — merged PRs, entries captured, consolidations, open automation tickets, each windowed since the last report, so the ticket's comment stream is the time series. No dashboard to build, no dashboard to forget.

Then the sections: high-friction points (log entries clustered by tag, plus every open automation ticket with its age — our first report showed six tickets sitting unprioritised for a month, which is the retro making its own case), routine upkeep like version bumps, and — the part I stole from the retrospective literature — the audit comes first: the report quotes the previous retro's decisions and asks what actually shipped.

A prep agent digests all of it into a ranked one-pager: at most three friction clusters, at most two candidate investments, and a papercut list where every item is under a day of agent work. Then a human decides, in a comment, which the next report quotes back. The two ways retros reliably die: long unowned action lists, and a ritual nobody ever reviews. The two-investment cap and the audit-first rule are the guards. Every sixth retro adds spring cleaning: prune stale rules, close ignored tickets, delete docs for lessons that stopped recurring. Rule files are trained weights; nobody schedules the pruning by default.

It debugged itself on day one

Here's the honest bit, and the part I'd lead with if I were selling you nothing.

Day one, the system caught five real defects in itself — three via the mandatory Bugbot gate from Part 1, two via sceptical reads of its own output. Four of the five were the same failure class: a text system matching text it wrote itself. The routing label attached before any batch was due. The decisions-audit that would have quoted a report back as a decision, forever. And my favourite: the consolidation-request comment mentions the consolidation marker in its instructions — so substring matching counted the request as the consolidation, and the batch reset itself the moment it was declared due. The first metrics row proudly recorded a consolidation that never happened.

Each fix shipped the same day with a regression test, through the same gates as any other PR. And the first entry the system ever logged — parse structured output by structure, never by position, because harness noise leaks into everything — is the exact lesson those fixes encode. The learning system's first lesson was about itself.

That's also your answer to "how do I trust this?" — the same answer as Part 1. You don't trust it; you gate it, measure it, and read its first outputs with maximum suspicion. The metric that keeps everyone honest long-term: does a lesson recur after its doc fix merged? A recurrence means the fix didn't work. That's the difference between a learning system and a diary.

Round two: adversarial agents vs. the learning system

We didn't stop at shipping it. The same afternoon we ran the repo's multi-agent review pattern against the pipeline itself: five finder agents, each with one adversarial lens — Goodhart gaming, silent-death paths, prompt signal quality, human factors, lean-ness — working blind to each other, each capped at three evidence-backed findings. Then frontier-model refuters, each briefed to kill a finding by reading the actual code.

Nineteen agents, six minutes, ~1.6M tokens — call it the price of a decent lunch, which is the whole reason this shape is affordable at all. Reserve it for systems you are about to trust; it would be absurd on a copy change. Fifteen findings came back; the refuters killed three of them — one by empirically probing the live ticket API, another by opening all fourteen of the day's capture transcripts and checking the claim against reality. That 20% rejection rate is the whole argument for the two-stage shape: without the refuters, three plausible-sounding non-bugs would now be "improvements" in my backlog.

The theme of what survived: the pipeline measured its own activity but not its own health. A dead capture hook rendered identically to a healthy quiet fortnight. Nothing actually notified a human that a retro was waiting. A consolidation could be marked done by an agent that dropped every entry — indistinguishable, to every downstream signal, from one that shipped real doc PRs. All fixed the same day: the metrics row now flags zero-entry fortnights, the retro ticket assigns itself to a human, and a consolidation close gets audited — the detector parses the disposition lines and resolves every claimed PR against GitHub before believing the batch happened.

Two details I'd frame on the wall. The review bot caught that my "assign the ticket" fix had the exact same bug class it had flagged on day one — the reviewer gate now recognises this system's signature failure mode faster than I do. And the seven verified-but-minor findings weren't dropped: they were posted to the retro ticket as its first papercut candidates. The adversarial sweep's leftovers became the retro's input. The loop closed on itself.

Round three, in one paragraph

Two days later we ran the sweep again with five fresh lenses and every known finding pasted in as an exclusion, so the agents could only spend tokens on novel signal. Ten findings, all new. The headline was nasty and belongs in Part 4: a log entry that quotes a consolidation verbatim carries the marker line, so posting it silently reset the batch and fabricated the audit tally — untrusted text crossing a trust boundary, inside our own system. This time the refuters had nothing to kill outright, so they earned their keep differently: they killed two proposed fixes, remedies that sounded right and provably weren't, before either reached a PR. Then the review bot found a defect in one of this sweep's fixes. A fix for a fix for a fix, each caught by a different gate — which sounds absurd until you notice that no layer of it involved me finding the bug.

The dimension nobody instrumented

Four days after this post went up, AWS billed us $6,973 for a month that should have cost about $1,600.

One storage pod had been left at its default log level. Not a bug — a default. At info, the service writes a separate line for the entry, exit and close of every internal operation, and a single file listing costs dozens of them. That was harmless for months while the lines went to a file nobody read. Then a perfectly sensible fix — the log volume was filling up, so send the logs to stdout instead — connected the firehose to a metered pipe. Roughly two thousand lines a second, around the clock, for twenty-nine days. 7,649 GB, which is 61% of that month's entire infrastructure bill, from a pod that was working exactly as configured.

Nineteen deterministic detectors were running the whole time. A learning loop was capturing lessons on every merge. Three adversarial sweeps had gone looking for what the pipeline was blind to, with fifteen lenses between them. Not one of them had a dollar sign anywhere in it.

That's the lesson, and it's not really about AWS. A self-improving system improves along the axes it measures, and is perfectly blind on the others. The blind spot wasn't a bug in any loop — every loop worked. It was a missing dimension, and no amount of adversarial review inside the pipeline was ever going to surface it, because the reviewers were briefed on the pipeline. Round two's finding was that the pipeline measured its own activity but not its own health. This is the same shape one level up: the org measured its own health but not what it spent.

The second half is nastier, and it's the part I'd want you to take away even if you never run a detector. We were on promotional credits. From February to July they covered 100% of usage — every single month netted to zero. Usage nearly doubled in July and the invoice still said nothing was due. So the signal wasn't missing. The signal was there, working correctly, reporting fine every month while the underlying thing ran away.

That is worse than having no signal at all, because it manufactures confidence. A dashboard that says nothing is wrong is indistinguishable, to a busy founder, from a dashboard that cannot see anything. If you take one thing from this: find the metrics in your business that currently read "fine" for a structural reason rather than a healthy one. Free tiers. Trial credits. A retry that hides a failing dependency. A queue that drains because nobody is filling it. Every one of them is a green light wired to nothing.

The fix was the same shape as everything else in this article, which is either reassuring or damning depending on your mood: a twentieth detector, deterministic, weekly, files one ticket. It compares each day against the twenty-eight before it — never month over month, because a month-over-month check on a $190/day runaway is worth $5,700 by the time it fires. It reports credit coverage falling away as a finding in its own right. And it states, in the ticket, which accounts it could not see, so nobody reads a partial sweep as an all-clear.

Then, inevitably: the first time I ran it, it reported no missing budget on an account I had just confirmed had none. The AWS command prints nothing and exits successfully when no budget exists, so "empty" was being parsed as "the call failed" and the finding was swallowed. The detector reported no missing budget precisely because none existed. A passing output indistinguishable from a failing one — the exact defect class the thing was written to catch — caught only because I ran it once and compared its answer against a fact I already knew. The reviewer then found a second one of the same shape in the same file. That is now four separate occasions in this article where a system built to catch a failure class shipped with that failure class in it.

I've stopped finding this embarrassing and started treating it as the strongest argument for the pattern. The bugs are not evidence the loops don't work. They're evidence that this class of mistake is so easy that the only sane response is to assume it's in your code and build the gate that catches it. What the loops buy you is that it gets caught on day one by a bot, rather than in month seven by an invoice.

Tuition for this particular lesson: about five thousand dollars.

Steal this

The whole thing is four small pieces, none clever. If you want to run it next week:

1. Pick the moment of truth. Ours is the merge command, because a session that merged has finished something. Hook it to spawn one background reflection over the session transcript.
2. Constrain the output to one entry or the word NO-ENTRY. What happened, why it cost time, evidence, proposed fix, one or two tags from a fixed list. Nothing else. The refusals keep the log readable.
3. Post it with a script, never the model. A dumb script appends the entry to a standing ticket. The model never holds a credential; the script never forms an opinion.
4. Gate the batch, not the entry. A deterministic job counts pending entries and only acts at five, or at one that has waited a fortnight. Then an agent clusters them, dedupes against existing docs, and opens small doc PRs through your normal review.
5. Audit before you plan. Every report opens by quoting the last report's decisions and asking what actually shipped. Cap new investments at two. Prune on a schedule, or your rule files rot.

It cost one day across five PRs, most of which was the system finding its own bugs — which, again, is the point.

If your agents ship fast but every session starts from zero, you don't have an engineering org. You have a very expensive Groundhog Day.

---

Part 3: Your agents react — incidents, outages, and the reaction ladder. Part 4: Securing the agentic org — blast radius, break-glass and prompt injection.