In my previous article, I described building a semi-autonomous SRE agent that investigates incidents and drafts fixes for human approval. This one is about the part of that build I have not told yet: how the agent knows what it knows, and how many tries it took to get there.

There is a pattern to AI DevOps demos. The agent gets a clean alert, queries a metric, finds an obvious anomaly, and produces a confident diagnosis. Everyone applauds. Then someone points it at a real environment and it confidently recommends restarting a service that has been deprecated for two years.

The gap between the demo and production is not intelligence. It is context.

An LLM knows what Kubernetes is. It does not know that your payments-api has a flaky liveness probe everyone ignores, that the checkout team owns the retry policy, or that the last three "database incidents" were actually cache misconfigurations. That knowledge lives in your runbooks, your postmortems, your architecture docs, and the heads of your senior engineers.

The agent I described in my previous article already pulled in runbook context when an incident required it. What I did not cover is how it got there. My first build had none of that knowledge, and my second attempt delivered it badly. Here is the path to runbooks plus retrieval-augmented generation, and what I learned about keeping an agent's knowledge honest.

The Two Knowledge Planes

An SRE agent needs two fundamentally different kinds of knowledge, and they should be architected separately.

The live plane is the current state of the world: metrics from Prometheus, container logs, Kubernetes events, deploy history, alert payloads. This data is fresh, factual, and machine-generated. The agent gets it through scoped, read-only tools at investigation time.

The organizational plane is everything your team knows that no dashboard shows: runbooks, SOPs, playbooks, postmortems, architecture docs, service ownership, escalation rules. This data is slow-moving, human-written, and full of judgment calls. It answers questions the live plane cannot: is this symptom normal for this service? Who do I page? What did we do last time?

My first build wired up the live plane and stuffed a summary of the organizational plane into the system prompt. That failed in two ways. The prompt grew until it crowded out the actual investigation, and it was perpetually stale because nobody updates a system prompt when they update a runbook.

Retrieval Instead of Stuffing

The fix was boring and effective: treat organizational knowledge as a retrieval problem.

All of it — runbooks, postmortems, architecture notes, ownership maps — lives as markdown in a git repository. A pipeline chunks the documents, embeds them, and loads them into a vector database. On every merge to main, the embeddings for changed files are rebuilt. The knowledge base is never more than one merge behind reality.

At incident time, the agent does not receive the whole library. It retrieves what the incident is about. A latency alert on checkout-api pulls the checkout runbook, the postmortems that mention checkout, and the architecture doc describing its dependencies. Nothing about the batch pipeline, nothing about the mobile gateway.

The investigation loop looks like this:

• Alert arrives; the agent identifies affected services from routing metadata
• Retrieval: runbooks, postmortems, and docs relevant to those services
• Live investigation: metrics, logs, deploy diffs through read-only tools
• Correlation: live signals interpreted in light ofretrieved knowledge
• If evidence is thin, retrieve more or escalate to a human

Step 4 is where the value concentrates. Raw telemetry says "cache hit ratio dropped." The retrieved postmortem says "we saw this exact pattern in March; the cause was a TTL misconfiguration; the fix was PR #1203." Those two together are a diagnosis. Either alone is a guess.

The Incident That Proved It

During testing, I injected a failure the agent had never seen: connection resets on a service that sat behind a third-party API. The live signals were ambiguous — error rates up, latency ragged, no recent deploy on the affected service itself.

The retrieval layer surfaced a postmortem from a previous incident that described the third-party provider's monthly maintenance window and its signature: connection resets starting precisely on the hour. The agent checked the timestamp, matched the pattern, and its response changed from "probable network issue, investigating" to "this matches the vendor maintenance pattern documented in the June postmortem; recommended action is the mitigation from that document; no code change needed."

That answer did not come from the model being smart. It came from a two-paragraph postmortem someone wrote months earlier, retrieved at the right moment. The agent's job was recognizing that the document and the telemetry described the same event.

Keeping the Knowledge Honest

RAG introduces a new failure mode: the agent is now only as good as the documents you feed it. Three rules kept mine trustworthy.

Runbooks live in git, next to the services they describe. Updating the agent's behavior means merging a PR, which means code review. Tribal knowledge gets the same rigor as code. Stale runbooks get caught in review, not at 3 AM.

Postmortems are the highest-value documents in the corpus. They encode symptom-to-cause mappings that exist nowhere else. My agent writes a structured postmortem after every resolved incident — published to Confluence for humans, with a markdown copy committed to the git corpus for the agent. Every incident makes the next investigation smarter. The knowledge base compounds.

Retrieved content is context, not command. A runbook that says "restart the service immediately" does not make the agent restart anything. Retrieved documents inform the diagnosis; actions still go through the same scoped tools, validation hooks, and human approval I described in the previous article. This matters for safety — documents can be wrong, outdated, or in a worst case, poisoned. The knowledge layer gets no authority, only influence.

What Still Breaks

Honesty section. Three problems I have not fully solved:

Retrieval misses. If an incident's vocabulary does not match the runbook's vocabulary, the right document does not surface. An alert about "connection pool exhaustion" will not retrieve a runbook titled "database saturation issues" unless the embeddings happen to bridge the gap. I now write runbooks with symptoms in the first paragraph, which helps but does not eliminate it.

Conflicting documents. Two runbooks, written two years apart, recommending opposite mitigations. The agent has no way to know which one is current unless the corpus tells it. I added a last_validated date to every runbook header and taught the retrieval layer to prefer recency — a crude fix for a real problem.

Confidence laundering. A retrieved document can make a weak diagnosis sound strong. "As documented in the runbook" is persuasive even when the runbook is only loosely relevant. The agent now cites which documents shaped its hypothesis, so the human reviewing the diagnosis can check whether the citation actually supports it.

The Takeaway

If your AI DevOps agent works in demos and fails in production, the missing ingredient is probably not a better model. It is the organizational knowledge your team already wrote down, retrieved at the right moment, and kept honest by the same review process you use for code.

Live telemetry tells the agent what is happening. Runbooks and postmortems tell it what that means here, in your system, with your history. An agent with both is a useful colleague. An agent with only the first is a demo.

If you have built a knowledge layer for an agent — especially if you have solved retrieval misses better than I have — I would love to hear how.