I

I’ve distilled my experience into 5 principles for building agent systems that succeed inside a business, earn people’s trust and improve over time. This article explains each one through an agent system I built for a $100M+ managed IT provider, to address one of its most pressing problems at the time: client churn.

• Tailor the product shape to the business workflow
• Give each workflow step to the simplest mechanism that handles it well
• Redesign the human’s job at the agent handoff
• Design the agent handoff for human verification and feedback
• Build a self-improvement loop from human feedback

The case study: Detecting churn risk hidden across service tickets

The company runs IT operations for its clients and was facing unprecedented client churn at the time. One of the clearest warning signs is a service issue that keeps coming back for the same client. The problem is that this warning almost never arrives as a warning.

Instead, it arrives as separate incidents spread over weeks or months. Each incident looks small and self-contained. The commercial risk only becomes visible when someone notices that the same underlying problem keeps returning.

Spotting that pattern meant relying on busy people to connect the dots. Support technicians saw individual service tickets raised by clients. Account managers reviewed client touch points and dashboards. For the risk to surface, someone had to remember the earlier incidents, judge whether they formed a pattern, gather the evidence and alert the person who owned the client relationship.

Across many clients and a high volume of service tickets, connecting those dots by hand was prohibitively manual. The company couldn’t keep tabs on every client and every risk signal consistently at scale.

Tailor the product shape to the business workflow

The common instinct I see most teams follow is to put an agent system behind a chat interface. This company tried that route, too. The team first investigated churn through its enterprise data agent, a separate system I’d previously built for them so employees could ask questions about company data in natural language.

But in my experience, a chat interface is often the wrong product shape for most enterprise workflows, for 2 reasons:

• It waits for someone to think of the right question at the right moment. Business work usually starts because something happened, not because a person remembered to ask about it. In this case, early churn detection has to begin when new evidence arrives and keep running in the background.
• It makes employee adoption a prerequisite for the business outcome. For adoption to just work, the agent system has to run without anyone needing to drive it. Ideally, it should also require minimal change to existing behaviour.

The company’s churn risk workflow also depends on patterns that build up over time. To run efficiently, the system should retain what it has already learned about each client’s ticket history instead of rebuilding that history in response to every question. For example, a stateful background process can store each client’s issue history and continue from where it left off, so every new run handles only the tickets that arrived since the last one.

You could bolt those capabilities onto a chat-based agent with an agent skill that explains how to store findings and process only new tickets. However, once that capability carries persistent state and fires on a business event, it already operates as a separate system. Hiding it behind chat doesn’t simplify the system, it only disguises a background workflow as a conversation.

As such, I guided the team away from treating churn detection as another question-and-answer use case for its chat-based enterprise data agent. Instead, I built a separate, event-driven agent system. This choice shifts the system’s job from helping employees inspect churn risk to owning the detection work itself. The system watches for new evidence, carries each client’s ticket history forward and starts the right workflow without waiting for someone to prompt it.

Give each workflow step to the simplest mechanism that handles it well

The mistake I see teams make most often is handing an agent the entire workflow. For example, a value needs comparing, so the agent compares it. Or a result needs routing, so the agent routes it. Putting everything in one prompt can look simpler, but in production it folds exact rules, ambiguous investigation and commercial judgement into one opaque LLM run.

The strongest production systems I’ve seen and built don’t use the most sophisticated agent at every step. They give each step to the simplest mechanism that handles it well, so the complete workflow stays accurate, understandable and cheap to run at scale.

The real design work is deciding which steps belong to deterministic code or a specialized method, which need an agent and which have to stay with a person. The agent is one decision-maker inside the system and goes where judgement is needed, not everywhere by default.

For this case study, the workflow takes this form: The system watches each client’s ticket history and groups incidents that might describe the same underlying issue. When a group crosses the company’s definition of recurrence, an agent investigates whether the tickets represent one continuing problem and whether the problem deserves an account manager’s attention. Only a finding that clears both bars becomes an alert for the account manager who owns that client.

The agent system I built runs this workflow in 2 linked stages:

• A once-off preparation builds a compact picture of each client’s recent issues.
• A scheduled weekly run compares new tickets with that picture, investigates recurring patterns and escalates only the ones that earn attention.

A summarizer agent handles language judgement

During the preparation stage, the system pulls each client’s last 3 months of service tickets from the ticketing system. A single ticket can hold a long email thread between the client and a support technician, which is far more context than the rest of the workflow needs. To cut context size and noise in every downstream step, I compress each ticket to only its useful signal.

This is the type of task an LLM shines at. As such, summarizer agent reads the full ticket and reduces it to one concise issue summary while keeping enough detail to identify and compare the problem.

Semantic similarity groups tickets without using an agent

The next step groups tickets that might describe the same underlying issue. Similarity is probabilistic, so no exact rule can draw the line. But the system also doesn’t need an agent to reason through every possible pair, which would be slow and expensive at volume.

As such, I use semantic similarity here. The system turns each issue summary into an embedding, a numerical representation of its meaning, then uses clustering to build the initial issue groups separately for each client. This way, the system handles similarity directly, cheaply and through a repeatable method.

Deterministic code owns the recurrence threshold

The once-off preparation creates the initial issue groups. The scheduled weekly run comes next. Every Monday morning, the system pulls the previous week’s new tickets for each client and puts each one through the same summarize-and-embed steps. It then compares each new embedding with that client’s existing issue groups. A close enough match joins the ticket to that group, while a ticket with no close match starts a new group.

Deterministic code checks the company’s rule for X similar tickets within Y days. A threshold like that is exact by definition, so it belongs in code rather than an agent that might apply it inconsistently.

If the threshold is crossed, the system doesn’t yet send an alert to the account manager. Semantic similarity is only a cheap, quick first pass at grouping tickets that might be related. Crossing the threshold means the group might contain a recurring issue, not that the similarity match proves the client has one continuing problem. An investigation agent has to make that call.

An investigation agent judges the evidence

When an issue group crosses the threshold, the system starts an investigation agent for that group. The agent pulls the full history of every ticket in the group from the ticketing system. From there, it decides what else it needs and retrieves it from the same source, such as details about the client or the IT system at fault.

No rule or embedding can settle this part. The agent has to procure evidence and make 2 judgements: whether the tickets represent one continuing issue and whether the pattern warrants an account manager’s attention.

The 2 judgements stay separate because a repeated pattern and a useful risk signal aren’t the same thing. For example, planned maintenance can generate several related tickets without pointing to a damaged relationship, or a recurring technical fault can be genuine and still not deserve an account manager’s time. Only a finding that’s both real and commercially relevant becomes an account manager alert.

A human handles the decision to act

Deterministic routing in code emails the alert to the account manager assigned to that client. The agent system doesn’t prescribe what the account manager should do next. The account manager keeps the relationship judgement because the right response depends on the client and relationship context outside the technical evidence.

Across the complete agent system, code handles exact rules; embeddings handle semantic similarity; agents handle messy evidence and investigations; humans keep the decision with real commercial consequences. This division gives each step to the simplest mechanism that handles it well, so the complete workflow stays accurate, understandable and cheap to run at scale.

Redesign the human’s job at the agent handoff

Adding an agent system to a business workflow changes what the people in it do. One of the most important product decisions is where the system stops, what it hands over and what the humans still own.

An agent system usually ends in one of 2 ways:

• It completes the workflow, while people spot-check its work from time to time
• It stops before the final decision and gives a person all of the context needed to decide and act quickly

In the second case, the agent system is responsible for filtering what needs human attention, surfacing only those cases and bringing the relevant context with them. Done well, that turns 30 minutes of a human reconstructing what happened into a 1-minute decision. The system completes the searching, analysis and evidence assembly that come before the judgement call, then stops where a person’s authority still matters.

This is the company’s case, where the account manager still owns the client relationship. The system emails a consolidated, fully investigated risk alert to whoever is assigned to that client. This means the account manager no longer has to dig through tickets or piece together what happened. They can spend their attention on protecting the relationship using the evidence the system has already assembled.

But this handoff only works if the account manager can trust the system’s findings.

Design the agent handoff for human verification and feedback

The agent handoff is the final interface a person sees of the agent’s work.

Verification starts at product design, not at the end

A background agent system raises the bar for verification. With a chat-based agent, the user sees each response as it arrives. But a background system can repeat a bad judgement across many cases before anyone notices.

Teams often put most of their effort into the AI engineering, and the handoff gets whatever attention is left. That misses one of the highest-leverage parts of the system. A sophisticated agent is useless if the people receiving its output can’t trust it enough to use it, and trust comes from verification and traceability. If the person receiving the agent handoff has to redo the agent’s work to confirm its conclusion, the system has exposed the wrong evidence.

That leads to a simple design principle: Make the agent’s work easy to verify before building evaluations around its output. Verification can’t be added at the end as a score around an otherwise opaque answer. It has to shape what the system produces, how the output is broken down and which evidence travels with every conclusion.

In this case, the agent handoff is the account manager alert. I designed that alert so reviewable evidence is an intentional output, instead of just stating a risk level and leaving the account manager to reconstruct why.

Start with how a human does the work

Here, don’t start by asking what an alert normally looks like. Start with how the person receiving it already checks whether a finding is correct. This means understanding how an account manager confirms that a recurring client issue is real.

They need to see what the system believes is recurring, how often it happened, which tickets support the conclusion and what each ticket says. They also need access to the source record. This checking process determines the handoff interface design.

The evidence packet therefore opens with the client, the recurring issue and its frequency. It then shows the tickets that support the finding. Each ticket has a concise summary and a direct link to its full record in the ticketing system.

Here’s what an example alert looks like:

Reveal the evidence without overwhelming the human

This interface uses progressive disclosure. The account manager sees the finding and the most useful ticket evidence immediately. It shows 2 representative tickets and makes clear that 3 more belong to the group. The reviewer doesn’t have to read every raw ticket before deciding whether the pattern makes sense, but every conclusion stays linked to its source of truth in the ticketing system.

This design reduces reading without hiding provenance. When the evidence is incomplete or contradictory, the agent states that beside the affected ticket instead of presenting its conclusion as settled. The account manager can see what the system knows, what remains uncertain and where each claim came from.

Collect feedback at each agent decision point

Feedback helps most when it identifies which part of the agent system got something wrong. When a system contains several agent decision points, a generic thumbs-up or thumbs-down on its final output can’t tell you where the mistake occurred. That makes the system hard to improve.

Products such as ChatGPT and Claude use a generic rating on the final response because one interface has to support everything from financial analysis to legal work and frontend design. But a system built for one specific enterprise workflow has an advantage. Its important decisions are known ahead of time, so its interface can ask for feedback exactly where each judgement occurs.

The design goal is to make such evaluation part of normal use. Account managers don’t have to leave their normal workflow to label abstract AI outputs or repeat the searching and analysis that produced the alert. They make 2 quick calls while reviewing a finding they already care about:

• Is this ticket part of this recurring issue?This checks whether the system grouped the right client incidents together.
• Is this issue worth flagging to you?This checks whether a confirmed recurring pattern deserves the account manager’s attention.

The 2 questions stay separate because the agent system can group the tickets correctly and still be wrong that the pattern matters commercially. They also produce 2 different evaluation signals. Ticket-level answers show whether the system groups incidents correctly, and issue-level answers show whether a confirmed pattern deserves attention. Instead of one rating on the final alert, the team learns which decision needs improving.

Clicking any Yes or No opens an optional free-text box that captures the reason behind the human judgement, which is what the improvement process needs.

The easier this feedback is to give during normal work, the more likely people are to give it. Breaking one large finding into smaller units also makes annotation less expensive and produces more useful signals for future evaluation.

Let verification become lighter as trust grows

The right amount of visible evidence can change as users build confidence in the agent system. During an early rollout, the handoff interface can show ticket evidence and agent reasoning by default so people can inspect how the system behaves. As the system earns trust, some detail can collapse by default while the full evidence remains available.

The product should reduce the effort required to review the work as confidence grows, but it shouldn’t remove the ability to check the work.

Build a self-improvement loop from human feedback

After collecting feedback, the next valuable step is to build a loop that turns it into proposed improvements for future runs. In my experience, teams often automate this loop too early. They point a coding agent like Codex or Claude Code at the feedback and the agent system’s codebase, then ask it to make improvements.

But automating this loop isn’t trivial. It must learn which feedback warrants a change, where that change belongs, and how to make a useful change without weakening behaviour that already works. A generic coding agent doesn’t have the company’s domain knowledge, taste or priorities. As such, it can often miss the issues that matter.

Instead, I recommend building the first version of this loop by hand. Start with a batch of feedback small enough to review carefully. For each item, have a domain expert annotate whether it should change the system and, if so, what that should look like. These annotations turn the team’s implicit judgement into examples an automated loop can use later on.

Once you’ve defined what matters, a coding agent can then help group related feedback, find similar agent runs, propose updates to the harness, and gradually automate more of this loop. I’ve found that coding agents work best when they scale the judgement you’ve developed by reviewing agent runs and feedback yourself, not when they let you avoid looking at the data entirely.

Closing Thoughts

In this case study, the agent system I helped the company build turns a prohibitively manual client churn detection workflow into a feasible ongoing operation at >90% lower costs.

But I think cost savings understate the value of automating enterprise business workflows. The more durable benefit is that agent systems capture and compound a company’s operating judgement. Without a system to capture it, that judgement lives in people’s experience, discussions and one-off decisions nobody writes down. Once it sits inside an agent system, they start shaping how work is handled the next time.

Right now, a lot of attention goes to new model releases and how much smarter each one is. But every company can access the same models. Instead, I think what truly belongs to a company is its accumulated record of how its people decide what matters, what evidence they trust and when to act. That record is how a company builds an operating capability that gets more valuable, and more its own, the longer it runs.

I had a lot of fun writing this, and hope you had fun reading too! See you in the next one 🙂

Sheila