Intro
In my organization I’ve worked as a backend engineer and architect. My main responsibility is ensuring the services we design meet their functional requirements, but also scale to millions of requests per minute, hold a 99.99% uptime, and stay cost-effective enough to keep OPEX in check. For most of my career, the traffic hitting those services followed the trend I could reason about — human-driven, forecastable, self-limiting. That’s changed. Agent traffic doesn’t behave that way, and the two dominant scaling models (on-demand and serverless) we’ve built both break under it. In this article, I walk through the mindset shift needed when scaling for agentic traffic — and the specific patterns worth considering.
If you’ve been in AI engineering, you’ve watched this shift happen. The traffic hitting your endpoints, your model gateways don’t look the way traffic used to look, or the way human traffic used to behave. Agentic traffic comes in unpredictable bursts. It repeats itself and retries relentlessly. It drains your scaling costs significantly compared to human-shaped traffic.
I want to walk through why that assumption is now broken, why the two dominant scaling models (on-demand and serverless) we’ve built inherit the problem, and what the solution to the problem should look like.
The assumption underneath everything
Every scaling framework we’ve built assumes traffic looks the way people generate it. Compare what’s on the left of this table with what’s now on the right
| Dimension | Human-driven traffic | Agent-driven traffic |
| Shape | Diurnal curve with forecastable peaks. Tomorrow looks like today. | No schedule. Bursts triggered by orchestration events, not clocks. The pattern itself shifts as agents and prompts change. |
| Onset speed | Ramps over seconds to minutes; you can watch it build. | Near-instantaneous. A parallel fan-out or a tight loop reaches full rate in milliseconds — faster than reactive scaling can respond. |
| Concurrency | Independent users; the aggregate smooths out by the law of large numbers. | Correlated fan-out from a single trigger. One orchestration spawns many synchronized calls. No statistical smoothing. |
| Retries | Bounded. People give up, refresh occasionally, back off out of frustration. | Programmatic and relentless. Without an explicit retry budget, an agent turns one fault into a retry storm. |
| Latency tolerance | Sub-second or the user abandons. | Often tolerant of seconds to minutes — reasoning runs in the background. This slack is exploitable. |
| Cost driver | Request count roughly tracks cost. | Request count is decoupled from cost. One heavy reasoning chain can consume more compute than a thousand lightweight calls. |
| Failure mode | Graceful degradation — users drop off. | Self-amplifying. Loops drain resources and, on serverless, bill you for every redundant call before any signal fires. |
Agentic traffic violates all seven of these assumptions at once. That’s why the answer isn’t a better version of either model we already have — it’s a fundamentally different place to put the intelligence. Let me show you what I mean by walking through how the discipline of scaling has evolved.
Generation 1: Anticipation (on-demand instances)
Earlier in my career, working on large-scale video streaming backends serving millions of concurrent viewers, capacity planning was a human exercise. When a major live event was about to kick off, we knew exactly when the spike was coming, roughly how steep it would be, and when it would flatten. The work was in the anticipation: pre-warming EC2 fleets days ahead, setting min/max autoscale bounds, staffing a war room through the event.
The traffic was human-shaped. It had a curve, a peak, a tail. You could reason about it and prepare for it.
I used to spend hours in war rooms. It used to start hours before the event, making sure the EC2 fleets were all pre-configured, instance types were updated, health checks were working fine, load balancers were all good, and the network was in good health. We used to constantly monitor the spikes in call volumes and the failures. There were instances where the demand anticipation did not fit well because of misconfigured client-side call volumes, which led us to change the auto-scaling group policy during the event and even absorb a brief period of failures. Once the event was over, we’d see the traffic fall, and then after the event we’d need to fall back to previous capacity that was ramped up — to save on OPEX.
Generation 1’s core assumption: the spike is forecastable, so provision ahead of it.
Generation 2: Reactive trust (serverless)
Then we started building serverless, which changed the game — API Gateway, Lambda, Step Functions, and the provisioning conversation largely disappeared. You stopped pre-warming and started trusting the platform to react. That worked because traffic was still mostly human-driven: users open the app, navigate, interact, and close it. Demand was still predictable, and platform reaction time was fast enough because onset was gradual.
Generation 2’s core assumption: you don’t need to anticipate, because the platform reacts faster than demand ramps.
The pivot: why machine orchestration breaks both at once
Non-deterministic machine orchestration — autonomous agents, multi-step tool-calling chains, retrieval loops — breaks both models simultaneously.
- It defeats Gen 1because there is no schedule to anticipate. Agent traffic has no clock and you can’t pre-warm for a spike you can’t predict.
- It defeats Gen 2because reactive scaling is alaggingsignal. Agent traffic reaches full rate in milliseconds; by the time CPU-based autoscaling fires, you’re already degraded. Worse, serverless faithfully executes every redundant call in a runaway agent loop — and bills you for the dysfunction.
To build and respond to agentic requests, the four-layer response below covers the key patterns.
Layer 1: Behavior-based scaling
You need to stop scaling on CPU usage. It’s a lagging signal and by the time it crosses a threshold, an agent loop has already drained the pool or run up the bill for dysfunction. The signal you want to monitor is request velocity and shape. Near-identical requests from one caller are an indicator of an agent loop to be checked long before it shows up in aggregate CPU metrics. An agent (retry, misconfigured) can send hundreds of near-identical requests in seconds. By the time CPU picks up the signal the loop cycles may already have degraded performance or wasted real money on serving those duplicate calls.
The pattern below uses request velocity + payload diversity to figure out whether caller X is in a loop, so you can quarantine them before they burn out CPU cycles.
```
import time
from collections import defaultdict, deque
class AgentLoopDetector:
"""Flags runaway agent loops by request velocity and payload repetition,
well before aggregate CPU reflects the load."""
def __init__(self, window_s=10, rate_threshold=50, diversity_threshold=0.2):
self.window_s = window_s
self.rate_threshold = rate_threshold
self.diversity_threshold = diversity_threshold
self.events = defaultdict(deque) # caller_id -> deque[(ts, payload_hash)]
def is_looping(self, caller_id: str, payload_hash: str) -> bool:
now = time.monotonic()
q = self.events[caller_id]
q.append((now, payload_hash))
while q and now - q[0][0] > self.window_s:
q.popleft()
rate = len(q)
if rate < self.rate_threshold:
return False
unique = len({h for _, h in q})
diversity = unique / rate
return diversity < self.diversity_threshold
Feed the boolean into an isolation decision
detector = AgentLoopDetector()
if detector.is_looping(caller_id="agent-1", payload_hash="test1"):
quarantine(caller_id="agent-1")
```
Layer 2: The AI gateway as a shock absorber
A traditional gateway counts HTTP/REST request-response traffic. An AI gateway meters cost for LLM interactions and prompt handling — it prices each call in tokens or compute and throttles a specific connection that exceeds its budget before it reaches your core inference systems.
Two capabilities that matter here: per-connection cost throttling (based on the token quota), and semantic caching — caching on prompt similarity, so repetitive agent queries never hit the model at all. Semantic caching is powerful for absorbing traffic shocks, but you need guardrails around it: a cache can return stale or hallucinated answers, and worse, it can leak one user’s private response to another if the cache keys aren’t properly scoped.
The example below is the middleware layer sitting between the caller and the model. On every incoming request, it decides:
- Can I skip the model? (cache hit)
- Can the caller afford it? (cache miss and execute)
- Do it and remember it (build the cache)
If the cache hits, return the stored response — that’s zero model cost, no token usage, no latency.
```
def gateway(request, ctx):
# Semantic cache: match on embedding similarity, not URL
hit = semantic_cache.lookup(request.prompt, threshold=0.95)
if hit:
return hit # absorbed at the edge, zero model cost
# Price the call and check the caller's remaining cost budget
est_cost = estimate_token_cost(request.prompt, request.model)
if not ctx.budget.can_afford(request.caller_id, est_cost):
return Response(
status=429,
headers={"Retry-After": ctx.budget.reset_in(request.caller_id)},
body="connection cost budget exceeded",
)
resp = forward_to_model(request)
ctx.budget.debit(request.caller_id, resp.usage.total_tokens)
semantic_cache.store(request.prompt, resp, ttl=3600)
return resp
```
Semantic caching trades exactness for absorption — you need a similarity threshold high enough to avoid returning wrong answers, and a bypass path for calls that must be fresh. In enterprise settings, where multiple people across a team often work on similar projects and end up sending similar prompts, this is a huge benefit. Semantic caching with the right guardrails can save significant money and still keep response times fast.
Layer 3: Async queuing
Human interactions expect sub-second responses. Autonomous agents usually don’t. A human at a payment gateway expects the transaction to complete immediately when the order is placed — that’s not the same expectation as an agent-facing API, which can run in the background as async queues. The SLAs for human APIs and agent APIs are fundamentally different, and shifting to async patterns helps flatten the spikes and remove the expectation of a synchronous hammer.
The example below shows the async queue pattern. It has a sender side and a worker side, which work in a decoupled way. Once your backpressure signal is reached, admitting more would make it worse. Every incoming request first checks how many jobs are sitting in the queue. If the queue’s already full, reject the request with the signal (HTTP 429). It’s an important signal to the client to slow down with a backoff strategy rather than queueing them.
```
QUEUE_HIGH_WATERMARK = 10000
def submit(request):
depth = queue.approx_depth()
if depth > QUEUE_HIGH_WATERMARK:
# Backpressure: tell the caller to slow down instead of queueing infinitely
return Response(
status=429,
headers={"Retry-After": backpressure_delay(depth)},
body="system saturated, retry later",
)
job_id = queue.enqueue(request.payload, caller_id=request.caller_id)
return Response(status=202, body={"job_id": job_id, "poll": f"/result/{job_id}"})
Worker side: pull at a controlled rate; concurrency caps protect downstream
def worker_loop():
for job in queue.consume(max_concurrency=200):
result = process(job)
results.put(job.id, result)
```
Layer 4: Token-based admission control
Instead of counting requests in aggregate, the intent is to shift the unit of admission from request count to resource cost. Don’t cap calls per minute; cap the compute a session can consume.
A token bucket keyed on session — debited by actual tokens or compute used, not by call count — lets a heavy reasoning chain that consumes disproportionate compute be cut off, while lightweight callers pass freely.
Below is an example of SessionTokenBucket, which implements per-session admission control by token cost, not request count. Each session gets its own bucket of capacity_tokens (default 100,000) that refills at refill_per_s (default 1,000 tokens/second). The admit() method estimates the cost of an incoming call and either debits the bucket if enough tokens are available or rejects the call.
The core mechanism is time-based refill: when a session tries to admit, _tokens() computes how many tokens have accrued since its last activity, capped at the bucket size. This lets an idle session build up capacity, while an active session gets throttled proportional to its consumption.
The usage is straightforward — if admit() returns False, respond with an HTTP 429 (Too Many Requests) telling the caller their session budget is exhausted. Lightweight callers keep passing through unaffected; a session running a heavy reasoning chain gets cut off before it drains resources everyone else needs.
```
import time
class SessionTokenBucket:
"""Admission by resource cost. Capacity and refill are in tokens (compute),
not requests — so one heavy reasoning chain can be rejected while many
light calls pass."""
def __init__(self, capacity_tokens=100_000, refill_per_s=1_000):
self.capacity = capacity_tokens
self.refill = refill_per_s
self.state = {} # session_id -> [tokens_available, last_refill_ts]
def _tokens(self, session_id):
now = time.monotonic()
avail, last = self.state.get(session_id, (self.capacity, now))
avail = min(self.capacity, avail + (now - last) * self.refill)
self.state[session_id] = [avail, now]
return avail
def admit(self, session_id, est_tokens) -> bool:
if self._tokens(session_id) < est_tokens:
return False
self.state[session_id][0] -= est_tokens
return True
bucket = SessionTokenBucket()
if not bucket.admit(session_id="s-7", est_tokens=40_000):
raise Reject(429, "session compute budget exhausted")
```
The real answer: move the intelligence upstream
To handle the non-deterministic pattern of agentic traffic, the four layers above help. They’re necessary. But notice what they have in common: they’re all valves at the pipe entrance. Lambda still executes and bills the redundant call. The gateway still has to inspect and reject. The queue still has to hold the flood. Absorbing non-deterministic load at the infrastructure layer is a game you can only lose slowly.
That’s why the intelligence has to move upstream. Admission control and backpressure can’t only live at the gateway — the client has to be smart enough to know when to stop asking. A well-behaved agent client:
- Carries a retry budgetand spends it — no infinite retries
- Honors
429andRetry-Afterinstead of hammering through them - Runs a client-side circuit breakerthat opens on sustained failure
- Treats backpressure signals as cooperative, not adversarial
```
import time, random
class BackpressureAwareClient:
"""A cooperative agent client. The most effective throttle lives
here, at the source — not at the gateway."""
def __init__(self, retry_budget=3, breaker_threshold=5, cooldown_s=30):
self.retry_budget = retry_budget
self.failures = 0
self.breaker_threshold = breaker_threshold
self.cooldown_s = cooldown_s
self.open_until = 0
def call(self, fn):
if time.monotonic() < self.open_until:
raise CircuitOpen("breaker open; not asking")
for attempt in range(self.retry_budget + 1):
resp = fn()
if resp.status == 429:
self.failures += 1
if self.failures >= self.breaker_threshold:
self.open_until = time.monotonic() + self.cooldown_s
raise CircuitOpen("breaker tripped")
delay = resp.headers.get("Retry-After") or (2 ** attempt + random.random())
time.sleep(float(delay)) # cooperate, don't hammer
continue
self.failures = 0
return resp
raise RetryBudgetExhausted("stopped asking") # the client decides to stop
```
Where this leaves us
We spent Generation 1 with the load anticipating. We spent Generation 2 trusting the platform to react. Generation 3 asks something harder: build clients and infrastructure smart enough not to generate the load in the first place. Even with Generation 1 and Generation 2 for the deterministic, human-driven load, I’ve seen misbehaving clients that lead to issues. The client needs to be smart enough to understand the ask in the first place and respect all backpressure signals.
If you’re designing an agent architecture today — orchestrating LLM calls, running retrieval pipelines, letting language models plan and act — build the retry budget, the circuit breaker, and the cooperative backpressure into the client from day one. Don’t leave it as an afterthought that surfaces when it starts costing you money.
Again, the smartest valve isn’t at the pipe entrance. It’s at the source.