Prefill/decode disaggregation has the elegance of a clean architecture diagram.

Prompt processing is compute-heavy. Token generation is memory-bandwidth-heavy. Put them on separate GPU pools, scale each phase independently, and stop them from interfering with each other.

That logic is sound.

Then a burst arrives. Prefill queues grow. Decode GPUs have spare compute but cannot begin because KV cache has not reached them. The network carries large long-context state. Time to first token rises while each pool looks locally “correct.”

Disaggregation did not remove the bottleneck. It distributed it across queues and a data transfer.

Decompose TTFT After the Split

Before disaggregation:

TTFT = scheduler queue + prefill + first decode step

After disaggregation:

TTFT = routing + prefill queue + prefill execution + KV transfer queue + KV serialization/transfer + decode admission queue + first decode step

Phase isolation can reduce contention while increasing path length.

Instrument every component with the same request ID. A dashboard that shows only prefill execution can report success while users wait in front of it.

Heavy-Tailed Workloads Break Static Ratios

Production prompts are not uniform.

A few long contexts can occupy prefill capacity while many short requests wait. Output lengths also vary, changing decode occupancy. Traffic bursts by tenant, event, and time.

A July 2026 paper on load-aware prefill deflection studied this asymmetry. In its production-style 2-prefill/2-decode A100 setup, prefill execution itself accounted for only 2–23% of P95 TTFT across evaluated workloads; queueing and inter-node KV transfer accounted for the rest. Decode nodes could have unused compute while prefill queues grew.

Those numbers are experimental, not universal. The structural failure is widely relevant:

static phase ownership + dynamic asymmetric load = stranded capacity

Capacity ratios should change with input/output distributions and queue pressure.

KV Transfer Is on the Critical Path

Decode cannot use state it does not have.

Transfer cost depends on:

  • context length;
  • model architecture and KV heads;
  • precision/compression;
  • tensor/sequence parallel layout;
  • network topology and contention;
  • chunking;
  • destination placement;
  • concurrent transfers.

Measure bytes and time per request, not only aggregate bandwidth. Tail latency comes from queueing and contention even when average link utilization seems acceptable.

Prioritize by deadline and expected value. Cancel transfers when the request is canceled. Avoid moving state to a decode node that will not admit the sequence soon.

The router must understand both compute and data locality.

Deflect Prefill When the Math Supports It

The load-aware deflection paper lets decode nodes execute some prefill work in chunks interleaved with active decoding. For each queued request, the scheduler estimates TTFT through the prefill pool and alternative decode nodes, then deflects only when it improves the request without violating time-between-token SLOs for current decodes.

The useful pattern is conditional flexibility:

  • preserve dedicated phase pools under balanced load;
  • borrow decode compute during prefill saturation;
  • eliminate transfer for the deflected request because prefill happens at the decode node;
  • protect existing decode cadence with an explicit constraint.

Do not turn every decode node into a prefill node. That recreates interference. Deflection is a controlled overload valve.

Progressive Transfer Changes “Ready”

Traditional transfer treats KV as one indivisible object: decode waits for all bytes.

Lynx explores progressive transfer. It sends high-priority information first, allows speculative decode to begin, then transfers residual precision and verifies. The paper reports TTFT improvements over standard quantized transfer in its workloads while matching higher-precision output through verification.

This reframes readiness:

not: full KV received but: enough state received to begin a verifiable draft

The technique adds complexity: stream prioritization, speculative state, verification, rollback, and quality guarantees. It is attractive when long-context network-exposed latency dominates.

Measure the whole pipeline, including rejected draft work.

Compression Is Model-Dependent

Reducing KV precision lowers transfer bytes. Aggressive compression is not universally safe.

SpectrumKV assigns precision by token importance and uses a deployment-time probe because INT4 tolerance differs across evaluated models. The paper reports that one tested Qwen model failed under aggressive INT4 while other tested models tolerated it, motivating a fallback policy.

This is the correct production shape:

probe → qualify model/workload → select policy → monitor → fall back

Do not deploy one compression level across all models and tool-calling workloads based on average perplexity.

Route on Predicted Completion, Not Shortest Queue

The shortest prefill queue may contain huge prompts. The nearest decode node may have no KV locality. A low-utilization node may have a congested path.

Estimate:

predicted_TTFT(node, path) = queue_work_ahead + prefill_compute + transfer + decode_admission

Include:

  • token lengths;
  • current batches;
  • link queue;
  • KV cache availability;
  • memory headroom;
  • tenant/SLO class;
  • cancellation probability.

Predictions will be wrong. Track error by workload bucket and fall back to robust policies under uncertainty.

Protect TPOT While Optimizing TTFT

Borrowing decode resources for prefill can delay tokens for active users.

Use two SLOs:

  • TTFT for new requests;
  • time between tokens or TPOT for active decodes.

The scheduler can improve one only within a bound on the other. Report p95 and p99 for both. Average throughput can hide visible stalls.

Use priority carefully. Interactive short requests may need fast first token; batch workloads can wait. Prevent starvation with aging or reserved capacity.

Know When Not to Disaggregate

Disaggregation may not pay when:

  • models or contexts are small;
  • traffic is low or balanced;
  • network is slow relative to recompute;
  • hardware pools cannot scale independently;
  • operational complexity exceeds benefit;
  • colocated batching already meets SLO;
  • KV transfer dominates the potential interference savings.

Run a colocated baseline. A sophisticated architecture must beat it on cost per SLO-compliant token, not merely phase utilization.

Benchmark Bursts and Imbalance

Test:

  • short/long prompt mixtures;
  • short/long output mixtures;
  • bursty arrivals;
  • changing prefill/decode ratios;
  • network contention;
  • transfer compression modes;
  • cancellation mid-transfer;
  • cold/warm prefix cache;
  • one pool degraded;
  • routing prediction error.

Report:

  • TTFT components at p50/p95/p99;
  • TPOT/TBT;
  • queue age by phase;
  • stranded capacity;
  • transfer bytes and queue time;
  • SLO attainment;
  • wasted work;
  • cost per completed token.

The Architecture Is a Control Loop

Prefill/decode disaggregation is not a one-time topology decision. It is a control loop:

observe queues, lengths, links, and SLOs → predict path latency → place/deflect/transfer → verify impact → adapt ratios and policy

The split is valuable when independent scaling and reduced interference outweigh the new queues and data movement.

If your measurement stops at the edge of each GPU pool, you will optimize the architecture and lose the user.