Agentic AI Faces an Infrastructure Wall Before an Intelligence Wall

A system that serves 1,000 conversational requests per second may handle only a few dozen concurrent agent sessions on the same hardware. That gap, argues a new analysis from Unite.AI, is the real constraint on agentic AI — not model capability.

The argument is simple. Everyone debates whether agents can plan, recover from errors, and take consequential actions. Those questions matter. But a nearer wall is arriving faster: agents are "economically awkward to serve, and the awkwardness grows with exactly the capabilities everyone wants to add."

Why is an agent a different workload than a chatbot?

A chatbot request is one round trip. Prompt in, tokens out, connection closed. It can be batched, cached, and measured with a single throughput number.

An agent request is a session. The model calls a tool, waits, reads the result, decides, calls another tool, and eventually answers. A modest task might involve six to eight model invocations, with waiting in between.

Between steps, the agent holds state — conversation, tool outputs, accumulated reasoning. On the serving side, that state lives in GPU memory as KV cache. It occupies memory for the entire session, including the seconds the agent spends waiting on an external API.

The capacity unit stops being "requests per second" and becomes "concurrent sessions resident in memory."

Why is the waiting the expensive part?

During a tool call, the agent generates no tokens and consumes no compute. The GPU memory stays occupied anyway, because evicting and recomputing the context would cost more than holding it. The result: "Your most expensive hardware is reserved for a request that is, at that moment, waiting on a database query."

Conventional distributed systems solved this long ago — release the resource, resume when the response arrives. Model serving would need to evict KV cache to host memory or disk and restore it. The transfer cost is high enough that most systems don't bother. Research on the problem is active precisely because it is expensive.

How bad is sequential execution?

Nearly every agent framework runs tool calls one after another. A weather lookup, then a calendar check, then a flight search. Those three calls often have no dependency on each other and could run simultaneously.

Executed sequentially, latency is the sum. Executed concurrently, it is the longest single call. Total latency should be the longest path through the dependency graph, not the sum of every stage.

This matters more for agents than for ordinary services because the session sits in GPU memory throughout. "Halve session duration and you roughly double concurrent capacity, on the same GPUs, with no model change."

What do retries do to costs?

Agent frameworks inherit retry defaults from underlying HTTP libraries — usually three attempts. Reasoning models also retry in their own way: when a tool returns something unhelpful, the model tries a different approach. Stack the two, and a three-attempt policy wrapped around a model that independently reasons into three more approaches becomes a nine-attempt policy nobody wrote down. Each attempt is a full model invocation with its own tokens and GPU seconds. When a dependency slows down, this amplification turns degradation into outage at a cost per attempt orders of magnitude above a retried REST call.

What does the wall actually look like?

Not a capability ceiling — a unit-economics problem that stops projects at the pilot-to-production boundary. The pilot works: ten users, acceptable latency, a rounding-error bill. Modeled at ten thousand users, the GPU requirement becomes implausible, because each concurrent session holds memory for its full lifetime, waiting included. This is where promising internal demos quietly stop.

What moves the wall?

The fixes are infrastructure, not modeling:

• Run independent tool calls concurrently. The largest single win, available today, with no new hardware.
• Treat session duration as the capacity metric. Not tokens per second, not GPU utilization — how long state stays resident and how many sessions fit.
• Budget retries globally, not per call. A cap on retries as a small percentage of total traffic prevents compounding. Google's SRE practice on overload and Envoy's retry budgets already implement this idea outside the AI stack.
• Separate reasoning from execution. A large model plans; a much smaller one handles mechanical steps. Most of what an agent does doesn't need a frontier model.
• Make context eviction cheap. Quantized KV cache, paged attention, and offload paths for idle sessions — where the most interesting serving work is happening now.

The models will keep improving, the author writes. Whether agents become infrastructure or stay demos depends on something less glamorous: whether the industry can serve a workload that holds expensive memory while it waits, and whether it stops paying the sum of a sequence when it could pay only its longest path. The concurrent execution model behind that point was published in "Asynchronous Inference Graph Execution for Model Routing in Machine Learning Systems," International Journal of Computer Trends and Technology 72(10), October 2024.