GPU Utilization Is the Wrong Metric for AI, New Scorecard Argues
NAND Research and WEKA's Inference Economics Scorecard says GPU utilization hides costly context rebuilding; an eight-hour benchmark shows 3.7x more tokens per GPU-hour with shared NVMe cache.
By Amara Osei
3 min read
What's News
- WEKA and NAND Research propose The Inference Economics Scorecard, replacing GPU utilization with useful output per watt, rack and dollar
- Eight-hour benchmark on NVIDIA HGX B300 showed 3.7x more output tokens per GPU-hour with shared NVMe cache
- Context recomputation fell from 60% of prefill cycles to 6.8% with shared cache
- 97% of successful requests hit first token within five seconds versus 44% for HBM-only baseline
- Benchmark used over 5,000 real agent traces with 512 active sessions and 32 concurrent requests
A busy GPU is not necessarily a productive GPU. That's the core claim behind The Inference Economics Scorecard, new research from NAND Research and WEKA that proposes measuring useful output against power, rack space, and cost instead of raw chip activity.
The timing matters. AI agents now hold long conversations, call tools, and return repeatedly to earlier context. An inference server can burn substantial compute rebuilding that context while its utilization dashboard still looks healthy. WEKA sponsored the research, and its storage technology benefits from this framing — the scorecard is a proposed framework, not an industry standard.
Why GPU Utilization Can Hide Expensive Repeated Work
Inference — running a trained model to produce an answer — starts with a phase called prefill, where the model processes input and conversation history. During prefill it builds a key-value (KV) cache: intermediate attention data it can reuse as the conversation continues.
The fastest home for that cache is the GPU's high-bandwidth memory, HBM. But HBM must also hold model weights and other working data. Long sessions and competing users force evictions. When the user returns, the server rebuilds the missing cache by reprocessing old tokens.
That creates a measurement trap, the researchers argue. Reprocessing yesterday's context keeps the GPU just as active as generating a useful answer. Utilization remains a valuable diagnostic signal, but it cannot tell an operator how much productive work customers actually received.
What Does the Scorecard Measure?
The framework demands continuous tests of at least eight hours — long enough to expose behavior short demos miss. Comparisons must hold the model, precision, workload, concurrency, cache conditions, quality threshold, and service-level objectives constant. Only successful output meeting quality and service targets counts as useful throughput; failed requests, retries, timeouts, and discarded output stay visible.
The scorecard tracks nine metrics:
- Tokens per watt and tokens per rack
- Cost per inference
- GPU goodput — productive output within service targets, not raw chip activity
- Time to first token and end-to-end latency
- Memory efficiency
- Operational density
- Agent concurrency
- Context retention
The throughput-versus-goodput distinction is central. A server producing more tokens but missing a customer's response deadline can be less useful than a slower system that consistently meets it.
What Did the Eight-Hour Benchmark Find?
WEKA's NeuralMesh with Augmented Memory Grid keeps reusable KV data on shared NVMe storage beyond GPU memory, retrieved over RDMA and NVIDIA GPUDirect Storage along a direct storage-to-GPU path.
The joint Nebius-WEKA test ran one eight-GPU NVIDIA HGX B300 node serving DeepSeek-V4-Pro. The replay workload drew on more than 5,000 real agent traces, maintained 512 active sessions, and allowed 32 concurrent requests.
Across eight hours, WEKA reports:
- 3.7 times more output tokens per GPU-hour with shared cache versus HBM-only serving
- 97% of successful requests hit a first token within five seconds, against 44% for the baseline
- Context recomputation consumed 6.8% of prefill cycles with shared cache, versus 60% for the baseline
The caveats are real. The cache used NVMe drives across an eight-node cluster while the serving result came from one GPU node, so the GPU-normalized gain doesn't prove an equivalent cut in total cost or energy. And a suitable DRAM-offload implementation for this model wasn't available at test time — the comparison is against HBM-only serving, not every alternative cache architecture.
Can Others Reproduce It?
WEKA's open-source kvlab project captures and replays the structure of agent sessions — timing and cache-reuse patterns — without distributing original conversations or source code. Its loader is documented in NVIDIA's AIPerf tooling, and the announcement identifies integration with SemiAnalysis's AgentX benchmark. Replays measure serving behavior; they don't prove an agent completed a business task correctly.
For enterprises, the scorecard's contribution is a sharper definition of success: useful responses, at required quality and speed, within available resources. As agents consume more context and run longer, keeping a GPU busy becomes a starting observation. What matters commercially is the work that reaches the customer.
Source: unite.ai
More from Amara Osei
Show full bio
Senior reporter covering companies, deals and strategy in entrepreneurship, startups and venture funding at Business Bearings.
2,136 articles