Tracing Tools Build AI Gateways to Own the Request Path

By Daniel Okafor

In 2026, LangSmith, Langfuse, Opik and Braintrust all added a gateway in front of model providers. The reason is control. A tracing tool can only record a model call after it happens. A gateway can stop the call before it reaches the provider.

What changed in the market?

For the past two years, many teams ran LLM applications with two separate tools. A gateway sat between the application and the model providers. It held the provider API keys, translated between provider APIs and retried failed calls. LiteLLM, an open-source proxy, became a common choice. Beside the application sat a tracer such as LangSmith or Langfuse, which recorded each model call, tool call and retrieval step.

The split made sense. A gateway must be fast and support every provider's API. A tracer needs a good interface, storage for large payloads and an SDK that wraps application code. Langfuse still documents this two-tool setup through a LiteLLM Proxy integration that sends every call into Langfuse. In 2026, the tracing vendors began building the gateway themselves.

Four launches in six months

The timeline is compressed:

• LangChain announced the LangSmith LLM Gateway in private beta in May 2026 and opened it to public beta on 30 July. It is now part of LangSmith's bring-your-own-cloud deployments.
• Braintrust deprecated its earlier AI proxy and pointed users to a new gateway, free in preview until general availability.
• Comet released the Opik LLM Gateway as a beta endpoint that accepts OpenAI-format requests.
• Langfuse opened its AI Gateway in public beta on 4 November, on Langfuse Cloud and for self-hosting. The gateway is MIT-licensed and written in Rust.

MLflow, which began as a machine learning lifecycle tool, has had an AI gateway longer, with budget alerts and limits.

Why does a budget need the request path?

A tracer records a call after the provider has answered and the tokens are paid for. That record draws a spend chart. It cannot stop the spending. Enforcing a budget, blocking a model or stripping a secret from a prompt all require checking the request before it reaches the provider.

LangChain makes this argument in its launch post. It describes teams combining a separate gateway, a guardrails product and an observability tool, and says preventing problems means enforcing policies at the request layer. Its gateway answers a call that would exceed a budget with HTTP status 402 instead of forwarding it.

How does one entry point capture every call?

Coverage is the second reason. An SDK records only instrumented code paths. Calls outside those paths are where agent failures go unnoticed. A script on a laptop or a coding agent using a copied API key leaves no trace. A gateway holding the provider keys sees every call, because there is no other route to the provider.

Langfuse builds its gateway around this idea. Its documentation says each gateway key belongs to a user or service account and can carry metadata such as a team or cost centre. Only the gateway holds the provider credential, and administrators approve which models each credential may call. LangSmith publishes setup pages for routing Claude Code, Codex and Gemini CLI through its gateway.

The reverse argument also holds. Langfuse's launch post says a standalone gateway can only count tokens and cost, while its own gateway makes each call part of a full trace. The gateway accepts a W3C traceparent header, so a call joins the trace the application already started. By default it records usage and cost, not prompt and response content.

What is the commercial logic?

There is a reading the vendors do not state. Whoever runs the gateway holds the provider keys and the spend report, and the spend report is what finance and platform teams read. That makes a gateway harder to replace than a trace viewer. LangSmith sells Gateway Credits, letting customers call hosted models such as Kimi without their own provider key. Langfuse's FAQ says it does not resell tokens.

What can the beta gateways not do yet?

The gaps are documented. Langfuse's gateway supports only OpenAI and Anthropic today, with Amazon Bedrock, Google Vertex AI and Google AI Studio on its roadmap. Fallbacks and hard spend caps per team, user or key are planned but not shipped. It does not convert between API formats, and the self-hosted version calls official provider URLs only, ruling out Azure OpenAI, Bedrock or an internal proxy. Comet describes the Opik gateway as designed for development and testing. Braintrust has not announced gateway pricing.

LangSmith's gateway does more. Its documentation lists fallbacks, spend limits, rate limits and data policies as current features, and it accepts OpenAI Chat Completions, OpenAI Responses and Anthropic Messages formats. Fallbacks and rate limits were future work in May, so they arrived during the beta.

A self-hosted benchmark of the Langfuse gateway, LiteLLM and AcruxCore found the Langfuse gateway added less latency per call than LiteLLM. With default Docker settings, it failed 739 of 1,000 simultaneous streams, because its process kept Docker's default limit of 1,024 open files. One line in Langfuse's Compose file fixed that.

LiteLLM, by comparison, supports more than 100 providers and already includes virtual keys, spend tracking, guardrails, load balancing and an admin dashboard. It is moving its request path from Python to Rust; the Rust AI Gateway is in beta, and the team expects Anthropic-style /v1/messages requests to run end to end in Rust by 1 December, with about 20 times less latency.

Why keep them separate?

The largest cost of merging is coupling. With a separate gateway, a team changes tracing vendors by changing a logging callback. When the tracer is the gateway, every service's base URL, keys and model identifiers change too — Langfuse writes model identifiers as credential:model.

The tracer also becomes a runtime dependency. If a tracing backend goes down, the team loses traces. If a gateway goes down, model calls fail. And security vendors want the same position: Palo Alto Networks completed its acquisition of Portkey in May 2026 and made it the AI gateway of its Prisma AIRS platform. Gateways have also become attack targets, as the LiteLLM supply-chain breach in March showed.

How should teams choose?

Start by listing what must happen before a model call. Visibility alone needs only a tracer beside the request path. Budgets, key policies or model approval need a gateway. Then write down every provider and API format used in production, including Azure OpenAI and Bedrock, and compare that list with each gateway's current support, not its roadmap. Teams starting fresh can keep the gateway as its own layer even when one vendor supplies both.

The open question is ownership. This year LangSmith and Langfuse built gateways and Palo Alto Networks bought one, yet in many stacks the request path still runs through LiteLLM.