Innovation & Tech

Propensity Labs Says Smaller AI Models Match Premium Rivals at 7% of the Cost

Propensity Labs claims prompt-adapted cheaper models matched premium rivals at 7% of cost on Sierra's Tau3 benchmark. CEO Rahul Nambiar explains why.

By Olivia Hart

4 min read

Rahul Nambiar, Co-Founder and CEO of Propensity Labs – Interview Series
Rahul Nambiar, Co-Founder and CEO of Propensity Labs – Interview SeriesAI-generated

What's News

  • Adapting prompts to Luna matched premium model Sol at 7% of the cost on Sierra's Tau3 agentic benchmark.
  • Routing between Luna and Sol matched Astra's performance at 16% of its cost.
  • CEO Rahul Nambiar previously built infrastructure at AWS and led an exabyte-scale data inventory project at Meta.
  • Propensity Labs does not train on or retain customer data; customers bring their own API keys.
  • Nambiar recommends measuring cost per successfully completed task rather than cost per token.

Propensity Labs says it adapted prompts for a cheaper model and matched a premium model's performance at 7% of the cost on a complex agentic benchmark. CEO Rahul Nambiar disclosed the figure in an interview with Unite.AI, and paired it with a second data point: routing between the adapted cheaper model and the premium one matched a third, larger model at 16% of its cost.

The company, co-founded by Nambiar and Mana Azarm, builds systems that select and steer AI models for specific workloads. It combines behavioral evaluations, automated prompt adaptation, and routing between models. Nambiar previously built large-scale infrastructure at AWS and led work at Meta on an exabyte-scale user data inventory using machine learning classification.

The benchmark claims deserve scrutiny, and Nambiar himself draws the boundary between lab and production. The results came on the Tau3 agentic benchmark from Sierra, where, in his words, "the adapted model Luna matched Sol's performance at 7% of the cost." He added: "Routing between Luna with adaptation and Sol enabled us to match Astra's performance at 16% of its cost."

What changes between a benchmark and production?

A great deal, by Nambiar's own account. Benchmarks provide controlled environments and graders; production workflows do not.

"Production workflows don't come so neatly packaged," he said. "You have traces of models completing tasks, but no reliable way to tell whether they actually succeeded and what good looks like."

His team's workaround: take existing usage data — evaluations or production traces — and reconstruct an evaluation setup to test different models and prompt adaptations on the same workload. He also warned that routing and adaptation must be designed together, because a cheaper model that fails with default prompts may succeed once prompts match its behavior. That, in turn, changes which model deserves the traffic in the first place.

Are any instructions truly universal?

Fewer than developers assume, according to Nambiar. He cited hallucination in single-turn responses as an example, studying DeepSeek V4 Pro and GPT-6 Luna. Both models fabricate answers rather than abstain — an artifact of training, he said — but respond to opposite fixes.

DeepSeek V4 Pro can be pushed to abstain at the level of Claude models by layering variations of "It's okay to say I don't know." The same tactic barely moves Luna. Luna responds to penalties: instructing it that "a wrong answer counts as -90 points, while a right answer only gives +10" makes it abstain like its larger sibling Sol.

Same flaw, different levers — and a reminder that per-model prompt engineering cannot be outsourced to generic best practices.

How does Propensity Labs handle data and privacy?

The company positions itself as a layer between users and model providers. Customers pick their providers and bring their own API keys; Propensity Labs selects and adapts the model per workload. Nambiar stated the firm uses only the evaluations or traces a customer chooses to share, and does not train on or retain customer data — a claim, notably, offered without independent audit.

On division of labor, Azarm — Chief Scientist and a professor — focuses on model behavior while Nambiar handles systems engineering. "Reality is messier and we each end up working on whatever is the main blocker at the moment," he conceded.

What should CTOs measure first?

Nambiar's advice for teams spending heavily on premium API calls: run a latest-generation smaller model on the actual workload and see what it handles. He suspects many teams remain on larger models from previous generations without realizing newer, cheaper alternatives can do the job.

His recommended metrics are task success rates, failure rates, latency, and cost per successfully completed task — not cost per token, which masks the price of retries, routing classifiers, and decision calls. He also observed a pattern: teams struggle to migrate agents between models, invest heavily in evaluations for a few months, then find switching far easier. His advice is to build solid evals early and keep them current.

The most common misconception, in his view, is that a failing cheap model simply lacks capability. Some failures are behavioral patterns that prompting can nudge away; others genuinely require a bigger model. Telling them apart is the work — and the sales pitch.

Looking ahead 12 to 18 months, Nambiar sees practitioners gripped by FOMO and exhaustion as LLMs grow harder, not easier, to deploy amid a proliferation of models and settings. His bet: value lies in abstracting that complexity away, so buyers focus on outcomes rather than model selection. Whether Propensity Labs' 7% figure survives outside benchmark conditions is the question its enterprise customers will answer first.

Source: unite.ai

Share this article:

More from Olivia Hart

Olivia Hart

Show full bio

Staff writer covering people, launches and day-to-day practice in entrepreneurship, startups and venture funding at Business Bearings.

2,205 articles

Related articles

« Previous articleNext article »