Innovation & Tech

Meta's AI Agents Ran 1,097 Experiments for a 0.23% Search Lift

Meta's MLE agents ran 1,097 experiments in 31 days on about 360,000 GPU-hours, lifting search sessions 0.23% while slowing time-to-render in a live test.

By Daniel Okafor

3 min read

Meta's AI Agents Ran 1,097 Experiments in a Month for a 0.23% Search Gain
Meta's AI Agents Ran 1,097 Experiments in a Month for a 0.23% Search GainPublicDomainPictures / pixabay

What's News

  • Meta's MLE agents completed 1,097 experiments in 31 days, using roughly 360,000 GPU-hours
  • The consolidated model raised search-session volume 0.23% in a one-week live A/B test, with a 95% confidence interval of ±0.15 percentage points
  • 325 of 1,097 experiments were accepted, a 29.6% acceptance rate versus 15.6% in the earlier sequential version
  • The target was Nullstate, Meta's search-suggestion ranker, which handles more than a billion ranking requests per day
  • Agents used Anthropic's Claude Opus 4.8, ran seven parallel candidates per iteration across 131 iterations, with a median 3.54 hours per run

By Daniel Okafor

Meta's autonomous AI agents ran 1,097 machine-learning experiments in 31 days, consumed roughly 360,000 GPU-hours, and lifted search sessions 0.23% in a live A/B test. The campaign, detailed in a November 18 preprint, targeted Nullstate, Meta's search-suggestion ranker, which decides what appears when a user opens the search bar before typing. Nullstate handles more than a billion ranking requests per day.

What did Meta's agents actually build?

Improving such a model normally means engineers hypothesize a change, implement it, train a model and evaluate the result — a slow, serial loop. Meta's researchers call their version "MLE agents," and the system automated much of that work. Using Anthropic's Claude Opus 4.8, the agents proposed architectural changes, wrote code, launched distributed training jobs on 64-GPU trainers and scored offline metrics. Engineers kept control of the objective, strategic direction and final model review.

The operating numbers:

  • Seven candidates ran in parallel per iteration, plus one variance probe, across 131 iterations
  • Each training-and-evaluation run took a median of 3.54 hours
  • 325 of 1,097 completed experiments were accepted, a 29.6% acceptance rate against 15.6% in an earlier sequential version of the system

The accepted changes compounded into a -0.0224 shift in a weighted normalized-entropy objective, where lower is better, driven mainly by better prediction of downstream engagement. Several accepted changes were architectural ideas the human team had never tried: Random Fourier Feature gating, learned-sigmoid attention gating and a zero-init gated distillation bypass.

Meta then placed the consolidated model into a live A/B test lasting one week. Search-session volume rose 0.23% against the control, with a 95% confidence interval of ±0.15 percentage points. Downstream engagement metrics moved in the same direction.

What did the agents break?

The campaign surfaced a trade-off Meta disclosed but did not quantify. The new model slowed the search interface's time-to-render. The agents optimized a calibrated offline objective tied to engagement; rendering speed was monitored separately and never entered the reward function. The authors acknowledged that an agent optimizing a pure quality objective does not surface such trade-offs on its own.

The compute bill was equally lopsided:

  • About 360,000 GPU-hours in total
  • 155 variance probes included in that total
  • 2,798 failed or retried attempts

What did Meta leave unanswered?

Meta did not disclose the financial cost of the campaign. It also did not run a matched comparison against a conventional engineering team pursuing the same objective. The preprint leaves permanent rollout and a human-only benchmark unresolved.

The agent work also has visible limits. The best model surfaced well before the campaign ended, and later iterations added nothing. The agents hit exploration collapse, an 0-for-8 streak on optimizer tuning before human redirection, and their self-correction proved unreliable. The offline-online gap persists: not every offline winner transfers, and the agent has no mechanism beyond the calibrated objective to predict which will.

What mattered more than the choice of model was the setup around it — the calibrated objective, the structured context, including a "what didn't work" blocklist, and the parallel evaluation loop. The underlying LLM stayed fixed throughout the campaign.

For a platform serving more than a billion ranking requests a day, a 0.23% lift in search sessions is statistically significant and operationally meaningful. Whether it repays the compute, the latency cost and the engineering overhead of the agent harness is a question the paper leaves open.

Source: epyhia.com

Share this article:

More from Daniel Okafor

Daniel Okafor

Show full bio

Correspondent covering money, funding and market shifts around entrepreneurship, startups and venture funding at Business Bearings.

2,538 articles

Related articles

« Previous article