Innovation & Tech

OpenAI agents excel at routine research tasks, fail core scaffolding

OpenAI says coding agents can replace months of work, yet succeed on only 25% of inference and serving tasks, exposing a widening gap between speed and reliability.

By Amara Osei

4 min read

OpenAI Researchers Rate AI Tools Highly for Routine Tasks But Flag Code Scaffolding Gaps
OpenAI Researchers Rate AI Tools Highly for Routine Tasks But Flag Code Scaffolding GapsPexels / pixabay

What's News

  • OpenAI researchers said they would rather give up 40% of team headcount or 30% of compute than lose coding agents for three months.
  • Median researcher token use reached $2,400 per day in October, up 62% from September.
  • Autonomous completion rose from 23% to 46% for two-to-four-hour tasks between March and October.
  • Agents wrote inference and serving code successfully 25% of the time and built evaluation and reinforcement-learning environments 29% of the time.
  • In an OpenAI poll of 28 employees, 27 preferred the latest internal model over a human engineer assisted by a 2024-era model.

By Amara Osei

OpenAI's own researchers said they would rather give up 40% of their team's headcount or 30% of their compute than lose access to frontier coding agents for three months. The researchers also reported that agents write inference and serving code successfully only 25% of the time and build evaluation environments 29% of the time.

OpenAI published its second research acceleration report, Research acceleration: The jagged frontier, on November 18. The report tracks how deeply AI agents have embedded themselves in developing new models. OpenAI frames the exercise as tracking indicators of recursive self-improvement, or RSI, in which AI systems speed up the creation of their successors.

Do researchers trust coding agents?

An OpenAI survey of 101 research staff was non-representative. Its median respondent said their team would need three months to produce the same output without coding agents. In a separate quick poll of 28 employees, 27 preferred OpenAI's latest internal model over a mid-career human engineer assisted by a 2024-era model.

OpenAI's qualitative interviews with 10 researchers exposed persistent quality problems. Interviewees said models lack any “taste” for good research ideas, code quality or data quality; agents get lost in unnecessary steps, make wrong assumptions in analyses, and submit subpar “AI slop” code changes alongside unusable data artifacts. OpenAI called close human oversight essential to avoiding wasted compute cycles.

One post-training researcher told OpenAI:

"I've tried saying, 'Just make this task like this other task,' and been burned too many times when it's not quite right and I need to make edits. Now I say, 'Go make one. I'll review it and make edits. Then go do the rest.'"

How much are agents consuming?

OpenAI said internal agent usage has grown exponentially. At public API list prices, the median researcher consumed $2,400 worth of tokens per day in October, up 62% from September. The top 10% of users consumed $23,000 per day, a 105% increase.

Converted into eight-hour workdays, agents now run the equivalent of 5.7 workdays per researcher per calendar day. That figure reaches roughly eight workdays for pre- and post-training teams, while more than two-thirds of tracked researchers ran at least four agents in parallel at peak usage.

OpenAI's report put theoretical API-equivalent spending at twice researchers' non-equity compensation in September. Equity made up 70% to 80% of total compensation at OpenAI, while internal agent use still accounted for only 2% of the company's total research compute.

How reliable are the agents?

OpenAI said success rates on autonomous completion rose sharply between March and October. They increased from 23% to 46% for two-to-four-hour tasks and from 13% to 35% for four-to-eight-hour tasks.

OpenAI said the aggregate improvement masked wide variation. Its task-level results showed:

  • Monitoring training runs: 79%
  • Answering technical questions: 72%
  • Writing inference and serving stacks: 25%
  • Building evaluation and reinforcement-learning environments: 29%
  • Launching training runs: 35%

OpenAI called the uneven pattern a “jagged frontier.” The company said capability gains remain uneven and humans remain central to the research loop.

Who originates GPT-6 Astra ideas?

Asked about the most important conceptual ideas behind GPT-6 Astra and GPT-6.2 Astra, post-training and safety researchers said all named ideas originated with humans, according to OpenAI's report. Models assisted only with implementation.

Pretraining researchers reported the same human origin for GPT-6 Astra. Only for GPT-6.2 Astra pretraining did individual researchers say they gained important insights through dialogue with Codex, OpenAI said.

What could change as agents improve?

OpenAI cautioned that historical trends from human-AI collaboration may not extrapolate linearly. It cited recent abrupt breakthroughs in mathematics and the possibility of similar jumps in deep-learning research as models gain autonomy.

OpenAI reiterated its call for research labs to be required to track and publicly report RSI metrics. The report does not signal an intention to pursue full RSI, the company said.

OpenAI's data points to near-term productivity gains from routine agent work. It also frames inference, evaluation and training infrastructure as the current constraint on wider deployment as internal usage expands.

Source: epyhia.com

Share this article:

More from Amara Osei

Amara Osei

Show full bio

Senior reporter covering companies, deals and strategy in entrepreneurship, startups and venture funding at Business Bearings.

2,422 articles

Related articles

« Previous articleNext article »