Valory's Open-Weight Olas Predict Model Nears GPT-5.6 in Forecasting

Valory's new open-weight forecasting model, Olas-Predict-Qwen3.8-27B, scored 67.3% balanced accuracy on real-world prediction markets — just 1.2 percentage points behind its GPT-5.6 comparator, which reached 68.5% when both systems worked from the same market evidence.

The announcement covers 6,824 researchable prediction markets. Both models ran with common prompts and evidence at temperature zero, and Valory applied a date filter tied to GPT-5.6's stated February 16, 2026 knowledge cutoff. The test measures how models reason over supplied evidence, not how well they gather information on their own.

That scope matters. These are company-reported results on selected markets, not an independent demonstration of general equivalence between the models. The final comparison is also separate from an earlier 10,916-sample experiment and from Valory's previously published 14-billion-parameter study.

What do the numbers show?

Valory reports two metrics. Balanced accuracy weights both outcome classes equally, so a common outcome can't dominate the score. Brier score evaluates probability forecasts and penalizes confident errors — lower is better, and it rewards well-calibrated uncertainty over simply picking the winner.

Here is Valory's reported final comparison subset:

• Historical GPT-4.1 baseline: Brier 0.3110, balanced accuracy 58.5%
• Olas-Predict-R1-14B: Brier 0.2722, balanced accuracy 58.9%
• Qwen3.8-27B before domain training: Brier 0.2235, balanced accuracy 66.4%
• Olas-Predict-Qwen3.8-27B: Brier 0.2124, balanced accuracy 67.3%
• GPT-5.6 comparator: Brier 0.2029, balanced accuracy 68.5%

One caveat: the GPT-4.1 baseline should be read as that specific baseline, not a new universal score for GPT-4.1.

The takeaway in plain words: the gap between an open 27B model and GPT-5.6 on this task is now about one percentage point.

Where does the gain actually come from?

The new model beats Olas-Predict-R1-14B by roughly 22% on Brier error and 8.4 percentage points on balanced accuracy. But the untrained Qwen3.8-27B starting point already hits 66.4% balanced accuracy and a 0.2235 Brier score — so the stronger base model accounts for most of the generational step.

Domain training adds a measurable but smaller lift. Brier error falls by roughly 5%, balanced accuracy rises 0.9 percentage points, and expected calibration error — how well stated confidence matches observed outcomes — drops from 0.1630 to 0.1429, a 12.3% improvement.

That's the more useful reading: a stronger base and specialized training contribute separately. The fine-tuning mainly sharpens how the model expresses probabilities, which matters when a downstream system must judge whether evidence is strong enough to act.

How does the system work?

Valory describes a training process that uses reviewed forecasting examples and reinforcement learning to improve probability estimates. A separate tool gathers market intelligence and evidence, then hands that material to the model.

The separation is the architecture. Evidence retrieval determines what information is available; the model interprets it and produces a probability. Strong reasoning over weak or outdated evidence still yields a poor forecast. It also means developers can refine retrieval, change evidence presentation, or swap the forecaster without rebuilding the whole agent — though the reported scores apply to one specific combination, and shouldn't be transferred automatically to a different retrieval pipeline.

What does Olas bring to the table?

Valory is a core contributor to Olas, whose Predict ecosystem connects agents that create markets, trade, provide forecasts, and finalize outcomes. Traders buy forecasting services from specialist agents through the Mech Marketplace. Once events resolve, forecasts can be checked against outcomes, creating records for evaluation and future training.

That feedback loop supplies task-specific evidence: the system can eventually compare a probability estimate with a resolved event rather than judging whether an explanation sounds convincing. Market resolution still defines the target, so question wording and resolution quality remain consequential.

Valory's earlier published work used more than 200,000 agent prediction records to specialize Olas-Predict-R1-14B. A separate later-market evaluation reported 75.8% accuracy across 2,628 markets — but that's ordinary accuracy from a different evaluation, not directly comparable with the new 67.3% balanced-accuracy figure.

What do open weights change for buyers?

Open weights let developers host and adapt the model instead of depending entirely on a remote proprietary service — more control over updates, serving infrastructure, and evidence handling.

A 27-billion-parameter model still needs suitable hardware, with memory, latency, and throughput depending on precision, context length, and concurrency. Valory provides no measured deployment costs or serving benchmarks, so the parameter count alone doesn't establish an operating-cost advantage. And forecasting quality isn't trading performance: returns also depend on market prices, transaction costs, execution, and strategy — a distinction reflected in the limitations of Valory's earlier model card.

The significance is focused but real: Valory reports a specialist model approaching its frontier comparator on supplied-evidence forecasting, with measurable gains from domain training. The next test is turning the stream of resolved predictions into improvements that hold up across later events and live deployments.