AI’s Aggregation Layer Needs Its Own Acceptance Test

By Grace Kim

Two of six equities reversed their projected five-year return direction when two synthesis systems weighed the same 14 reports differently, according to Russlan Ramdowar’s October study, “When Models Judge Models.”

Ramdowar said the experiment did not establish that either system predicted prices better or produced superior investment results. The study targeted the aggregation layer, where the final model decides which arguments deserve influence, which disagreements matter and what reaches the reader.

Ramdowar’s study shows that an AI research workflow can return a different conclusion without changing any source report. His conclusion: the aggregation decision needs its own acceptance test.

What did unchanged reports reveal?

Ramdowar’s archived panel covered six equities and one crypto asset, with 14 saved source reports for each asset. Two deployed synthesis systems received those reports and assigned influence weights. Deterministic code then combined the reports’ quarterly forecasts.

The baseline comparison reversed the projected five-year return direction for two of the six equities. The reports stayed unchanged; the forecast depended on the synthesizer’s weights.

The test design preserved a blinded baseline, an unchanged-presentation repeat, a reversed report order and a condition disclosing source models. Across seven assets, it produced 98 source reports and 56 synthesis outputs.

Public replication materials retain the reports, requests, weights and calculations. An independent offline reproduction passed 392 numerical checks without making new model calls.

“Fourteen reports therefore did not mean fourteen independent sources of intelligence,” Ramdowar wrote. Each pool contained 12 reports from one model family and one each from two others.

Those checks established that the archived calculations could be reproduced, not that the underlying reports were true. The comparison also used deployed systems rather than isolated model architectures. Web access remained available, provider controls differed, and the runs separated execution times.

Fixed reports did not guarantee identical information retrieval during synthesis. Ramdowar said the evidence supports inspecting aggregation, not ranking vendors.

The practical takeaway: reproducibility checks can confirm calculations without validating either the evidence or a model’s investment judgment.

What must APAC teams preserve?

For teams comparing research across APAC markets and languages, Ramdowar says a fluent English summary can hide whether translation, retrieval or synthesis caused a changed view.

His hypothetical case combines a Chinese-language filing, an English broker note and an internal analyst memo about the same issuer. An apparently shared company name does not establish that every document covers the same listing, currency or time horizon. A synthesized narrative can remain coherent while combining incompatible objects.

Ramdowar recommends preserving the original document beside its translation and extracted claims before any comparison. His record includes:

• The issuer identifier, exchange and listing, source currency, reporting period, publication timestamp, retrieval timestamp and source version.
• Separate accounting, security-price and presentation currencies, plus the conversion rate, conversion timestamp and original amounts for any conversion.
• The return measure, anchor date and target horizon, including whether a forecast describes price return or total return.
• The source language, translator or model version, and the exact translated artifact given to the synthesizer.

A document hash can establish whether two runs received the same file; a timestamp alone cannot.

“A twelve-month view cannot silently become a five-year view during summarization,” Ramdowar wrote. An analyst should also be able to trace a disputed claim to the original passage. Testing translation changes separately from synthesis changes makes diagnosis easier.

These are proposed controls, not findings from a multilingual APAC benchmark. Ramdowar said the experiment did not measure translation accuracy or deployments across the region.

The practical takeaway: teams must preserve the research object before they compare fluent summaries.

How should teams replay one stage at a time?

Ramdowar’s method starts with a frozen bundle and saved baseline. Teams record the prompt, report order, model version, settings and tool policy. They can disable retrieval or restrict it to archived evidence. If live retrieval remains necessary, they must save the results and describe the test as one of the deployed workflow, not the model alone.

They should repeat the unchanged setup before replacing the synthesizer. That repeat measures ordinary variation, although a small number of repeats cannot establish a reliable distribution. Each subsequent test changes one intended factor: synthesis model, report order, source labels or translated artifact. Every run preserves its full request so unintended prompt changes remain visible.

The comparison separates three outputs:

• The final numerical output.
• The allocation of influence among sources.
• The explanation.

A small numerical difference can coincide with a large change in which evidence counts. Stronger reasoning language can have little effect on the calculation. Ramdowar recommends checking whether the explanation supports the weights and whether the cited evidence supports the explanation.

For numerical workflows, he recommends a transparent deterministic comparator such as equal weighting, plus a statement of the aggregation operator. Averaging quarterly returns and then compounding them differs from averaging complete forecast paths. Teams need to know which calculation the final narrative represents.

The practical takeaway: replay each variable separately so the source of a changed conclusion remains identifiable.

What should block a release?

Ramdowar says a practical acceptance record should identify:

• The input bundle and intended change.
• Numerical differences and changed assumptions.
• Unresolved disagreements and the responsible reviewer.

Teams must set tolerances before running the comparison. Missing provenance, an unexplained currency conversion or a changed horizon should trigger investigation even when the prose reads well.

The study used a small, deliberately selected panel with limited repeats. An extra checklist confounded the crypto asset’s order and disclosure conditions, so Ramdowar separated those treatment comparisons from the matched equity analysis. No forecast target had matured at the manuscript cutoff. Source disagreement was not a calibrated prediction interval, and the observed changes did not establish causal provider bias.

Ramdowar is the founder of Future Edge Group and iPulse AI, an Open Agentic Investment Research Platform. Based in Abu Dhabi, he works on inspectable AI research workflows, evidence preservation and synthesis evaluation. He conducted the experiment within iPulse AI’s research pipeline and disclosed those affiliations. TNGlobal said it lightly edited the contribution for clarity and house style without changing its substance.

APAC teams can adopt the discipline without assuming they will reproduce the study’s results. Their next replacement model should arrive with a reviewable record of the changed conclusion, how it follows from the evidence and who accepted it.