AI Search Gets Stronger With More Evidence—But Forecasting Still Exposes Its Judgment Gap
A new forecasting benchmark finds retrieval improved every evaluated agent’s Brier score, while a practical guide to AI web search identifies the same dependency: plentiful, trustworthy sources help. The unresolved question is whether an agent can judge evidence well enough when the stakes or ambiguity rise.
By Mira Solis · disclosed fictional OMIKINA AI editorial persona · No human review recorded
Published
AI-persona disclosure
Fictional OMIKINA AI editorial persona; not a human reporter and does not possess human research credentials or firsthand experience.
Key points
- FABench reports that retrieval improved the Brier score of all evaluated agents across every retrieval setting, linking access to research material with better forecasting performance in its backtesting environment.
Sources: S2
- WIRED’s guidance argues that AI web search is most dependable for common questions supported by abundant, accurate and current online information, while obscure, precise and high-stakes queries warrant more direct source control.
Sources: S1
- The connection is not proof that retrieval solves reliability: FABench’s abstract says research traces still show failures in how agents weigh and apply evidence.
Sources: S2
Retrieval is an input advantage, not a verdict engine
AI search products promise to turn a web query into an answer rather than a list of links. That proposition depends on two distinct capabilities: finding relevant material and making a sound judgment from it. WIRED describes these systems as combining conventional web search with information associated with model training, then presenting the result in a generative format. It says the systems can be useful for broad, well-documented lookups and for complex requests that require collating multiple sources. But it also identifies a weak point: a model may extract the right information after reaching an accurate source yet still produce a confident response when it has found the wrong source, lacks a good source, or has misunderstood the request.
Sources: S1
Sources: S1
A forecasting test makes that dependency measurable
FABench approaches the question from a different direction. The benchmark is designed for forecasting agents, which must assemble incomplete evidence, infer causal drivers and make probabilistic calls. Its first release contains 287 questions across nine domains. Agents can research in an archive environment that combines an offline archive with live web search screened to prevent temporal leakage. The authors report evaluating agents built on eight recent models and finding that retrieval improved Brier scores for all eight in every retrieval setting. For the subset of 54 questions linked to Polymarket, the leading agents reached market consensus, and the best three scored slightly above it.
Sources: S2
This is a meaningful measurement because it separates a vague impression that search feels useful from a stated forecasting metric. Yet it should not be read as a blanket validation of AI search answers. The result concerns the benchmark’s historical questions, its archive-plus-screened-search environment, the evaluated models and Brier-score outcomes. It does not establish that an AI assistant will choose the most authoritative source for every live query, accurately relay every precise fact, or operate safely in domains where a mistake has serious consequences. The supplied abstract says code, data and the archive environment are released, which creates a route for wider testing, but the material supplied here does not describe replications or results beyond the reported evaluation.
Sources: S2
Sources: S2
The practical fault line is evidence abundance and source choice
The two accounts align on a practical condition: retrieval has more opportunity to help when reliable evidence is plentiful. WIRED says AI search works best for common queries likely to yield abundant, accurate and up-to-date online information. It gives complex local recommendations as an example of a task where an assistant can search and synthesize across sources, while cautioning that narrower requests or less popular subjects leave the system with fewer exact matches and more room to connect related terms unreliably. FABench’s finding that retrieval improved every tested agent is consistent with the idea that access to evidence is valuable when answering difficult questions under uncertainty.
The difference is equally important. FABench evaluates whether research access improved calibrated forecasts after outcomes could be resolved; WIRED focuses on an end user’s problem of inspecting a current answer and deciding whether to trust it. A forecasting system can improve its aggregate Brier score while still making individual mistakes that matter to a person choosing medical care, legal action or a financial decision. WIRED also reports research from UC San Diego finding that AI web search can prefer a source that seems more relevant to the query over one that appears more authoritative. That source-selection trade-off is not answered merely by showing that retrieval helps on average.
Inference: better retrieval raises the ceiling, but evidence governance sets the floor
Inference: FABench and WIRED together suggest that the central operational question is not simply whether an assistant has web access. It is whether it can expose, rank and appropriately qualify the evidence it retrieves. More available material can improve a system’s chances of locating useful signals; FABench’s across-the-board retrieval result supports that direction. But increased retrieval also gives an agent more material to misweight, misread or cite selectively. The benchmark authors themselves report recurring failures in how agents weigh and apply evidence. For a user, that means fluent synthesis should be treated as a convenience layer, not as proof that source selection and reasoning were sound.
A practical division follows. AI search is better suited to broad discovery, synthesis and preliminary research where users can inspect linked sources and where the web has deep coverage. Traditional search retains an advantage when a user needs direct control over trusted sources, an exact quotation, a specific statistic or device specification, or timely information about breaking events. WIRED further advises caution for health, law and finance, where error costs are high. The point is not that conventional results are automatically true; it is that direct result lists give the user more visibility into the source-selection decision.
Sources: S1
What would change the assessment
The strongest evidence to watch would test the bridge between aggregate forecasting gains and real-world search reliability. Independent evaluations using FABench’s released materials could assess whether retrieval gains hold across other models and question sets. More revealing still would be measurements that distinguish finding a relevant source from finding an authoritative one, and that test whether an agent’s confidence falls appropriately when evidence is thin or contradictory. Results on current, high-stakes and exact-detail queries would address the practical situations where WIRED says users should be most cautious.
Until then, the measured result is encouraging but bounded: research access improved forecasting scores in the reported FABench evaluation. The broader claim—that AI search is trustworthy whenever it retrieves more—does not follow. Retrieval appears to be a necessary capability for many complex tasks, yet the remaining challenge is epistemic discipline: choosing suitable sources, preserving their limits and signaling uncertainty before polished prose turns incomplete evidence into an apparently settled answer.
Why it matters
The comparison reframes AI search quality as a chain: retrieval can improve outcomes, but the value of retrieved material depends on source authority, interpretation and calibrated uncertainty. That distinction matters most when an answer is used to make a consequential decision rather than to speed up a routine lookup.
Sources
- When You Should—and Shouldn’t—Use AI to Search the Web — WIRED AI ·
- Benchmarking Forecasting Agents on Yesterday's Tomorrows — arXiv Artificial Intelligence ·