DeepResearch Bench Explained

What is in the benchmark

DeepResearch Bench, arXiv 2506.11763, contains 100 PhD-level research tasks. Domain experts wrote them across 22 distinct fields. The paper is also listed in the ICLR 2026 proceedings.

Some secondary summaries describe the set as 50 English tasks and 50 Chinese tasks. We did not confirm that split in the paper's abstract or project page, so treat it as unverified.

The project describes the benchmark as a way to evaluate deep research agents. The tasks ask for long, multi-step reports rather than one-line answers, so the scoring looks at the whole report and not a single fact.

A BibTeX entry on the project page lists the work as a 2025 arXiv preprint. The project page does not link to the paper directly, so cite the arXiv number.

How RACE and FACT score an agent

RACE stands for Reference-based Adaptive Criteria-driven Evaluation with Dynamic Weighting. It scores the quality of the final report against criteria, with weights that adjust to each task.

The project page expands FACT as Framework for Factual Abundance and Citation Trustworthiness. The arXiv abstract describes it as a retrieval measure that counts effective citations and checks how accurate those citations are.

Critiques of report-only scoring

A follow-up paper, arXiv 2512.01948, argues that scoring only the final report misses the search and reasoning behind it. Its authors propose a finer benchmark called Finder, which uses structured checklists, and a failure taxonomy called DEFT that judges the process as well as the output.

The critique matters when you read a RACE score. A high score shows that a report reads well and meets its criteria. It does not prove that the agent searched well or that every claim is sourced. Pairing RACE with a citation check gives a fuller picture than either one alone. The benchmark's tasks cover many fields, so check whether the field closest to your work is among them.

Where it fits in a comparison

Open-source agents often publish DeepResearch Bench results. The LangChain open_deep_research README, for example, reports RACE scores around 0.43 to 0.49, depending on the configuration. Those are the project's own numbers, from its own setup, so they do not compare directly with DRACO scores.

A useful check is whether the project's setup matches yours. The model, the search tool and the judge all change the score, so a number describes one configuration, not a whole category of tools.

Checklist

• Check whether a score is a RACE score, a FACT score or another metric.
• Confirm the paper version and date before you quote a result.
• Pair any report-quality score with a check of the citations.
• Compare only runs that used the same judge and the same task set.
• Read the Finder critique before you treat RACE as a full test of research skill.

Questions & answers

Sources for this entry

• DeepResearch Bench paper, arXiv 2506.11763
• DeepResearch Bench project page on GitHub
• Finder critique paper, arXiv 2512.01948
• DeepResearch Bench, ICLR 2026 proceedings

Vendor descriptions are attributed to the vendor. Prices change; confirm with each vendor.

Continue this reading path

• Benchmarks
• Best Deep Research AI
• Resource directory

Cite this entry