Research Snapshot

I. Problem and Solutions

Problem definition

• Input: a research area or topic $A$ plus 3-5 papers $P_1$ - $P_5$.
• Output: a research survey paper $S$ for area $A$.
• Criteria:
  1. Conform to the structure, style, and formatting of published surveys.
  2. Content can help readers understand the field, decide appropriate methods, identify open problems, and avoid duplicating existing work.

Motivations

Why the task is interesting

• Literature review and synthesis consume substantial researcher time. With the increasing rate of published literature, it’s impractical for one or a small group of researchers to keep track with prominent and fast-moving fields. Automating this work can accelerate scientific development by substantially reducing the need for humans to perform the task themselves, and let them spend more time on other elements of the scientific process.
• Survey generation is also a foundational capability for downstream automated-research tasks, including problem identification, hypothesis generation, research planning, experiment design, and autonomous research agents.

Research objective

• Build a robust, highly capable auto-survey generation system that can automate a substantial portion of the research labor involved in literature review and synthesis.
• Understand how well frontier models can perform end-to-end auto-survey generation when supported by our agentic system, what the limitations and failure types are, and whether researchers can safely and productively use the system.

Challenges

Core challenges

1. Writing surveys is necessarily iterative: every stage is connected, so a purely linear pipeline often fails.
2. There is no single best survey for an area: area boundary can be fluid, and there are different perspectives to view and outline a survey. This makes both choice making and evaluation hard.

Pipeline

1. Paper collection
2. Outline design
3. Paper categorization
4. Paper summary generation
5. Survey writing

Problems that might occur
These are cross-stage failure modes, not separate challenges: each is a concrete way that a later stage can expose a problem upstream and force the workflow to loop back.

• The writer might need facts the summaries did not extract. Expanding summaries to cover every possible field would make them unbounded and bloated.
• Cross-paper contradictions are invisible during single-paper summary generation. The writing stage is the first to compare papers and summaries side by side, and resolving contradictions may require returning to the source papers.
• The outline clarifies the paper’s precise content, which shapes the required literature evidence and therefore paper collection.
• Limited or nonexistent matches during paper categorization suggest either that a section is too granular or irrelevant, or that paper collection failed to retrieve relevant work.

Other challenges

• C1: Dealing with hallucinations and giving citations appropriately: LLMs still struggle to cite accurately and depict statistical figures, sometimes making them up. Both are particularly relevant to survey generation.
• C2: Writing a high-quality survey requires a strong field model—an accurate, organized mental representation of how the field fits together and behaves. Helping an LLM build this model is nontrivial:
  • Small context window: Human experts spend immense time reading and thinking about a field—the equivalent of an immense reasoning and retrieval budget. An LLM must construct a field model from far less information and within a comparatively small context window.
  • In practice, an LLM may imitate the impression of a strong field model rather than build one. It resembles a capable researcher writing about an adjacent field with too little time to research and fully understand it.
• C3: Evaluation is difficult: survey quality is subjective, and assessing whether the output grasps the larger context is harder without preexisting knowledge.
• C4: Avoid making humans the upper bound of survey quality. This directly challenges evaluation and agent creativity because an LLM can anchor on existing human surveys for both tasks.
• C5: Creating figures and tables is challenging because their non-textual nature makes them a non-native language for LLMs.

Intuitions / Innovations

Core challenge

1. An agentic solution to dynamically decide the next step in the pipeline, allowing revision and iteration mingled with development when necessary.
2. Create a taxonomy graph to model different topics and their relationship, rather than requiring strict hierarchical structure native to paper outlines
3. Combining offline taxonomy maintenance and runtime survey generation to allow a broader view for survey writing

Other challenges

• C1: Use temporal-aware paper rankings to ensure work from different time periods is represented during paper collection.
• C1: Ground every major data point—including challenges, methods, and experimental results—with citations. Delegate subagents to verify information at each potential failure step.
• C2: Generate summaries that highlight important ideas without requiring full paper text, and use subagents to exploit the entire context window for broader understanding. [Needs more thinking]
• C3: Begin evaluation in areas where we have expertise and can manually review results. Later, for more automated evaluation, use both quantifiable metrics and LLM-as-judge grounded by official conference peer-review metrics and human-written surveys:
  • Paper collection: recall and precision against a silver truth.
  • Paper outline: structural depth and granularity, ordering, and coverage—for example, whether a major point is covered by at least five papers or at least three recent papers.
  • Survey writing: coverage, correctness / faithfulness, organization / clarity, synthesis / critical analysis, and originality / added value.
• C3: Evaluate intermediary steps with the specific end goal of well-generated surveys in mind. For instance, it may not matter if paper collection locates only 50% of direct RAG-contributing papers if the retrieved 50% is adequate for producing a strong survey.
• C5: Use tools that can create figures programmatically, such as Mermaid.

Solution architecture

Baseline pipeline

1. Collect papers from a constructed corpus via arXiv and Semantic Scholar that directly contribute to $A$.
2. Determine a paper outline.
3. Perform paper categorization, finding relevant papers for each section.
4. Generate summaries for each paper, digesting each into a more concise evidence card.
5. Synthesize an area survey.

II. Experimental Plan and Results

Benchmarks

1. Test the system’s ability to generate surveys for 1,000 unique research areas. These areas should vary in domain (AI / ML, medical, etc.) and granularity (auto-survey generation < deep research < RAG < factuality).
   • Research area list acquisition:
     • Find existing surveys across various domains and broad research fields through lexical search and semantic clustering, then extract the covered topics.
     • Use an LLM to identify gaps by locating research areas without existing surveys across varying granularities.
2. SurGE — End-to-end ASG benchmark with a fixed ~1.09M-paper corpus. It directly tests literature retrieval alongside organization and survey writing, making it useful for evaluating systems that iteratively return to paper collection.
3. SurveyBench — Reader-oriented benchmark using high-quality human surveys as references. It uses reference-grounded evaluation and quiz-based testing of whether a generated survey contains the information researchers need to understand the field.
4. DeepSynth-Eval — Controlled synthesis benchmark where relevant literature is supplied rather than retrieved. It converts human surveys into thousands of fine-grained requirements, allowing precise measurement of which important concepts, comparisons, and insights the generated survey covers or misses.
5. SurveyLens — Large cross-disciplinary benchmark with 1,000 human surveys across 10 academic fields. It tests whether ASG systems generalize beyond CS by combining discipline-specific quality rubrics with alignment against canonical human-written surveys.

Items 2–5 are AI-written and need further manual reading and investigation.

Evaluation metrics

1. Overall metrics considering the following rubrics
2. Comparison or matching w. an existing survey

Substantive quality

1. Coverage — whether it found and included the right content: the problem, major approaches, important recent papers, and major subareas.
2. Correctness / Grounding — whether claims, citations, and figures are accurate and supported.
3. Organization / Clarity — whether the survey is structured well and makes the field easy to understand and navigate.
4. Synthesis / Critical Analysis — whether the survey integrates papers instead of merely summarizing them, helping the reader understand methods, tradeoffs, trends, disagreements, and gaps.
5. Added Value / Originality (optional bonus) — whether the survey contributes a useful perspective beyond existing surveys. This is not a minimum requirement when an existing survey is already excellent.
6. Excitement / Significance (holistic score) — whether the survey is valuable to the research community and should be recommended to researchers.

Technical / deliverable quality

1. Formatting — whether it follows LaTeX formatting and produces a .tex file whose PDF matches published survey papers.
2. Visuals and figures — whether generated tables and figures are readable, relevant, correctly captioned, referenced in the text, and useful to the survey.
3. Component completeness — whether expected components such as the title, abstract, body, references, figures, and tables are all present.

Annotation methods

1. Human annotation, where someone with expertise in the research area annotates a written survey. A small subset of human-annotated surveys can provide important grounding for validating LLM judging through human–LLM grading alignment, and can support more specific rubrics and targets for tuning LLM judging to particular topics.
2. LLM judging:
   • Use several methods:
     • Reference-free judging: The judge receives only the research area, generated survey, and rubric.
     • Reference-grounded judging (when possible): The judge also receives existing surveys for the research area as grounding.
     • Need to consider how to evaluate surveys in areas with no existing surveys. This may require building an evidence packet through deep research or summarized papers.
   • Judging can also apply to intermediary steps, such as outline design or section writing.
   • During development, build a more specific domain-specific rubric for selected research areas, using human annotations to add targeted expert judgment.
3. More quantifiable citation statistics:
   • Recall, precision, and combined F1 for reference-set overlap against human-written surveys.
     • This can evaluate paper collection itself and the papers matched and cited in the written survey.
   • Claim–citation support: whether citations support claims, whether cited claims accurately represent the cited papers, and whether citations refer to real papers.
   • Citation statistics such as the number of distinct references, citation density, and recency.

Baseline / SOTA solutions

1. Simplest baseline: ask an LLM to write a research survey using only its parametric knowledge.
2. Naive RAG: retrieve relevant papers, provide them to the LLM, and use the evidence to generate a survey.
3. General-purpose deep research systems:
   • Codex
   • GPT Deep Research
   • Gemini Deep Research
4. Common baselines used by recent solutions:
   1. AutoSurvey — 2024-06-10
   2. SurveyX — 2025-02-20
   3. SurveyForge — 2025-03-06
5. Current SOTA solutions:
   1. DAS — 2026-08-18
   2. DeepSurvey — 2026-05-28
   3. IterSurvey — 2025-10-24

Ablation study (Testing key technical decisions)

End-to-end ablations

1. Pipeline-based versus agentic or iterative orchestration.
2. With versus without an explicit taxonomy.
3. With versus without offline preprocessing.

Component-scoped ablations

• Paper Collection
  • Remove citation-based expansion.
  • Remove the expensive LLM-based filtering step.
  • Replace the entire system with simple top-K semantic and lexical retrieval.
• Outline Design
  • Remove web research.
  • Remove staged tags.
  • Remove structural variants and comparisons.
  • Remove editorial review, or use naive single-agent review.
• Paper Categorization
  • Remove LLM-based reranking.
• Summary Generation
  • Remove the system and use the title and abstract instead.
• Survey Writing
  • Remove section writers and use a global system instead.

Subject to change based on iterations to each stage and the global-loop implementation.

Robustness test

• Size of the field’s literature — small / medium / large.
• Field evolution — stable / rapidly evolving / major paradigm shift.
• Literature maturity — no prior survey / new field / mature field.
• Corpus quality — distractors / missing critical papers / contradictory evidence.
• Domain — diverse domains.