Paper Collection Overview

problem def

input: a research area $A$, and an optional description of the area, $A_d$
output: a set of papers $P$ where each paper $p$‘s main contribution is in-scope to $A$, or any direct-order component $A_c$

solution

paper collection is broadly a two stage system:

stage 1:
focused on recall; finding a broad set of paper candidates that are plausibly in-scope for the area.

stage 2:
focused on precision; filtering candidates and identifying papers whose main contribution is in scope for the area.

how each stage works

stage 1:

• create an area card
  • area name
  • area definition
  • semantic query
  • lexical match phrases
• perform retrieval via 3 routes, and take the union
  • keyword-search — using lexical match phrases
  • vector similarity — using embedded semantic query
  • citation-based expansion

evaluation

we have a silver truth, being the generated topic / primary-topic for papers in the corpus
however, the silver truth has flaws:

• many papers in the corpus don’t yet have a summary, and thus a topic/primary-topic generated yet
• there can be classification errors, meaning that the recall / precision is imperfect

however, using this silver truth, we can compute:

let:

• $R$ be the set of papers retrieved
• $L$ be the set of papers with the evaluated silver-label field populated
• $S\subseteq L$ be the set of silver-positive papers labeled with the area

then:

• silver recall: $|R\cap S|/|S|$ — the fraction of silver-positive papers that were retrieved.
• silver precision: $|R\cap S|/|R\cap L|$ — the fraction of retrieved, labeled papers that are silver-positive.

papers in $R\setminus L$ are reported separately as unevaluated and are not included in the silver-precision denominator. these metrics measure agreement with the silver truth rather than true relevance: missing and incorrect labels can distort both metrics.

the metrics can be calculated separately using either primary_topic = area or area in topics as the definition of $S$, with $L$ defined using the corresponding field.

questions to consider

what does it mean for the main contribution to be in-scope?
for example for RAG, should papers on ANN / vector search be included? these can include key, helpful citations for explaining associated retrieval steps. however, it’s main contribution is further upstream from RAG. how do you differentiate between too-far upstream vs. not?

this is a design decision. we can have two different paradigms, one where we have one set of papers, which are all in-scope, while another is that we have two sets of papers, where one set directly contributes to rag, and another only enables it.

benefit to two sets: you can end up w. way too many papers if they’re all already in one bucket. for instance, imagine there are 2k embedding-focused papers. this would quickly overwhelm the system, and reduce the space that it has to intake information from rag papers
negative: adds a lot of complexity, as we’re basically doing this categorization during paper collection

answer for now:
a paper is in-scope when it is a direct contribution to the area, or a contribution to a first-order component of the area.