6# Literature Review Agent (Step 3)
7
8Faithful implementation of the Hybrid Literature Agent from PaperOrchestra
9(Song et al., 2026, arXiv:2604.05018, §4 Step 3, App. D.3, App. F.1 p.46).
10
11**Cost: ~20–30 LLM calls.** This is one of the two longest steps (the other is
12plotting). Wall-time floor is set by Semantic Scholar's 1 QPS verification
13limit.
14
15## Inputs
16
17- workspace/outline.json — specifically intro_related_work_plan with the
18 Introduction search directions and the 2-4 Related Work methodology
19 clusters
20- workspace/inputs/conference_guidelines.md — used to derive cutoff_date
21- workspace/inputs/idea.md, workspace/inputs/experimental_log.md — for
22 framing the Intro and grounding the Related Work positioning
23
24## Outputs
25
26- workspace/citation_pool.json — verified Semantic Scholar metadata for
27 every paper that survived verification
28- workspace/refs.bib — BibTeX file generated from the verified pool
29- workspace/drafts/intro_relwork.tex — drafted Introduction and Related
30 Work sections, written into the template, with the rest of the template
31 preserved verbatim
32
33## Two-phase pipeline (App. D.3)
34
35```
36PHASE 1 — Parallel Candidate Discovery
37 For each search direction in introduction_strategy.search_directions:
38 For each limitation_search_query in each related_work cluster:
39 - Use the host's web search tool to discover up to ~10 candidate papers.
40 - Run up to 10 discovery queries in parallel (host-permitting).
41 - Collect (title, snippet, url) tuples — no verification yet.
42 → PRE-DEDUP before Phase 2 (see Step 1.5 below)
43
44PHASE 2 — Sequential Citation Verification (1 QPS, with cache)
45 For each candidate (after pre-dedup), sequentially:
46 0. Check s2_cache.json first (scripts/s2_cache.py --check).
47 If HIT: use cached response, skip live S2 call. No throttle needed.
48 If MISS: proceed with live request below.
49 1. Query Semantic Scholar by title:
50 GET https://api.semanticscholar.org/graph/v1/paper/search?query=<title>
51 &fields=title,abstract,year,authors,venue,externalIds&limit=5
52 (Public endpoint, no key. Throttle to 1 QPS for live requests only.)
53 2. Store the S2 response in cache: s2_cache.py --store.
54 3. Pick the top hit. Check Levenshtein title ratio against the original
55 candidate title. If ratio < 70: discard.
56 4. Bonus: if year and venue exactly align with hints, add a +5 point
57 match-quality bonus.
58 5. Require: abstract is non-empty.
59 6. Require: paper.year (or month if known) strictly predates cutoff_date.
60 Months default to day-1: e.g., "October 2024" → 2024-10-01.
61 7. If all checks pass, add to verified pool.
62 After all candidates are verified, dedup by Semantic Scholar paperId.
63```
64
65The host agent does the LLM/web work; the deterministic helpers in scripts/
66do the math.
67
68## Step-by-step
69
70### 0. Derive cutoff_date
71
72Parse conference_guidelines.md for the submission deadline. The paper aligns
73research cutoff with venue submission deadline (App. D.1):
74
75| Venue | Cutoff |
76|---|---|
77| CVPR 2025 | Nov 2024 |
78| ICLR 2025 | Oct 2024 |
79| Other | One month before the stated submission deadline |
80
81Encode as YYYY-MM-DD. Months default to day-1 (e.g., 2024-10-01).
82
83### 1. Phase 1: Parallel Candidate Discovery
84
85From outline.json:
86
87- All introduction_strategy.search_directions (3-5 queries)
88- For each cluster in related_work_strategy.subsections:
89 - The cluster's sota_investigation_mission becomes a search query
90 - All limitation_search_queries (1-3 each)
91
92For each query, **use your host's web search tool** (e.g., WebSearch in
93Claude Code, @web in Cursor, the search tool in Antigravity). Collect the
94top ~10 candidates per query: title, abstract snippet, source URL.
95
96If your host supports parallel sub-tasks, fire up to 10 concurrent search
97queries. If not, run sequentially — slower but functionally equivalent.
98
99#### Optional: Exa as a Phase 1 backend
100
101If your host has no native web search, OR you want a research-paper-focused
102backend with better signal-to-noise, you can use [Exa](https://exa.ai) via
103the bundled scripts/exa_search.py helper. It is **opt-in** and reads
104EXA_API_KEY from the environment — the repo never commits a key.
105
106```bash
107export EXA_API_KEY="your-key-here" # get one at https://dashboard.exa.ai/
108python skills/literature-review-agent/scripts/exa_search.py \
109 --query "Sparse attention long context transformers" \
110 --num-results 15 \
111 --discovered-for "related_work[2.1]"
112```
113
114Output is a normalized candidate list ready to merge into
115raw_candidates.json. Phase 2 verification (Semantic Scholar fuzzy match,
116cutoff, dedup) is unchanged. See references/exa-search-cookbook.md for
117the full recipe, query patterns, cost estimates, and security notes.
118
119#### Optional: Tavily as a Phase 1 backend
120
121If your host has no native web search, OR you want an LLM-optimized search
122backend with high relevance scoring, you can use [Tavily](https://tavily.com)
123via the bundled scripts/tavily_search.py helper. It is **opt-in** and reads
124TAVILY_API_KEY from the environment — the repo never commits a key.
125
126```bash
127export TAVILY_API_KEY="tvly-your-key-here" # get one at https://app.tavily.com
128python skills/literature-review-agent/scripts/tavily_search.py \
129 --query "Sparse attention long context transformers" \
130 --num-results 15 \
131 --academic \
132 --discovered-for "related_work[2.1]"
133```
134
135Output is a normalized candidate list ready to merge into
136raw_candidates.json. Phase 2 verification (Semantic Scholar fuzzy match,
137cutoff, dedup) is unchanged. See references/tavily-search-cookbook.md for
138the full recipe, query patterns, cost estimates, and security notes.
139
140Combine all discovered candidates into a single working list. Tag each with
141the originating query ID so you can later attribute it to "intro" vs
142"related_work[i]".
143
144### 1.5. Pre-dedup before Phase 2
145
146**Always run this before starting Phase 2.** Multiple search queries routinely
147return the same papers (e.g., "Attention is All You Need" appears in almost
148every NLP discovery query). Verifying duplicates wastes 30-40% of S2 quota
149at 1 QPS.
150
151```bash
152python skills/literature-review-agent/scripts/pre_dedup_candidates.py \
153 --in workspace/raw_candidates.json \
154 --out workspace/deduped_candidates.json
155# Prints: "150 candidates → 97 unique (53 duplicates removed)"
156```
157
158Use workspace/deduped_candidates.json as input to Phase 2.
159
160### 2. Phase 2: Sequential Verification via Semantic Scholar (with cache)
161
162For each candidate in deduped_candidates.json, in **sequential** order:
163
164**Step A — check cache first** (no S2 call, no throttle needed):
165```bash
166python skills/literature-review-agent/scripts/s2_cache.py \
167 --cache workspace/cache/s2_cache.json \
168 --check "<candidate title>"
169# exit 0 + prints JSON → use cached response, skip Step B
170# exit 1 → proceed to Step B
171```
172
173**Step B — live S2 request** (cache MISS only, throttle to 1 QPS):
174
175**Preferred:** use the bundled scripts/s2_search.py helper — it handles
176auth, retries, and 429 back-off automatically:
177
178```bash
179python skills/literature-review-agent/scripts/s2_search.py \
180 --query "<URL-decoded candidate title>" --limit 5
181# If SEMANTIC_SCHOLAR_API_KEY is set the key is forwarded automatically.
182# If not, the public unauthenticated endpoint is used (≤1 QPS, still works).
183```
184
185Check whether the key is configured before starting Phase 2:
186
187```bash
188python skills/literature-review-agent/scripts/s2_search.py --check-key
189```
190
191**Fallback:** if you prefer your host's URL fetch tool, GET:
192```
193https://api.semanticscholar.org/graph/v1/paper/search?query=<URL-encoded title>&limit=5&fields=title,abstract,year,authors,venue,externalIds
194```
195Add header x-api-key: <SEMANTIC_SCHOLAR_API_KEY> if the env var is set.
196Be polite: ≤1 request per second for live requests. Cache hits are free.
197
198**Step C — store in cache** (after every successful live request):
199```bash
200python skills/literature-review-agent/scripts/s2_cache.py \
201 --cache workspace/cache/s2_cache.json \
202 --store "<candidate title>" \
203 --response '<full S2 JSON response>'
204```
205
206For the top hit:
207
208```bash
209python skills/literature-review-agent/scripts/levenshtein_match.py \
210 --candidate "Original candidate title" \
211 --found "S2 returned title"
212# prints integer 0-100. Discard if < 70.
213```
214
215Then check the temporal cutoff:
216
217```bash
218python skills/literature-review-agent/scripts/check_cutoff.py \
219 --paper-year 2024 \
220 --paper-month 9 \
221 --cutoff 2024-10-01
222# exit 0 if strictly predates, exit 1 if not
223```
224
225If both checks pass AND the abstract is non-empty, append the paper's full
226S2 metadata to the verified pool.
227
228### 3. Dedup and assemble the pool
229
230After all candidates are verified:
231
232```bash
233python skills/literature-review-agent/scripts/dedupe_by_id.py \
234 --in raw_pool.json \
235 --out workspace/citation_pool.json
236```
237
238The dedupe script keys on paperId (Semantic Scholar's internal unique ID),
239falling back to externalIds.DOI, then externalIds.ArXiv, then a
240normalized title.
241
242The script also computes and writes min_cite_paper_count =
243floor(0.9 * len(papers)) — the minimum number of papers the writing step
244must cite (the paper's ≥90% integration rule, App. D.3).
245
246**Immediately after dedupe_by_id.py**, validate and auto-fix the pool schema:
247
248```bash
249python skills/literature-review-agent/scripts/validate_pool.py \
250 --pool workspace/citation_pool.json --fix
251# Catches and fixes authors-as-strings, reports missing required fields.
252# Must pass before proceeding to Step 4.
253```
254
255### 3.5. Cross-index verification (Crossref + OpenAlex)
256
257Semantic Scholar is one index and can return a plausible record for a paper
258that does not exist, or attach wrong metadata. Re-check every S2-verified
259paper against two **independent** indices before building the bibliography —
260this is the practical defense against hallucinated citations leaking in.
261
262```bash
263# Optional but recommended: a polite-pool email gives faster, more reliable
264# service. The repo never commits an address.
265export PAPER_ORCHESTRA_MAILTO="you@example.com"
266
267python skills/literature-review-agent/scripts/cross_verify.py \
268 --pool workspace/citation_pool.json --inplace
269# Annotates each paper with a cross_verification field and writes
270# workspace/cross_verification_report.json.
271# exit 0 = all corroborated; exit 1 = WARN (something flagged or an index
272# was unreachable); exit 2 = usage error.
273```
274
275This is a **WARN gate, not a hard gate** (like validate_consistency.py): it
276flags suspicious citations but does not block the pipeline or delete anything.
277Review the low and conflict tiers in the report:
278
279- high — corroborated by ≥1 external index → keep.
280- medium — corroborated but year disagrees → keep, spot-check the year.
281- low — not found in Crossref or OpenAlex → **review by hand**. Note that
282 arXiv-only preprints (no DOI) are a common benign cause; low means
283 "could not corroborate," not "fabricated." S2 already confirmed it exists.
284- conflict — pool DOI disagrees with the external DOI → likely wrong record.
285
286Drop only the entries you genuinely cannot corroborate, then re-run
287dedupe_by_id.py onward. If both indices are unreachable (offline), the script
288degrades gracefully and the pipeline continues on S2 verification alone.
289
290See references/cross-index-verification.md for the full rationale, confidence
291tiers, and the arXiv false-positive note.
292
293### 4. Build the BibTeX file
294
295```bash
296python skills/literature-review-agent/scripts/bibtex_format.py \
297 --pool workspace/citation_pool.json \
298 --out workspace/refs.bib
299```
300
301The script generates citation keys deterministically from `firstauthor + year
302+ first significant word of title (e.g., vaswani2017attention`). It writes
303out only @article / @inproceedings / @misc entries — never invents
304fields. It also writes the canonical bibtex_key back into each paper record
305in citation_pool.json.
306
307**Immediately after bibtex_format.py**, sync keys in intro_relwork.tex:
308
309```bash
310python skills/literature-review-agent/scripts/sync_keys.py \
311 --pool workspace/citation_pool.json \
312 --tex workspace/drafts/intro_relwork.tex \
313 --inplace
314# Replaces every \cite{agent_key} with \cite{canonical_bibtex_key}.
315# Eliminates citation_coverage gate failures caused by key mismatch.
316```
317
318These two steps replace the manual Python snippets that were previously
319required. The pipeline is now:
320
321```
322dedupe_by_id → validate_pool --fix → cross_verify --inplace → bibtex_format → sync_keys
323```
324
325### 5. Draft Introduction + Related Work
326
327This is where you (the host agent) actually write text. Load the
328**verbatim Literature Review Agent prompt** at references/prompt.md.
329Substitute the template placeholders:
330
331| Placeholder | Value |
332|---|---|
333| intro_related_work_plan | full JSON object from outline.json |
334| project_idea | contents of idea.md |
335| project_experimental_log | contents of experimental_log.md |
336| citation_checklist | the BibTeX keys from refs.bib |
337| collected_papers | list of {key, title, abstract} from citation_pool.json |
338| paper_count | len(citation_pool.papers) |
339| min_cite_paper_count | from citation_pool.json |
340| cutoff_date | the date you derived in Step 0 |
341
342**Also prepend the Anti-Leakage Prompt** from
343../paper-orchestra/references/anti-leakage-prompt.md.
344
345Run your LLM with the combined prompt against template.tex. The agent's
346job is to fill in the empty Introduction and Related Work sections of the
347template **and leave everything else untouched**. Output: the full
348template.tex with those two sections filled. Save to
349workspace/drafts/intro_relwork.tex.
350
351### 5b. Append §2 to research_brief.md
352
353After intro_relwork.tex is drafted and before the citation coverage check,
354append §2 to workspace/research_brief.md (see skills/shared/research_brief_template.md).
355
356Template:
357
358```markdown
359## §2 · Literature Landscape
360_Written by: literature-review-agent, Step 3_
361
362**What the literature says about the core claim:** <2-3 sentence synthesis>
363
364**Strongest prior work (must address in the paper):**
365- <bibtex_key>: <why this is the strongest comparator or predecessor>
366
367**Gaps confirmed by the literature:** <list>
368
369**Baseline comparisons — verification status:**
370| Baseline | In citation_pool? | Confidence tier |
371|---|---|---|
372
373**Related Work cluster coverage:**
374| Cluster | Papers found | Notes |
375|---|---|---|
376
377**Anything the section-writing agent should know:** <important context>
378```
379
380This synthesises what was actually found — not what the outline assumed.
381
382### 6. Verify ≥90% citation coverage
383
384```bash
385python skills/literature-review-agent/scripts/citation_coverage.py \
386 --tex workspace/drafts/intro_relwork.tex \
387 --pool workspace/citation_pool.json
388# exit 0 if ≥90% of pool is cited; exit 1 otherwise
389```
390
391If the gate fails, re-prompt the writing step explicitly listing the missing
392keys and asking the agent to integrate them where contextually appropriate.
393
394## Critical rules from the prompt
395
396These are excerpted from references/prompt.md. The host agent MUST honor
397them on the writing call:
398
399- **Cite ONLY from collected_papers.** Never invent BibTeX keys, never
400 reference papers not in the pool.
401- **Cite at least min_cite_paper_count of them** in Intro + Related Work
402 combined.
403- **TIMELINE RULE**: Do not treat any papers published after cutoff_date
404 as prior baselines to beat. They are concurrent work only.
405- **EVALUATION RULE**: Do not claim our method beats / achieves SOTA over a
406 specific cited paper UNLESS that paper is explicitly evaluated against in
407 experimental_log.md. Frame other recent papers strictly as concurrent,
408 orthogonal, or conceptual work.
409- **Output format**: return the full code for the updated template.tex,
410 with the two empty sections (Introduction and Related Work) filled in,
411 and **all the other code** (packages, styles, other sections) **identical
412 to the original** template.tex.
413- Wrap output in ``` `latex ... ` ``` fences.
414- Do not change \usepackage[capitalize]{cleveref} to cleverref (there is
415 no cleverref.sty).
416
417## Degraded mode (no web search)
418
419If your host has no web search tool, switch to degraded mode:
420
4211. If the user has placed a pre-built workspace/inputs/refs.bib in the
422 workspace, load it directly into workspace/refs.bib and skip Phase 1
423 and Phase 2.
4242. Otherwise, emit workspace/drafts/intro_relwork.tex containing the
425 template with two TODO markers in the Intro and Related Work sections,
426 and tell the user the pipeline cannot complete Step 3 without web search.
427
428## Resources
429
430- references/prompt.md — verbatim Literature Review Agent prompt from App. F.1
431- references/discovery-pipeline.md — Phase 1 + Phase 2 explained in detail
432- references/verification-rules.md — Levenshtein cutoff, year alignment, dedup
433- references/citation-density-rule.md — the ≥90% integration rule
434- references/s2-api-cookbook.md — Semantic Scholar URLs, fields, rate limits
435- references/cross-index-verification.md — Crossref + OpenAlex corroboration, confidence tiers, arXiv false-positive note
436- references/exa-search-cookbook.md — optional Exa backend for Phase 1 (research-paper-focused web search)
437- references/tavily-search-cookbook.md — optional Tavily backend for Phase 1 (LLM-optimized web search)
438- scripts/pre_dedup_candidates.py — **NEW** dedup Phase 1 candidates before Phase 2 (saves 30-40% S2 quota)
439- scripts/s2_cache.py — **NEW** persistent S2 response cache (eliminates re-verification on re-runs)
440- scripts/validate_pool.py — **NEW** validate & auto-fix citation_pool.json schema (authors format)
441- scripts/sync_keys.py — **NEW** sync cite keys in .tex with canonical bibtex_keys after bibtex_format.py
442- scripts/levenshtein_match.py — fuzzy title match (ratio > 70)
443- scripts/check_cutoff.py — date cmp w/ month → day-1 default
444- scripts/dedupe_by_id.py — dedup verified pool by S2 paperId
445- scripts/bibtex_format.py — build refs.bib from JSON pool
446- scripts/citation_coverage.py — ≥90% citation coverage gate
447- scripts/s2_search.py — **NEW** Semantic Scholar title-search helper; reads SEMANTIC_SCHOLAR_API_KEY from env (optional — falls back to unauthenticated)
448- scripts/exa_search.py — optional Exa Phase 1 backend (reads EXA_API_KEY from env)
449- scripts/tavily_search.py — optional Tavily Phase 1 backend (reads TAVILY_API_KEY from env)
450- scripts/crossref_client.py — **NEW** Crossref title/DOI lookup for cross-index corroboration (no key; reads CROSSREF_MAILTO / PAPER_ORCHESTRA_MAILTO)
451- scripts/openalex_client.py — **NEW** OpenAlex title/DOI lookup for cross-index corroboration (no key; reads OPENALEX_MAILTO / PAPER_ORCHESTRA_MAILTO)
452- scripts/cross_verify.py — **NEW** cross-corroborate the S2-verified pool against Crossref + OpenAlex; flags hallucinated citations (WARN gate)
453- skills/shared/research_brief_template.md — **NEW** §2 schema; append after intro_relwork.tex is drafted
454