Research
40 of 48 correct. Hippocampus against OpenAI File Search and PageIndex.*
Research · 17 September 2026
Same documents, same questions, same answering model. Hippocampus, OpenAI File Search and PageIndex on 2,198 pages of FDA documents, with GPT-5.4 mini writing every answer. Run 17 September 2026.
Result
Sixteen expert questions over ten public FDA reviews, labels and a trial publication, each asked three times. Every system answered through the same small model, so the only difference between rows is how each system found the material.
| System | Correct of 48 | Weighted score | Hard and medium (of 24) | Median response |
|---|---|---|---|---|
| Hippocampus | 40 | 79% | 17 | 20 s |
| OpenAI File Search, average of two runs | 36 (38 and 34) | 64% | 12 | 7 s |
| PageIndex | 33 | 65% | 14 | 8 s |
| OpenAI vector store, single search | 29 | 46% | 8 | 5 s |
| No documents, the model alone | 15 | 25% | 3 | 2 s |
Weighted score counts easy questions once, medium twice, hard three times.
Hippocampus answered 40 of 48, the most in the comparison. OpenAI File Search averaged 36 across two identical runs, and those two runs differed by four answers, so the gap between the top two rows is within run-to-run variation. The gaps to PageIndex, to a single vector search and to the model on its own are larger.
Where the difference is
Lookups are easy for everyone. Every system with document access got 19 to 24 of the 24 single-fact answers, and the model alone got half from memory, since these are public documents.
The separation is on comparison and synthesis. On the 24 medium and hard answers, Hippocampus scored 17, PageIndex 14, File Search 12 on average (14 in one run, 10 in the other) and single vector search 8. Every answer that changed between File Search's two runs was on one of these questions.
The question with no answer
One question asks for a ranking the documents cannot support: which therapy faces generic competition first. Declining is the correct answer. PageIndex declined every time. Hippocampus said each time that the documents cannot support a real ranking, and in two of three answers offered the approval order as an explicitly labelled proxy. File Search gave a ranking as its answer in four of six attempts.
Scanned pages
About 410 of the 2,198 pages have no text layer, including all 300 pages of a 2009 pharmacology review. OpenAI's vector store held no text from that review. Hippocampus and PageIndex both read scanned pages at ingest. No system ended up citing that review, and the one question that draws on it can also be answered from a label.
Speed
Hippocampus is the slowest system here: a median of 20 seconds per answer against 5 to 8 for the others, and over two minutes for its slowest 5%. Single-search answers took about 10 seconds. The slow ones were comparison and synthesis questions, where the model searched many times and read more of the source before answering.
Why a small model
OpenAI File Search only works with OpenAI models. To hold the answering model fixed across every system, all of them used GPT-5.4 mini. Hippocampus is not tied to one model: with Claude Opus 5 answering, the same configuration scored 93 of 96 on this question set. The other systems were not run with Opus, so that figure is not a comparison. A comparison with a frontier OpenAI model answering for every system is under way.
What this comparison does not show
- Hippocampus configurations tested on these questions on 16 to 17 September scored between 37 and 40 of 48. The table shows a run that scored 40, made earlier the same day on the deployed service, separately from the other systems.
- The search guide Hippocampus gives the model was refined in earlier runs on these same sixteen questions. The other systems ran on vendor defaults plus a one-sentence instruction to answer only from the material.
- The grading model is GPT-5, three votes, majority verdict. The answering model and two of the systems tested are also OpenAI's.
- Our PageIndex setup is an adaptation of their published tree-search approach: one step, up to six sections, 12,000 characters per section. PageIndex's own retrieval and chat services were not used. Those limits can disadvantage answers spread across many sections or sitting in long tables.
- File Search ran twice. The other systems ran once, so differences of a few answers can come from variation alone.
Reproduce it
The documents are public. We will load the same ten into a separate Hippocampus workspace so you can ask your own questions, and put the same questions to the other systems with your own accounts. The question set and reference answers belong to the customer they were written with; with their agreement, we take qualified parties through every graded answer under NDA.
*Comparison runs 17 September 2026. 16 questions, three attempts each, over 10 public FDA documents (2,198 pages). Answering model GPT-5.4 mini for every system. Grading by GPT-5, three votes, majority verdict, in an evaluation designed and run by Thalius. Per-answer records retained.