NF Data Portal
Research BriefPortal UpdateFeatured

Faster search, deeper answers: modernizing discovery on the NF Data Portal

Authors

Introduction

A researcher can bring different kinds of questions to the NF Data Portal. Some are straightforward:

  • show studies by a specific investigator name
  • find datasets filtered by data type and funder
  • find an animal model that mentions "glioma"

The search box was meant to handle these, though our old MySQL full-text backend often fell short even here.

But other questions are harder than they look, like a researcher planning an experiment who needs validated MPNST cell lines suitable for a drug screen. Answering this means understanding domain-specific criteria and applying a range filter on doubling time or checking literature references for the "validated" part. While search can return a candidate list, the researcher still has to translate what they want into manual UI filtering, literature cross-references, and more lookups.

Last quarter, the portal significantly upgraded its capability for both kinds of questions with these updates:

  1. Migration to OpenSearch
  2. Alpha release of the AI Portal Assistant

We had in mind the shopping experience at REI, where there's a very good product search box on the website, but also in-store staff for more research and consultation before dropping serious money on gear. The aim for the new Portal Assistant is to get closer to that second experience, handling the kind of resource research and exploration the search box alone cannot.

OpenSearch

Going from MySQL full-text to OpenSearch provides a faster, more forgiving search box meant for most searches. Modern text analyzers and custom synonym configuration let researchers use shorthand terms and eliminate the need for perfect spelling or phrasing. The new autosuggest aids search by showing potential queries as one starts typing.

Portal search box autosuggest dropdown showing three NF1 mutation variant matches for the partial query "c."
Autosuggest surfaces matching mutation variants as soon as a researcher starts typing.

Behind the scenes, we can now fine-tune relevance ranking directly, deciding what additional data to index, what to boost, and how to keep improving results over time, giving us the control and ability to continuously improve that we didn't have with the old MySQL search. We publish benchmark results and continual improvements at this public dashboard.

The Portal Assistant

The Portal Assistant can pull from multiple knowledge sources, some of which is content that OpenSearch currently can't index or link: the NF knowledge graph, help docs, and a limited set of permissibly-licensed portal publications. So you should use it to ask a basic help doc question, but we also expect it to serve users better on those complex (multi-step, highly interconnected) resource research and discovery questions. Two example interactions are illustrated below.

NF Portal Assistant chat answering "how to submit new tool?" with step-by-step instructions linking to the Research Tools Central submission page
A basic help doc question, answered directly with the submission steps.
NF Portal Assistant chat answering a question about MPNST cell line validation, returning a table of cell lines with the publication that characterized each one
A question that requires literature synthesis, answered with a table linking each cell line to the publication that validated it.

However, before it can serve users, we anticipate the question "Is it trustworthy?" The Assistant has been developed with this at top of mind. Thus, seeing how we run testing and evaluation as part of quality assurance provides understanding of where to trust and where to remain conservative.

We run evaluations with custom benchmark datasets upon a version of AstaBench, and the results are analyzed not only at the metrics level but also poured over at the transcript detail. While we have many different evaluations, we focus on explaining the Assistant's expected performance with two evaluations in particular:

  • Portal help and discovery (Phd) evaluation
  • Literature question-answering evaluation

The Assistant is based on a selected model, and the harness runs the same benchmarks against different models and prompts to understand our best configuration. Everything reported below reflects a configuration that has been deployed on the portal, and our aim would be to continue to improve on what's reported here. These are therefore the typical results a researcher can expect in the alpha release. The full data covering different models and question sets is published on our evaluation dashboard, which will be the best source for the latest evaluation results.

Can it find the right resources?

Our first evaluation uses a set of curated discovery questions across nine categories: mutations, animal models, cell lines, antibodies, genetic reagents, studies, publications and authors, investigators, and cross-resource queries. The current set (v1.3) has 46 questions. For each question, we have verified ground truth (a set of resource IDs for the expected relevant results). Each question is also tagged with UX attributes, like whether it's a "hard" question that takes real effort to answer with just the search box or facets, or which type of user it's most relevant to. Is this new assistant good at things that are currently hard, or only at the things that were already easy?

For each question, the metric is recall over the expected identifiers. Put simply, recall is the share of the expected resources that actually turn up in the results. The result below is a run on v1.3, where overall recall is 0.80 (± 0.05).

Difficulty does indeed affect accuracy. Questions the curators marked "baseline" (answerable today with some patience on the portal) score 0.96. The "advanced" ones score 0.70.

Aside from the baseline/advanced axis, search questions were characterized by how "painful" they were to portal users. Again, while search and faceting can yield some results, for "painful" questions the user needs additional legwork to piece together the final answer, sometimes resorting to obscure or impossible workarounds.

The figure below shows that the Assistant outperforms search on more painful questions. This is in part because the agent can use richer, pre-connected information in the knowledge graph, or because it can perform analytical steps that simple search retrieval cannot, making some of the connections for the user. If the Assistant were only good at what search is already good at, the line would fall off a cliff at the right-hand end. Instead, on the twenty questions rated on the highest end of pain, it still returns two thirds of the expected resources. That is, 16 of the 27 questions on the harder end on the current portal were answered perfectly.

Recall against how hard the same question is on today's portal Low pain (12 questions): recall 0.94. Moderate pain (7 questions): recall 0.98. High pain (7 questions): recall 0.84. Very high pain (20 questions): recall 0.66 0.00 0.25 0.50 0.75 1.00 Recall 0.98 0.66 Low n=12 Moderate n=7 High n=7 Very high n=20 How hard the question is on today’s portal
Figure 1: Recall falls as the question gets harder to answer on the current portal, but it does not fall off a cliff. Each question in the benchmark carries a curator's estimate of how painful it is on today's faceted search, from "answerable with minimal effort" to "cannot be answered at all". Question set v1.3 (46 questions), claude-sonnet-5, run 2026-08-18, harness commit aa78cfa.

But the figure below shows where the Assistant still falls short. Animal models are the soft spot, at 0.58, along with the publication category, which contains even more complex questions.

Recall by resource category Antibody (3 questions): recall 1.00. Investigator (2 questions): recall 1.00. Mutation (6 questions): recall 0.98. Genetic reagent (5 questions): recall 0.90. Cell line (9 questions): recall 0.83. Study (5 questions): recall 0.78. Cross-resource (4 questions): recall 0.71. Publication & people (6 questions): recall 0.70. Animal model (6 questions): recall 0.58 0.00 0.25 0.50 0.75 1.00 Recall Antibody n=3 1.00 Investigator n=2 1.00 Mutation n=6 0.98 Genetic reagent n=5 0.90 Cell line n=9 0.83 Study n=5 0.78 Cross-resource n=4 0.71 Publication & people n=6 0.70 Animal model n=6 0.58
Figure 2: Recall by resource category, best to worst. The two categories added in v1.3, Study and Publication & people, are among the weakest, as is Animal model. Several categories rest on very few questions, so n is given for each: a perfect score over two questions is not the same evidence as a perfect score over nine. Question set v1.3, claude-sonnet-5, run 2026-08-18.

One important note is that even the easiest question is slower than the search box. Because the Assistant is slower overall, much of its benefit is on the more complex questions. For one-off straightforward queries such as finding a study by an exact name, using the search box is still highly recommended.

Can it be trusted about the literature?

The Portal Assistant was funded in some part because researchers were interested in a "chat with papers" feature. Currently, the Assistant can answer questions on 139 publications registered on the portal; publications are included if their license permits redistribution and derivative use, and if they are relevant to NF Tools Central, per our alpha-release scoping.

We achieved our target accuracy of at least 95%, but citation capabilities could be improved. Accuracy measures how often the model selects the correct answer from a set of choices, which requires retrieving and understanding the right passage of the paper. Citation is scored with F1, measuring whether the model cited the exact references expected for the answer (papers have numbered passages to make it easier to validate). Both metrics were measured on our "Pub RAG" benchmark of 130 multiple-choice questions from 14 of the papers (about 10% of the total corpus), the same format used by well-known benchmarks LitQA2 and Humanity's Last Exam. Items were originally generated by diverse frontier models against full text, then reviewed and edited by the NF-OSI team.

Curation for this benchmark can be tricky, so it's not a perfect benchmark. As a pertinent example, an earlier version of Humanity's Last Exam was found to have roughly 30% of its text-only chemistry and biology questions in conflict with the peer-reviewed record. We had to resolve a similar issue: two indexed papers reported different NF1 population incidence as 1:2000 and 1:2500, so a question asking for "the" incidence had no correct answer and was dropped. Where a fact was only ambiguous rather than contradictory, the question could be made more specific instead of removed entirely.

Other problems were internal to individual questions. Generated ideal answers sometimes cited mutation variants that never appeared in the source paper, thus every ideal answer was checked against the text. Some questions were too loosely anchored to be answerable, so these were rewritten to anchor more specifically to a study or section. The evaluation does not tell the model which paper to read, reflecting real-world retrieval conditions where finding the right paper out of the entire collection is part of the task. Finally, poor or trivial test items were dropped; these are questions answerable without retrieval or with implausible distractors, which only inflate scores without adding information.

Accuracy is 100% (130 of 130, under both phrasings), which may make this a too-easy evaluation. But it does suggest that the Assistant should be able to answer questions accurately from its current collection of papers.

We actually ran every question two ways: phrased with the exact terminology of the paper, and phrased the way a user would actually ask in more informal language. Accuracy did not differ much between them. Practically, that means you do not have to use the paper's terminology to get the right answer. On the other hand, attribution is a weaker spot.

Citation F1 by question type, precise against natural phrasing Methodological (31 questions): citation F1 0.76 with precise phrasing, 0.73 with natural phrasing. Comparative (29 questions): citation F1 0.76 with precise phrasing, 0.71 with natural phrasing. Factual (46 questions): citation F1 0.70 with precise phrasing, 0.67 with natural phrasing. Inferential (10 questions): citation F1 0.66 with precise phrasing, 0.58 with natural phrasing. Causal (14 questions): citation F1 0.56 with precise phrasing, 0.59 with natural phrasing 0.00 0.25 0.50 0.75 1.00 Citation F1 Methodological n=31 0.76 Comparative n=29 0.76 Factual n=46 0.70 Inferential n=10 0.66 Causal n=14 0.56 Precise phrasing Natural phrasing
Figure 3: Citation F1 by question type. Answer accuracy was 1.00 for every type under both phrasings, so the only thing that moves is attribution. Causal questions are the weakest and the one type where asking naturally scores slightly better than asking in the paper's own terms. Labelled values are the precise phrasing. Pub RAG v1, 130 questions, claude-sonnet-5, run 2026-08-19.

Citation F1 is strict by design. It penalizes both over-citing and under-citing, rather than just checking whether at least one correct reference showed up. We get 0.71 with precise phrasing and 0.68 when the question is asked naturally. In practice, the Assistant reliably reaches the right conclusion while getting nearly a third of its supporting evidence wrong, citing "irrelevant" passages it did not need or omitting ones it did. For comparison, Google AI overviews are accurate 85% to 91% of the time. The takeaway is still to check citations.

One other finding is where attribution breaks down. It varies by paper, and by more than twice as much.

Citation F1 by paper PMC7305302 (7 questions): citation F1 0.81. PMC8742817 (13 questions): citation F1 0.79. PMC8959601 (13 questions): citation F1 0.79. PMC8705852 (8 questions): citation F1 0.78. PMC3484870 (10 questions): citation F1 0.77. PMC9221468 (12 questions): citation F1 0.77. PMC9929861 (13 questions): citation F1 0.73. PMC7952412 (6 questions): citation F1 0.72. PMC8172195 (8 questions): citation F1 0.72. PMC8469245 (6 questions): citation F1 0.69. PMC9646701 (11 questions): citation F1 0.68. PMC9184558 (11 questions): citation F1 0.60. PMC8447793 (7 questions): citation F1 0.47. PMC8150846 (5 questions): citation F1 0.35 0.00 0.25 0.50 0.75 1.00 Citation F1 PMC7305302 n=7 0.81 PMC8742817 n=13 0.79 PMC8959601 n=13 0.79 PMC8705852 n=8 0.78 PMC3484870 n=10 0.77 PMC9221468 n=12 0.77 PMC9929861 n=13 0.73 PMC7952412 n=6 0.72 PMC8172195 n=8 0.72 PMC8469245 n=6 0.69 PMC9646701 n=11 0.68 PMC9184558 n=11 0.60 PMC8447793 n=7 0.47 PMC8150846 n=5 0.35
Figure 4: Citation F1 for each of the 14 indexed papers, precise phrasing. Answer accuracy was 1.00 on every paper, so the spread here is attribution alone, ranging from 0.81 on PMC7305302 down to 0.35 on PMC8150846. Citation F1 by question difficulty, by contrast, is nearly flat (easy 0.68, medium 0.73, hard 0.70). Attribution breaks down by paper, not by how hard the question is. Run 2026-08-19.

A paper at 0.35 and a paper at 0.81 got every answer right. What differs is how cleanly each paper's passages map onto the claim being made. This suggests reviewing curated citations to understand what's different between what we think should be cited vs what the agent thinks should be cited.

Limitations

Both benchmarks have limitations, some already mentioned.

Recall on the resource discovery does not penalize over-retrieval. This selects for agents that surface a large list of resources "to be safe", i.e. the larger list will likely cover truly relevant resources at the cost of including items that are irrelevant. But in future versions we'll likely be moving to another metric.

The Pub QA is small because of the human curation. The question set has had one round of expert review, not the several we would prefer. It's also narrow in scope. Questions require synthesis across passages within a paper, but never across papers, which we consider a harder tier and a promising sequel to this dataset. Ground-truth attribution lists are hard to finalize, too, so citation F1 carries some noise from the ground truth itself, not just from the model.

How the community can collaborate

The community can continue to submit new tools or observations to ensure important resources show up via the portal. Other than that, we also see the Assistant as something the community can help shape. The ability to answer "Is it trustworthy?" depends on having benchmark questions, and we could always use larger and better benchmarks developed with the help of NF experts and portal users.

One part of that is having more people sit down and think about what meaningful questions to include in the first place. The Assistant might not seem trustworthy when it is not using the same reasoning that NF researchers use and expect, and that's another area where we welcome NF expert input.

Portal users can help provide evaluation and feedback. If you have a resource-discovery question the portal handles poorly today, whether or not the Assistant does any better, let us know. If you'd like to collaborate on benchmarking and evaluation, reach out to nfosi@sagebionetworks.org.

If you have ideas for new features, you can submit them at sageb.io/nf-roadmap. Lastly, feel free to subscribe to our newsletter to hear about the next features and enhancements.

Acknowledgements

This work was made possible with funding from the Gilbert Family Foundation.