NanoDAPFAM / NanoDAPFAMOutTitlAbsToFullText

Overview

NanoDAPFAMOutTitlAbsToFullText is an English patent-family retrieval task. The query contains only the title and abstract of a source patent family, while target documents contain full patent-family text. Positives are DAPFAM OUT-domain citation relations, meaning relevant target families do not share IPC3 domain with the source.

This split tests compact-query cross-domain prior-art retrieval over very long target documents. The model must infer a cross-domain technical relationship from a short patent summary and find it inside full-text records from different fields.

Details

What the Original Data Measures

DAPFAM benchmarks family-level patent retrieval using citation links and IPC3 domain labels. OUT-domain positives are citation-related families outside the source domain. This split uses title-abstract queries and full-text targets.

It measures cross-domain retrieval with minimal query context and maximal target length. This is one of the more realistic settings for difficult prior-art discovery from a compact patent summary.

Observed Data Profile

This Nano split contains 200 queries, 10,000 documents, and 1,259 positive qrels. There are 159 multi-positive queries. Positives per query average 6.30, with a minimum of 1, median of 4.0, and maximum of 20. Queries average 786.61 characters, while target documents average 71,902.31 characters.

The short query and very long target create both semantic and length challenges. Relevant cross-domain evidence may be sparse inside the target full text.

BM25 Evaluation Profile

BM25 reaches nDCG@10 of 0.0638, hit@10 of 0.2100, and recall@100 of 0.1875 with a top-500 candidate pool. Exact title and abstract terms are not reliable enough for cross-domain search, and full-text targets introduce many incidental matches.

BM25 can help when a cross-domain target uses shared mechanism or material names, but most OUT-domain positives require more abstract matching than term frequency provides.

Dense Evaluation Profile

The dense harrier-oss-270m profile is strongest by nDCG@10 and hit@10, with nDCG@10 of 0.0952, hit@10 of 0.3350, and recall@100 of 0.2518. Dense retrieval improves over BM25 by matching technical similarity beyond shared terms.

The absolute scores remain low because the query is compact and the relevant target is cross-domain and very long. Dense retrieval must represent analogy or technical transfer, not just topical similarity.

Reranking Hybrid Evaluation Profile

The reranking_hybrid candidate subset reaches nDCG@10 of 0.0858, hit@10 of 0.3050, and recall@100 of 0.2653. It uses top-100 candidates with optional rank-101 safeguards; 74 rows contain 101 candidates and 74 safeguard-positive rows are recorded. Hybrid provides the best recall@100, while dense has better top-rank ordering.

This suggests that lexical evidence adds some additional positives but also introduces noise. A downstream reranker would need to identify the cross-domain relationship among many weakly related candidates.

Metric Interpretation for Model Researchers

This is a difficult short-query, long-target, cross-domain patent retrieval task. The gap between BM25 and dense shows that semantic retrieval is necessary, but the low absolute recall shows that current dense candidates still miss many positives.

The task is useful for testing cross-domain prior-art discovery and long-document retrieval, especially under limited query context.

Query and Relevance Type Tendencies

Queries are title-abstract summaries. Documents are full patent texts. Positives are citation-related families outside the query's IPC3 class.

The relevant relationship may involve analogous functions, mechanisms, control methods, materials, or processes that are described differently across domains.

Representative Failure Modes

BM25 retrieves long documents with incidental term overlap. Dense retrieval retrieves broadly related technology but misses citation relevance. Hybrid retrieval can recover more positives by rank 100 but may rank lexical distractors above them.

Training Data That May Help

Useful training data includes cross-domain title-abstract patent retrieval, cross-IPC patent citation pairs, and long-target prior-art search. Training should exclude NanoDAPFAM evaluation family IDs, positives, and qrels.

Synthetic data should pair compact source patent summaries with long full-text patent records from different technical classes.

Model Improvement Notes

Improving this task requires cross-domain semantic expansion from short summaries and evidence extraction from long targets. Models should learn functional analogy and technical effect matching across IPC classes.

Passage-level target representations are likely important because a full-text patent may contain the relevant cross-domain evidence only in a small section.

Example Data

Source Reference Table

Dataset Information

Candidate Subsets

Training and Leakage Metadata

• Original train split: not_confirmed
• Evaluation split origin: DAPFAM OUT-domain title-abstract to full-text patent-family retrieval
• Train/eval overlap audit: not_audited
• Leakage note: exclude NanoDAPFAM evaluation family IDs, positives, and qrels
• Multi-positive training: citation_family_multi_positive
• Useful training data: cross-domain title-abstract patent retrieval, cross-IPC patent citation pairs, long-target prior-art search