NanoDAPFAM / NanoDAPFAMOutTitlAbsClmToTitlAbsClm

Overview

NanoDAPFAMOutTitlAbsClmToTitlAbsClm is an English patent-family retrieval task. Both queries and targets contain title, abstract, and claims, while relevance is restricted to DAPFAM OUT-domain citation relations. Positive target families are technically related but do not share IPC3 domain with the source.

This split tests cross-domain claim-to-claim retrieval. Claims provide more technical evidence than abstracts, but the lack of shared IPC3 domain means relevant families may use different terminology. The task is a hard benchmark for patent analogy and technology-transfer retrieval.

Details

What the Original Data Measures

DAPFAM evaluates family-level patent retrieval using citation labels and domain relations. OUT-domain relations identify citation-positive families outside the source's IPC3 technical class. This split represents both sides with title, abstract, and claims.

The task measures whether claim-rich semantic matching can recover cross-domain cited families when both source and target provide detailed but domain-specific legal and technical language.

Observed Data Profile

This Nano split contains 200 queries, 10,000 documents, and 1,259 positive qrels. There are 159 multi-positive queries. Positives per query average 6.30, with a minimum of 1, median of 4.0, and maximum of 20. Queries average 9,315.66 characters, and target documents average 7,257.21 characters.

The field lengths are balanced and claim-rich, but cross-domain vocabulary mismatch remains substantial.

BM25 Evaluation Profile

BM25 reaches nDCG@10 of 0.0640, hit@10 of 0.2400, and recall@100 of 0.1684 with a top-500 candidate pool. It is stronger here than in the title-abstract target OUT variant because target claims add more lexical anchors.

Even so, BM25 remains weak. Cross-domain positives may share mechanisms or functions without sharing many exact terms. Claim boilerplate also creates false matches among non-relevant targets.

Dense Evaluation Profile

The dense harrier-oss-270m profile is strongest by nDCG@10 and recall, reaching nDCG@10 of 0.0952, hit@10 of 0.3100, and recall@100 of 0.2462. Dense retrieval captures more cross-domain relevance than BM25 by abstracting over terminology.

The scores remain low because cross-domain patent relevance is subtle. A model must compare technical effects and claim structure across different domains, not just topical similarity.

Reranking Hybrid Evaluation Profile

The reranking_hybrid candidate subset reaches nDCG@10 of 0.0811, hit@10 of 0.3100, and recall@100 of 0.2423. It uses top-100 candidates with optional rank-101 safeguards; 72 rows contain 101 candidates and 72 safeguard-positive rows are recorded. Dense and hybrid tie on hit@10, but dense is better by nDCG and recall.

The hybrid profile suggests that BM25 adds some useful claim-term candidates but also adds noise. Dense retrieval remains the best first signal for cross-domain claim matching.

Metric Interpretation for Model Researchers

This is a difficult cross-domain patent retrieval task with claim-rich evidence on both sides. Scores are much lower than IN-domain variants, showing that shared IPC vocabulary is a major advantage in patent retrieval. OUT-domain retrieval requires more abstract technical reasoning.

The target claims improve lexical and dense matching relative to title-abstract targets, but not enough to make the task easy. Candidate recall remains a major bottleneck.

Query and Relevance Type Tendencies

Queries and documents contain title, abstract, and claims. Positives are citation-related families outside the source IPC3 domain. Relevance may be based on functional equivalence, transferred mechanism, materials behavior, or analogous control/process structure.

Representative Failure Modes

BM25 retrieves claim-rich targets with shared formal language but no cross-domain relevance. Dense retrieval retrieves broad technical analogies that are not citation positives. Hybrid retrieval can tie hit@10 with dense but rank positives lower due to lexical distractors.

Training Data That May Help

Useful training data includes cross-domain patent claim retrieval, cross-IPC citation prediction, and patent technology-transfer retrieval. Training should exclude NanoDAPFAM evaluation families and qrels.

Synthetic data should create claim-rich patent records from different technical classes, with positives that are cited cross-domain patent families and hard negatives that share either function or vocabulary without citation relation.

Model Improvement Notes

Improving this task requires models that compare claim elements across domains. Useful representations should capture function, mechanism, technical effect, and problem-solution structure even when field terminology changes.

For reranking, claim-element alignment and citation-graph-aware training may be more effective than lexical overlap alone.

Example Data

Source Reference Table

Dataset Information

Candidate Subsets

Training and Leakage Metadata

• Original train split: not_confirmed
• Evaluation split origin: DAPFAM OUT-domain title-abstract-claims to title-abstract-claims patent-family retrieval
• Train/eval overlap audit: not_audited
• Leakage note: exclude NanoDAPFAM evaluation families and qrels
• Multi-positive training: citation_family_multi_positive
• Useful training data: cross-domain patent claim retrieval, cross-IPC citation prediction, patent technology-transfer retrieval