NanoLongEmbed / NanoSummScreenFD

Overview

NanoLongEmbed / NanoSummScreenFD is the ForeverDreaming subset of SummScreen as used in LongEmbed. Queries are human-written TV episode recaps, and documents are long screenplay-style transcripts. The retrieval goal is to find the episode transcript whose dialogue and scene actions correspond to the recap. The Nano split has 200 queries, 336 documents, and one positive transcript per query. Queries average 600.67 characters, and documents average 30,854.33 characters. Current diagnostics are high across all methods, with BM25 nearly saturated, reranking_hybrid second, and dense retrieval close but lower.

Details

What the Original Data Measures

SummScreen was introduced as an abstractive screenplay summarization dataset pairing TV transcripts with episode recaps. The paper notes that important plot details may be communicated indirectly through dialogue, that many transcript lines are not central to the plot, and that parallel subplots can be separated by scene breaks.

LongEmbed adapts SummScreenFD into retrieval by using the recap as the query and full episode transcripts as candidate documents. The task therefore tests whether a model can match a concise plot summary to a long, dialogue-heavy transcript.

Observed Data Profile

The Nano split contains 200 recap queries, 336 transcript documents, and 200 positive qrel rows. Every query has exactly one positive, with no multi-positive queries. Queries average 600.67 characters. Documents average 30,854.33 characters.

Representative examples come from shows such as Buffy the Vampire Slayer, One Tree Hill, Charmed, Frasier, and Roswell. The queries mention characters, conflicts, supernatural events, relationships, episode outcomes, and major plot points. Transcripts contain dialogue, speaker names, scene labels, and action descriptions.

BM25 Evaluation Profile

The dataset-provided BM25 candidate subset covers the 336-document corpus and achieves nDCG@10 = 0.9813, hit@10 = 1.0000, and recall@100 = 1.0000. BM25 is near ceiling. Recaps contain many character names, show-specific terms, event phrases, and episode-specific details that also appear in the transcript.

The task is long-document retrieval, but not lexically sparse. A recap often names the main characters and conflict directly, making exact matching very powerful. Remaining difficulty is mostly distinguishing adjacent or same-show episodes where recurring character names and settings overlap.

Dense Evaluation Profile

The dense harrier_oss_v1_270m candidate subset covers 336 documents per query and achieves nDCG@10 = 0.9198, hit@10 = 0.9600, and recall@100 = 0.9700. Dense retrieval is strong, but below BM25. It can match paraphrased plot summaries to dialogue-heavy transcripts, but it may blur specific episode identity when several episodes share characters and themes.

This pattern shows that semantic plot similarity is useful but insufficient. In this split, exact character names, episode-specific events, and rare phrases provide the clearest evidence for the correct transcript.

Reranking Hybrid Evaluation Profile

The reranking_hybrid candidate subset contains exactly 100 candidates per query, with no safeguard rows. It achieves nDCG@10 = 0.9443, hit@10 = 0.9700, and recall@100 = 1.0000. Hybrid retrieval improves over dense retrieval and matches BM25's full recall@100, but it does not surpass BM25's top-10 ranking.

Hybrid search is still useful because it combines exact recap terms with semantic plot matching. However, BM25's near-ceiling result indicates that the lexical side already captures most episode identity cues. A reranker should focus on verifying the episode-specific event sequence rather than broad show or character similarity.

Metric Interpretation for Model Researchers

This is a single-positive retrieval task. Hit@10 measures whether the correct episode transcript appears in the top ten, nDCG@10 rewards placing it near the top, and recall@100 measures whether candidate generation keeps it available.

The metric pattern shows a high lexical-overlap long-document task. BM25 is the best observed profile, hybrid is second, and dense is strong but lower. This task is useful for testing whether long-document representations preserve names, scenes, and episode-specific events.

Query and Relevance Type Tendencies

Queries are recap-like summaries that name characters, conflicts, decisions, and outcomes. Documents are long TV transcripts with speaker labels, scene breaks, dialogue, and action descriptions. The positive transcript may contain the relevant evidence across multiple scenes rather than one contiguous span.

The task rewards models that can connect plot summaries to dialogue and scene structure. It also tests whether a model can distinguish episodes from the same show with shared recurring vocabulary.

Representative Failure Modes

BM25 can fail when multiple episodes share characters, settings, and recurring phrases. Dense retrieval can fail by ranking another episode with similar plot themes or character relationships above the exact recap match. Hybrid retrieval can include the correct transcript but still require event-level reranking.

Long-document issues remain relevant: important recap events may be distributed across scenes, while large sections of dialogue do not matter for the summary.

Training Data That May Help

Useful training data includes non-overlapping SummScreen train examples, TV recap-to-transcript retrieval pairs, long dialogue summarization pairs, and adjacent-episode hard negatives from the same show. Training should preserve speaker labels, scene breaks, recurring character names, and parallel subplots.

Comparable evaluation should exclude SummScreenFD evaluation examples, Nano queries, qrels, and positive transcripts likely to overlap with this task.

Model Improvement Notes

Dense retrievers can improve by preserving exact character and episode-event signals while still matching abstractive recap language. Sparse systems already perform near ceiling, but should handle names, capitalization, and script formatting robustly. Rerankers should compare recap events with transcript scenes, especially for same-show hard negatives.

For hybrid systems, NanoSummScreenFD is a high-ceiling calibration task: semantic retrieval helps, but exact plot and character terms are usually the strongest signals.

Example Data

Source Reference Table

Dataset Information

Candidate Subsets

Training and Leakage Metadata

• Original train split: available
• Evaluation split origin: unknown
• Train/eval overlap audit: not_audited
• Leakage note: exclude SummScreenFD evaluation examples, Nano queries, qrels, and positive transcripts likely to overlap with this task
• Multi-positive training: single_positive_question_document_focus
• Useful training data: non-overlapping SummScreen train examples, TV recap-to-transcript retrieval pairs, long dialogue summarization pairs, adjacent-episode hard negatives from the same show