Overlapping-speech diarization datasets: benchmarks and annotation design Overlapping-speech diarization data: speaker intervals, benchmark choices, collars, reference review, source splits and transparent pilot costs. Overlapping-speech diarization datasets describe who spoke when, including intervals in which more than one person was active. They require a shared timebase, speaker-consistency policy, overlap annotations and an explicit scoring configuration. A transcript with speaker names attached is not necessarily a diarization reference, and a clean two-person recording may not test interruptions or brief acknowledgments. This guide explains benchmark choices, annotation scope, evaluation metrics and budget components across recording domains. Fiund can discuss project-specific sources and reference preparation; source availability, permitted use and commercial terms are established for the actual collection. Primary task Who spoke when, including simultaneous speech Reference layer Time intervals, anonymous speaker labels and evaluation regions Scoring controls Collar, overlap policy, mapping and aggregation Critical split Source-session and relevant speaker groups Price status Separate source access from interval annotation and adjudication Diarization is not the same as transcription Diarization assigns speaking activity to consistent speaker labels over time. Transcription identifies words. A system can get the words right but attribute them to the wrong person, or identify the speakers while missing a short utterance. A useful dataset treats those targets as related but distinct layers. Overlap makes the distinction especially important. When two people speak simultaneously, a single active-speaker label cannot represent the full reference. Decide whether the task allows multiple labels at one time, whether backchannels count as speech and how uncertain activity is marked. The annotation policy should be chosen before a large collection is labeled. What real requests and developer documentation show A [public diarization-data request](https://www.reddit.com/r/LanguageTechnology/comments/1m78mod) specifically asks about overlapping speakers and noise, while another [accented-speaker question](https://www.reddit.com/r/datasets/comments/11zfxlt) seeks two-person recordings with different accents. These are qualitative needs, not verified purchases or estimates of market demand. [pyannote’s original project](https://github.com/pyannote/pyannote-audio) explicitly includes overlap detection, speaker change and embedding components. Its [metric reference](https://pyannote.github.io/pyannote-metrics/reference.html) explains why evaluation needs a collar policy and can restrict scoring to annotated regions. Those primary technical sources support a detailed task specification without implying a Fiund model has been tested or certified. Useful benchmark families [AMI](https://groups.inf.ed.ac.uk/ami/corpus/) is a 100-hour meeting resource with synchronized recording devices and annotations. It is useful for multi-person interaction but differs from telephone calls and edited podcasts. [MagicData-RAMC](https://openslr.org/123/) offers Mandarin conversational material under noncommercial/no-derivatives terms. Source domain and access conditions belong beside any comparison. [AppTek’s call-center evaluation corpus](https://huggingface.co/datasets/apptek-com/apptek_callcenter_dialogues) supplies role and speaker metadata for commissioned service conversations. It is not automatically an overlap-complete diarization reference simply because speaker fields exist. Inspect the released annotation layer and intended task. No benchmark’s total duration guarantees the amount of manually reviewed overlap available for a new project. Define the annotation target precisely Speaker labels can be local to a recording or linked across a collection. Most within-recording tasks do not require personal identity. Use anonymous stable labels and include only identity relationships necessary and permitted for the experiment. A known job role such as agent is different from a speaker identity. Decide whether laughter, whispered speech, nonlexical acknowledgments and distant voices are included. Some applications need all audible participants; others focus on a foreground conversation. Do not leave that decision to annotators file by file. A written policy with examples is more useful than an undefined requirement for perfect labels. Overlap needs its own coverage measures Total hours can conceal a collection with very little simultaneous speech. Request the overlap duration, number of overlap events, typical event length and speaker-count distribution where those figures can be established. Distinguish intentional collection targets from measured source properties. A pilot should contain short interruptions, longer concurrent answers, laughter over speech and quiet backchannels if those are in scope. Clean alternation between speakers is a useful control but not a substitute for overlap. Preserve source context so a short sound can be interpreted rather than forcing every event into an isolated clip. Reference timing and uncertainty Human annotators cannot place every speech boundary at exact sample precision. Define the practical boundary convention and how uncertain regions are represented. A broad uncertain interval should not silently become nonspeech, because that changes both the reference and the score. Use independent review or adjudication on difficult regions and preserve the decisions. If annotations begin from a model proposal, record that lineage and identify the human-reviewed subset. A recent automated export should not be represented as an independent gold standard without evidence of its preparation and review. A schema for reproducible speaker activity Package source recordings, interval annotations and the evaluation map together. The buyer should be able to tell which regions have a reference and which speaker labels can be compared within a file. Artifact Suggested content Important distinction Audio manifest recording_id, source group, timebase, version One source can have several mixes or encodings Speaker table anonymous label, local/global scope, role if allowed Role and personal identity are separate Activity intervals start, end, speaker, event type Several intervals can overlap in time Evaluation regions included and excluded spans, reason Unannotated is not automatically silent Review record policy version, review status, adjudication Automatic proposals are not gold references Scoring configuration collar, overlap setting, mapping, aggregation Two scores are comparable only under stated conditions DER, JER and diagnostic measures Diarization error rate commonly summarizes missed speech, false alarms and speaker confusion relative to the scored reference activity. Jaccard-based measures provide another view of speaker overlap. The exact implementation and mapping matter, so record the tool, version and configuration rather than comparing a bare percentage across papers. The [pyannote metric documentation](https://pyannote.github.io/pyannote-metrics/reference.html) supports collars and evaluation maps. A collar removes boundary regions from scoring; ignoring overlap changes the target again. Neither policy is inherently a complete description of application quality. Report what was scored and retain diagnostic counts so a change in one component does not disappear inside a single total. Collars and overlap settings can change the result Two systems evaluated on the same audio can appear different if one report excludes overlapping speech and another includes it. A boundary collar can also reduce the penalty for small timing disagreement. State these choices in the benchmark specification and in every result table. For a hypothetical comparison, a buyer might request both an application-aligned primary condition and a stricter diagnostic condition. Those settings should be predeclared and interpreted separately. Do not choose the most favorable configuration after seeing which one improves the headline. A score is evidence about the stated protocol, not a universal property of a model. Channels, mixtures and enrollment information Channel-separated source audio can help annotation, but it can also leak information unavailable to the evaluated system. Decide whether the model receives a mono mixture, multiple channels, an exact speaker count or enrollment samples. A system with those hints answers a different question from one operating without them. Keep reference-assistance artifacts separate from test inputs. If annotators use individual tracks to resolve overlap, document that method and ensure the submitted model sees only the allowed representation. All channels and derivatives of a source session should remain in the same split. Otherwise a model can encounter a closely related version during training. Speaker counts and short turns A diarizer can fail by merging similar voices, splitting one person into several labels or missing brief contributions. A collection should include the speaker-count and turn-length patterns relevant to deployment. Long single-speaker stretches alone do not establish performance on rapid conversational exchange. Inspect short acknowledgments and returns after long pauses. Decide whether a speaker who appears only briefly belongs in the evaluated set and how unknown background voices are treated. These details affect both label consistency and interpretation of the result. They should not be left as undocumented exceptions in a preparation script. Provider comparison and budget components A source owner may supply appropriate recordings but no reliable interval labels. An annotation provider may label existing licensed audio but not grant rights to the recordings. A public benchmark may already have references, subject to its terms and domain limitations. Separate those roles when comparing proposals. No public market average specifically for overlap-reviewed diarization data was established here. Request a quote for source access, candidate preparation, interval annotation, independent review and adjudication. The proportion of difficult overlap and number of active speakers can change effort even when elapsed hours remain identical. A per-hour quote should explain which review depth it covers. An explicit hypothetical pilot model Suppose a team plans eight accepted hours with several interaction structures. Using invented placeholders, it allocatesUSD 800 for scoped source access, USD 1,600 for first-pass interval labeling, USD 1,200 for independent overlap review andUSD 600 for adjudication and packaging. The modeled total isUSD 4,200. These assumptions are a planning illustration rather than supplier prices or a Fiund estimate. Before scaling, measure the pilot’s actual annotation disagreements and repair effort. A collection with a similar duration but more brief turns or more simultaneous speakers can require different work. Keep accepted hours, overlap event counts, speaker groups and review status alongside the cost model so a headline rate does not obscure the deliverable. Pilot acceptance and held-out design Choose complete source recordings that contain the difficult events in the specification. Reconcile interval validity, label consistency, evaluation regions and the mapping to the selected audio version. Test the scoring pipeline on deliberately understandable examples before trusting aggregate outputs. Group related sessions and speakers according to the experiment. A system intended for unseen people needs a different held-out boundary from one meant to track recurring participants. If speaker relationships are not available, document the limitation. Do not claim a speaker-disjoint benchmark simply because filenames were randomly assigned to partitions. Reference availability by region A recording can be valid while only some regions have a reviewed diarization reference. Preserve that evaluation map and report the scored duration separately from the whole file. Unreviewed regions should not silently become nonspeech or be counted as independently labeled hours. Discuss diarization requirements with Fiund Contact hello@fiund.com with the recording domains, languages, speaker counts, overlap patterns, allowed model inputs and scoring policy. Include whether the collection is for training or held-out evaluation and what independent reference review is required. Keep source acquisition and annotation expectations separate in the brief. Fiund can discuss candidate source fit, a representative pilot and project-specific preparation. Availability, permissions, pricing and delivery are confirmed for selected material. No category page guarantees a model score, a universally complete reference or access to every named benchmark. Those claims require evidence from the actual collection and experiment. Is a speaker-labeled transcript enough? Not necessarily. A diarization reference needs time intervals and consistent speaker labels, including a policy for simultaneous activity and unannotated regions. Should overlap be excluded from scoring? Choose the policy according to the task and state it explicitly. If deployment includes interruptions, an overlap-excluded headline alone can conceal an important limitation. What is a collar? It is a region around reference boundaries that a scoring protocol can exclude to account for timing uncertainty. The chosen value must accompany the reported result. Can isolated tracks help create the reference? Yes, if their use is documented. Keep them separate from model inputs when the evaluated system is supposed to receive only a mixture. Is a big speech-sector deal a diarization-data price? No. This research did not verify a largest category-specific transaction value. Funding, company acquisitions and licensed voice partnerships should not be relabeled as diarization corpus purchases. Research method: Fiund research desk; public sources, explicit worked-example labels and project-specific availability review. Editorial contact: hello@fiund.com. Sources AMI corpus original project: https://groups.inf.ed.ac.uk/ami/corpus/ MagicData-RAMC official OpenSLR: https://openslr.org/123/ AppTek Call-Center Dialogues official dataset: https://huggingface.co/datasets/apptek-com/apptek_callcenter_dialogues Accented-speaker diarization request: https://www.reddit.com/r/datasets/comments/11zfxlt Overlap/noise diarization request: https://www.reddit.com/r/LanguageTechnology/comments/1m78mod pyannote.audio original project: https://github.com/pyannote/pyannote-audio pyannote.metrics primary metric reference: https://pyannote.github.io/pyannote-metrics/reference.html