Speech intelligence / Research guide

Overlapping-speech diarization datasets: benchmarks and annotation design

Overlapping-speech diarization data: speaker intervals, benchmark choices, collars, reference review, source splits and transparent pilot costs.

FIELD NOTESBY FIUND
01Signal02Transcript03Reference

Audio · language · evaluation

Cited references
7 cited sources
Inside this guide
1 worked example
Reading time
9 minutes
Reviewed
2026-11-12

Prepared by the Fiund research desk from the cited public documentation. Technical examples describe their evidence and limits. Research method & editorial contact · Suggest a correction

Reading tools
The takeaway

Overlapping-speech diarization datasets describe who spoke when, including intervals in which more than one person was active. They require a shared timebase, speaker-consistency policy, overlap annotations and an explicit scoring configuration. A transcript with speaker names attached is not necessarily a diarization reference, and a clean two-person recording may not test interruptions or brief acknowledgments. This guide explains benchmark choices, annotation scope, evaluation metrics and budget components across recording domains. Fiund can discuss project-specific sources and reference preparation; source availability, permitted use and commercial terms are established for the actual collection.

Primary task
Who spoke when, including simultaneous speech
Reference layer
Time intervals, anonymous speaker labels and evaluation regions
Scoring controls
Collar, overlap policy, mapping and aggregation
Critical split
Source-session and relevant speaker groups
Price status
Separate source access from interval annotation and adjudication

Your reading path

Start with the right questions.

  1. 01Diarization is not the same as transcription
  2. 02What real requests and developer documentation show
  3. 03Useful benchmark families

From principle to practice

A schema for reproducible speaker activity

ArtifactSuggested contentImportant distinction
Open the worked example

Diarization is not the same as transcription

Diarization assigns speaking activity to consistent speaker labels over time. Transcription identifies words. A system can get the words right but attribute them to the wrong person, or identify the speakers while missing a short utterance. A useful dataset treats those targets as related but distinct layers.

Overlap makes the distinction especially important. When two people speak simultaneously, a single active-speaker label cannot represent the full reference. Decide whether the task allows multiple labels at one time, whether backchannels count as speech and how uncertain activity is marked. The annotation policy should be chosen before a large collection is labeled.

What real requests and developer documentation show

A public diarization-data request specifically asks about overlapping speakers and noise, while another accented-speaker question seeks two-person recordings with different accents. These are qualitative needs, not verified purchases or estimates of market demand.

pyannote’s original project explicitly includes overlap detection, speaker change and embedding components. Its metric reference explains why evaluation needs a collar policy and can restrict scoring to annotated regions. Those primary technical sources support a detailed task specification without implying a Fiund model has been tested or certified.

Useful benchmark families

AMI is a 100-hour meeting resource with synchronized recording devices and annotations. It is useful for multi-person interaction but differs from telephone calls and edited podcasts. MagicData-RAMC offers Mandarin conversational material under noncommercial/no-derivatives terms. Source domain and access conditions belong beside any comparison.

AppTek’s call-center evaluation corpus supplies role and speaker metadata for commissioned service conversations. It is not automatically an overlap-complete diarization reference simply because speaker fields exist. Inspect the released annotation layer and intended task. No benchmark’s total duration guarantees the amount of manually reviewed overlap available for a new project.

Define the annotation target precisely

Speaker labels can be local to a recording or linked across a collection. Most within-recording tasks do not require personal identity. Use anonymous stable labels and include only identity relationships necessary and permitted for the experiment. A known job role such as agent is different from a speaker identity.

Decide whether laughter, whispered speech, nonlexical acknowledgments and distant voices are included. Some applications need all audible participants; others focus on a foreground conversation. Do not leave that decision to annotators file by file. A written policy with examples is more useful than an undefined requirement for perfect labels.

Overlap needs its own coverage measures

Total hours can conceal a collection with very little simultaneous speech. Request the overlap duration, number of overlap events, typical event length and speaker-count distribution where those figures can be established. Distinguish intentional collection targets from measured source properties.

A pilot should contain short interruptions, longer concurrent answers, laughter over speech and quiet backchannels if those are in scope. Clean alternation between speakers is a useful control but not a substitute for overlap. Preserve source context so a short sound can be interpreted rather than forcing every event into an isolated clip.

Reference timing and uncertainty

Human annotators cannot place every speech boundary at exact sample precision. Define the practical boundary convention and how uncertain regions are represented. A broad uncertain interval should not silently become nonspeech, because that changes both the reference and the score.

Use independent review or adjudication on difficult regions and preserve the decisions. If annotations begin from a model proposal, record that lineage and identify the human-reviewed subset. A recent automated export should not be represented as an independent gold standard without evidence of its preparation and review.

A schema for reproducible speaker activity

Package source recordings, interval annotations and the evaluation map together. The buyer should be able to tell which regions have a reference and which speaker labels can be compared within a file.

Compare the evidence
ArtifactSuggested contentImportant distinction
Audio manifestrecording_id, source group, timebase, versionOne source can have several mixes or encodings
Speaker tableanonymous label, local/global scope, role if allowedRole and personal identity are separate
Activity intervalsstart, end, speaker, event typeSeveral intervals can overlap in time
Evaluation regionsincluded and excluded spans, reasonUnannotated is not automatically silent
Review recordpolicy version, review status, adjudicationAutomatic proposals are not gold references
Scoring configurationcollar, overlap setting, mapping, aggregationTwo scores are comparable only under stated conditions

DER, JER and diagnostic measures

Diarization error rate commonly summarizes missed speech, false alarms and speaker confusion relative to the scored reference activity. Jaccard-based measures provide another view of speaker overlap. The exact implementation and mapping matter, so record the tool, version and configuration rather than comparing a bare percentage across papers.

The pyannote metric documentation supports collars and evaluation maps. A collar removes boundary regions from scoring; ignoring overlap changes the target again. Neither policy is inherently a complete description of application quality. Report what was scored and retain diagnostic counts so a change in one component does not disappear inside a single total.

Collars and overlap settings can change the result

Two systems evaluated on the same audio can appear different if one report excludes overlapping speech and another includes it. A boundary collar can also reduce the penalty for small timing disagreement. State these choices in the benchmark specification and in every result table.

For a hypothetical comparison, a buyer might request both an application-aligned primary condition and a stricter diagnostic condition. Those settings should be predeclared and interpreted separately. Do not choose the most favorable configuration after seeing which one improves the headline. A score is evidence about the stated protocol, not a universal property of a model.

Channels, mixtures and enrollment information

Channel-separated source audio can help annotation, but it can also leak information unavailable to the evaluated system. Decide whether the model receives a mono mixture, multiple channels, an exact speaker count or enrollment samples. A system with those hints answers a different question from one operating without them.

Keep reference-assistance artifacts separate from test inputs. If annotators use individual tracks to resolve overlap, document that method and ensure the submitted model sees only the allowed representation. All channels and derivatives of a source session should remain in the same split. Otherwise a model can encounter a closely related version during training.

Speaker counts and short turns

A diarizer can fail by merging similar voices, splitting one person into several labels or missing brief contributions. A collection should include the speaker-count and turn-length patterns relevant to deployment. Long single-speaker stretches alone do not establish performance on rapid conversational exchange.

Inspect short acknowledgments and returns after long pauses. Decide whether a speaker who appears only briefly belongs in the evaluated set and how unknown background voices are treated. These details affect both label consistency and interpretation of the result. They should not be left as undocumented exceptions in a preparation script.

Provider comparison and budget components

A source owner may supply appropriate recordings but no reliable interval labels. An annotation provider may label existing licensed audio but not grant rights to the recordings. A public benchmark may already have references, subject to its terms and domain limitations. Separate those roles when comparing proposals.

No public market average specifically for overlap-reviewed diarization data was established here. Request a quote for source access, candidate preparation, interval annotation, independent review and adjudication. The proportion of difficult overlap and number of active speakers can change effort even when elapsed hours remain identical. A per-hour quote should explain which review depth it covers.

An explicit hypothetical pilot model

Suppose a team plans eight accepted hours with several interaction structures. Using invented placeholders, it allocatesUSD 800 for scoped source access, USD 1,600 for first-pass interval labeling, USD 1,200 for independent overlap review andUSD 600 for adjudication and packaging. The modeled total isUSD 4,200. These assumptions are a planning illustration rather than supplier prices or a Fiund estimate.

Before scaling, measure the pilot’s actual annotation disagreements and repair effort. A collection with a similar duration but more brief turns or more simultaneous speakers can require different work. Keep accepted hours, overlap event counts, speaker groups and review status alongside the cost model so a headline rate does not obscure the deliverable.

Pilot acceptance and held-out design

Choose complete source recordings that contain the difficult events in the specification. Reconcile interval validity, label consistency, evaluation regions and the mapping to the selected audio version. Test the scoring pipeline on deliberately understandable examples before trusting aggregate outputs.

Group related sessions and speakers according to the experiment. A system intended for unseen people needs a different held-out boundary from one meant to track recurring participants. If speaker relationships are not available, document the limitation. Do not claim a speaker-disjoint benchmark simply because filenames were randomly assigned to partitions.

Reference availability by region

A recording can be valid while only some regions have a reviewed diarization reference. Preserve that evaluation map and report the scored duration separately from the whole file. Unreviewed regions should not silently become nonspeech or be counted as independently labeled hours.

Discuss diarization requirements with Fiund

Contact hello@fiund.com with the recording domains, languages, speaker counts, overlap patterns, allowed model inputs and scoring policy. Include whether the collection is for training or held-out evaluation and what independent reference review is required. Keep source acquisition and annotation expectations separate in the brief.

Fiund can discuss candidate source fit, a representative pilot and project-specific preparation. Availability, permissions, pricing and delivery are confirmed for selected material. No category page guarantees a model score, a universally complete reference or access to every named benchmark. Those claims require evidence from the actual collection and experiment.

Related sourcing decisions

Long-form ASR evaluation datasets: benchmarks, reference quality and budgets — Long-form ASR evaluation data: complete recordings, reviewed references, reproducible scoring, benchmark access, coverage and budget assumptions.

Noise and channel robustness speech datasets — Noise and channel robustness data for speech evaluation: real versus mixed audio, clean-reference limits, codec/device conditions, benchmark sources and pilot budgets.

Questions buyers ask

Is a speaker-labeled transcript enough?

Not necessarily. A diarization reference needs time intervals and consistent speaker labels, including a policy for simultaneous activity and unannotated regions.

Should overlap be excluded from scoring?

Choose the policy according to the task and state it explicitly. If deployment includes interruptions, an overlap-excluded headline alone can conceal an important limitation.

What is a collar?

It is a region around reference boundaries that a scoring protocol can exclude to account for timing uncertainty. The chosen value must accompany the reported result.

Can isolated tracks help create the reference?

Yes, if their use is documented. Keep them separate from model inputs when the evaluated system is supposed to receive only a mixture.

Is a big speech-sector deal a diarization-data price?

No. This research did not verify a largest category-specific transaction value. Funding, company acquisitions and licensed voice partnerships should not be relabeled as diarization corpus purchases.

Sources & research notes

Public documentation reviewed 2026-11-12. Provider descriptions and published terms are attributed below. Published facts are attributed to their sources. Worked files are labeled as illustrative or as an inspected editorial source; neither implies commercial dataset clearance.

  1. AMI corpus original projectgroups.inf.ed.ac.uk · Reviewed 2026-11-12

    Primary source reviewed November 12, 2026; check the current release and project-specific terms.

  2. MagicData-RAMC official OpenSLRopenslr.org · Reviewed 2026-11-12

    Primary source reviewed November 12, 2026; check the current release and project-specific terms.

  3. AppTek Call-Center Dialogues official datasethuggingface.co · Reviewed 2026-11-12

    Primary source reviewed November 12, 2026; check the current release and project-specific terms.

  4. Accented-speaker diarization requestwww.reddit.com · Reviewed 2026-11-12

    Original public request; qualitative signal, not a verified purchase or search-volume measure.

  5. Overlap/noise diarization requestwww.reddit.com · Reviewed 2026-11-12

    Original public request; qualitative signal, not a verified purchase or search-volume measure.

  6. pyannote.audio original projectgithub.com · Reviewed 2026-11-12

    Primary source reviewed November 12, 2026; check the current release and project-specific terms.

  7. pyannote.metrics primary metric referencepyannote.github.io · Reviewed 2026-11-12

    Primary source reviewed November 12, 2026; check the current release and project-specific terms.

Sourcing with Fiund

Turn the requirements into a sourcing brief.

Send the model task, required source characteristics, target volume, annotations, intended use and review criteria. Fiund can assess sourcing options and the preparation needed. Availability, source documentation, permitted uses and pricing are confirmed for the proposed project.

Discuss overlapping-speech diarization datasets: benchmarks and annotation design ↗

Build with AIPodcast

Start with the data
your model needs.

Share your use case. Get a focused next step.

Request a sample