Speech intelligence / Research guide

Long-form ASR evaluation datasets: benchmarks, reference quality and budgets

Long-form ASR evaluation data: complete recordings, reviewed references, reproducible scoring, benchmark access, coverage and budget assumptions.

FIELD NOTESBY FIUND
01Signal02Transcript03Reference

Audio · language · evaluation

Cited references
9 cited sources
Inside this guide
1 worked example
Reading time
9 minutes
Reviewed
2026-11-12

Prepared by the Fiund research desk from the cited public documentation. Technical examples describe their evidence and limits. Research method & editorial contact · Suggest a correction

Reading tools
The takeaway

Long-form ASR evaluation datasets pair complete or extended recordings with reference transcripts and a reproducible scoring policy. They test behavior that short, pre-segmented clips can hide: repeated text, missed late sections, drift, chunk-boundary errors and loss of context. A useful benchmark defines source and speaker independence, transcript normalization and how partial or failed outputs are scored. Public corpora provide important controls, but a team may also need a held-out collection matching its deployment. Fiund can discuss source-specific evaluation material and preparation; this guide describes the test design rather than promising a particular benchmark score or inventory.

Primary task
Transcribe complete or extended recordings
Reference requirement
Versioned audio plus independently specified text
Core score
WER with explicit normalization and failure policy
Important slices
Duration, domain, accent, noise and speaker structure
Budget basis
Source rights, reference creation and independent review

Your reading path

Start with the right questions.

  1. 01Why long recordings deserve their own test
  2. 02What public buyers and requests show
  3. 03Benchmark families to compare

From principle to practice

A compact evaluation schema

ArtifactImportant fieldsPurpose
Open the worked example

Why long recordings deserve their own test

A system can perform well on isolated utterances while failing on a complete conversation. Segmentation may remove silence, supply clean boundaries or discard difficult overlap before recognition begins. If the deployment receives long audio, the benchmark should expose the same decision process rather than supplying perfect boundaries that the product will not have.

Long-form does not have one universal duration cutoff. Define a duration distribution that reflects the application and retain complete source units where possible. A corpus consisting only of five-minute excerpts can miss failures that appear later in an hour-long recording. The test should also distinguish file-based transcription from streaming behavior, where future context is unavailable.

What public buyers and requests show

AssemblyAI’s benchmark documentation lists full talks and licensed accented-speech conditions among its evaluation sources. That is first-party evidence that real model developers evaluate more than one controlled read-speech domain. It is not evidence of a purchase from Fiund or an open procurement request.

A public request for longer multi-speaker audio reports that many available examples were too short for the poster’s transcription and diarization tests. Such requests motivate useful content and sourcing questions. They do not establish search-volume estimates or prove that every requester has a commercial budget.

Benchmark families to compare

LibriSpeech is a foundational read-speech reference, but its usual segmented evaluation does not reproduce every spontaneous long-form condition. Earnings-22 supplies 119 hours across 125 files of accented English earnings calls. The source domain and recording structure should remain visible when using it as an evaluation control.

SPGISpeech and GigaSpeech are substantial speech resources with important access restrictions. GigaSpeech’s audio gating terms limit use to noncommercial research even though its metadata displays an Apache tag. Do not choose a commercial evaluation source from a license badge alone, and do not call all source hours independently human-referenced without checking the release.

Define the unit, context and failure policy

The scoring unit should match the product input: an episode, a meeting, a call or another complete recording. Specify whether the model receives speaker counts, language hints, reference prompts or segment boundaries. These can be legitimate inputs, but they change the question the benchmark answers.

Failed outputs should remain visible. Decide how timeouts, empty transcripts, truncated results and undecodable files are recorded. Silently excluding difficult files can make a system look better by reducing the test set after the fact. Report both coverage and quality, with a stable denominator and an explanation of any predeclared exclusions.

Reference text must be independent and policy-driven

A transcript used for evaluation should have a documented creation and review process. If it starts from automatic recognition, identify how it was corrected and how uncertain regions were handled. A polished editorial transcript that removes repetitions is not the same target as literal spoken-word recognition.

Define conventions for fillers, partial words, punctuation, capitalization, numbers and named entities. Preserve the unnormalized reference as well as the scoring form. Otherwise a later normalization change can obscure whether a result improved because the model changed or because its errors were made equivalent to the reference. Keep audio and text versions linked.

Word error rate needs a reproducible policy

Word error rate compares substitutions, deletions and insertions with the number of reference words. Reporting the number without the tokenization and normalization policy leaves important ambiguity. NIST’s SCTK repository is a primary reference for established speech-scoring tools; record the exact tool/version and configuration used in an experiment.

The basic arithmetic is straightforward: if a hypothetical reference contains 1,000 words and alignment finds 40 substitutions, 30 deletions and 20 insertions, WER is 9%. This is an illustrative calculation, not a result from any provider. A long hallucinated repetition can contribute many insertions, which is one reason complete-recording tests should not be replaced with cherry-picked short clips.

Scores beyond one corpus-wide average

Keep per-recording outputs and report slices that matter to the deployment. Duration, source domain, channel condition, accent coverage, speaker count and overlap can expose very different error patterns. A single micro-averaged score can be dominated by a large easy subset.

Also consider the distribution of errors across recordings and whether a small number of catastrophic failures drives user harm. If confidence intervals are reported, preserve the grouping unit and method rather than treating individual correlated words as independent observations. These are recommended evaluation-design choices; no fixed slice target is a substitute for the actual product requirement.

Timestamps and content alignment are separate from text accuracy

A transcript can have good word recognition but poor timing. If the product supports search, captions or source citations, define an alignment metric or acceptance procedure separately. Inspect early and late points in a long recording to distinguish a fixed offset from accumulating drift.

An edit can invalidate timestamps even when the spoken words remain correct. Keep references to the selected audio version and document any removed regions. Do not count a model’s timestamps as independently verified reference boundaries unless a separate review supports that claim. Text scoring, timing quality and speaker attribution should remain distinguishable in the report.

Chunked, streaming and full-context systems

If a system internally chunks long audio, document overlap windows, context carryover and deduplication. Repeated boundary text and dropped words at joins can be hidden when each chunk is scored separately. Evaluate the assembled output against the complete source if that is what a user receives.

A streaming system also needs a timing definition: initial output, revised output and final output can differ. Decide whether latency and revision behavior are in scope and preserve the corresponding observations. Comparing a full-context offline model with a low-latency system can be useful, but the operating conditions should be clear rather than implied equal.

A compact evaluation schema

Package the benchmark so that another team can reproduce selection and scoring without private production access. Keep references, system hypotheses and metric results as separate artifacts tied by stable identifiers.

Compare the evidence
ArtifactImportant fieldsPurpose
Recording manifestrecording_id, source_group, duration, audio_versionDefines stable evaluation units
Coverage labelslanguage, domain, channel, speaker structureSupports predeclared slices
Referencetext, review status, normalization versionExplains the target
Timing/regionsevaluation spans, uncertain/excluded regionsControls what is scored
System runmodel/config, input hints, chunking, failuresRecords the actual test condition
Score outputalignment, per-file metrics, aggregation methodMakes totals traceable

Source rights and independent evaluation material

A public research benchmark can be a useful control if its terms fit the project. A proprietary held-out source may be appropriate when the team needs a different domain or wants material with a more clearly documented availability history. New packaging alone does not prove the underlying recordings were absent from training.

Ask for source dates, prior distribution and relationships to other releases where available. Treat contamination risk as an evidence question rather than a guarantee. The model may have encountered public speech or a related transcript. Keep all edits, segments and alternate encodings from a source recording in the same partition.

Public pricing context and a benchmark budget

Defined.ai’s September 2024 announcement offered a 315-hour speech-data plan forEUR 50,000. That is a dated vendor program, not a current universal evaluation rate and not necessarily independently referenced long-form material. Current availability and scope require reconfirmation.

A hypothetical evaluation project can instead budget its components explicitly. For 15 accepted hours, suppose placeholder source access isUSD 1,200, independent reference preparation isUSD 1,800, adjudication isUSD 900 and packaging/scoring review isUSD 600. The planning total isUSD 4,500. These invented assumptions illustrate cost structure; they are not a quote, a measured market average or a claim that all 15 hours can presently be sourced.

Pilot acceptance before scaling

Choose a small pilot that spans the intended duration and difficulty range. Include a complete long recording, a domain-vocabulary example, quiet speech, interruptions and a failure-prone boundary case. Review reference policy disagreements before preparing a much larger collection.

Acceptance should verify that audio decodes, the version is stable, references follow the written policy, exclusions are explicit and the scoring procedure can be repeated. Keep an issue log tied to specific files and labels. A successful local scoring run does not by itself establish reference correctness, license suitability or the performance of an untested model.

Reference yield and review effort

Budget source duration and accepted reference duration separately. A recording may be usable for an unsupervised experiment but unsuitable for a high-confidence evaluation reference because important regions cannot be resolved. Establish how unclear words, overlapping speech and excluded content affect accepted volume before scaling. This prevents a large source-hour claim from becoming an unsupported reference-hour claim.

Also separate transcription correction from independent adjudication. One reviewer can create a consistent transcript without revealing where another reviewer would disagree. A small second-pass sample can identify policy problems before they propagate across the collection. Track those issues by source condition and annotation rule, then update the written policy and version affected references. The aim is a reproducible target, not a cosmetic label of perfect accuracy.

Discuss an evaluation brief with Fiund

Email hello@fiund.com with model task, input duration distribution, languages, domains, speaker structure, reference requirements and intended uses. State whether the collection is for training or held-out evaluation and which source relationships must remain separate. Include the scoring and acceptance policy if one already exists.

Fiund can discuss candidate source fit and a representative preparation plan. Availability, documentation, pricing and delivery are confirmed for the selected material. No page can guarantee a model result or that a public corpus is uncontaminated; those conclusions require evidence from the actual source and experiment.

Related sourcing decisions

Overlapping-speech diarization datasets: benchmarks and annotation design — Overlapping-speech diarization data: speaker intervals, benchmark choices, collars, reference review, source splits and transparent pilot costs.

Code-switching speech datasets for ASR evaluation — Code-switching ASR benchmarks: natural versus prompted speech, mixed-script references, scoring rules, source rights and pilot acceptance.

Questions buyers ask

What duration counts as long-form?

There is no universal threshold. Define the duration distribution and complete-source units that match the deployment, then report them explicitly.

Can I evaluate with automatic transcripts?

They can be useful candidate labels, but an evaluation reference needs a documented independent review policy. Otherwise the test may reward agreement with another recognizer’s errors.

Is a low WER enough?

Not for every product. Coverage failures, timestamp quality, speaker attribution, duration slices and catastrophic errors can matter separately.

Can a new benchmark be called contamination-free?

Not merely because it was packaged recently. Review the source’s prior availability and relationships to other releases; document uncertainty.

What major ASR dataset deal price is established?

No largest category-specific completed transaction was verified here. A dated vendor plan is useful pricing context but should not be relabeled as a transaction or market average.

Sources & research notes

Public documentation reviewed 2026-11-12. Provider descriptions and published terms are attributed below. Published facts are attributed to their sources. Worked files are labeled as illustrative or as an inspected editorial source; neither implies commercial dataset clearance.

  1. Rev Speech Datasets official repositorygithub.com · Reviewed 2026-11-12

    Primary source reviewed November 12, 2026; check the current release and project-specific terms.

  2. Earnings-22 original paperarxiv.org · Reviewed 2026-11-12

    Primary source reviewed November 12, 2026; check the current release and project-specific terms.

  3. SPGISpeech maintainer dataset cardhuggingface.co · Reviewed 2026-11-12

    Primary source reviewed November 12, 2026; check the current release and project-specific terms.

  4. SpeechColab GigaSpeech official cardhuggingface.co · Reviewed 2026-11-12

    Primary source reviewed November 12, 2026; check the current release and project-specific terms.

  5. OpenSLR LibriSpeechwww.openslr.org · Reviewed 2026-11-12

    Primary source reviewed November 12, 2026; check the current release and project-specific terms.

  6. AssemblyAI benchmark documentationwww.assemblyai.com · Reviewed 2026-11-12

    Primary source reviewed November 12, 2026; check the current release and project-specific terms.

  7. Long recordings and diarization requestwww.reddit.com · Reviewed 2026-11-12

    Original public request; qualitative signal, not a verified purchase or search-volume measure.

  8. Defined.ai Data Access Plan announcementdefined.ai · Reviewed 2026-11-12

    Primary source reviewed November 12, 2026; check the current release and project-specific terms.

  9. NIST SCTK official repositorygithub.com · Reviewed 2026-11-12

    Primary source reviewed November 12, 2026; check the current release and project-specific terms.

Sourcing with Fiund

Turn the requirements into a sourcing brief.

Send the model task, required source characteristics, target volume, annotations, intended use and review criteria. Fiund can assess sourcing options and the preparation needed. Availability, source documentation, permitted uses and pricing are confirmed for the proposed project.

Discuss long-form asr evaluation datasets: benchmarks, reference quality and budgets ↗

Build with AIPodcast

Start with the data
your model needs.

Share your use case. Get a focused next step.

Request a sample