Long-form ASR evaluation datasets: benchmarks, reference quality and budgets Long-form ASR evaluation data: complete recordings, reviewed references, reproducible scoring, benchmark access, coverage and budget assumptions. Long-form ASR evaluation datasets pair complete or extended recordings with reference transcripts and a reproducible scoring policy. They test behavior that short, pre-segmented clips can hide: repeated text, missed late sections, drift, chunk-boundary errors and loss of context. A useful benchmark defines source and speaker independence, transcript normalization and how partial or failed outputs are scored. Public corpora provide important controls, but a team may also need a held-out collection matching its deployment. Fiund can discuss source-specific evaluation material and preparation; this guide describes the test design rather than promising a particular benchmark score or inventory. Primary task Transcribe complete or extended recordings Reference requirement Versioned audio plus independently specified text Core score WER with explicit normalization and failure policy Important slices Duration, domain, accent, noise and speaker structure Budget basis Source rights, reference creation and independent review Why long recordings deserve their own test A system can perform well on isolated utterances while failing on a complete conversation. Segmentation may remove silence, supply clean boundaries or discard difficult overlap before recognition begins. If the deployment receives long audio, the benchmark should expose the same decision process rather than supplying perfect boundaries that the product will not have. Long-form does not have one universal duration cutoff. Define a duration distribution that reflects the application and retain complete source units where possible. A corpus consisting only of five-minute excerpts can miss failures that appear later in an hour-long recording. The test should also distinguish file-based transcription from streaming behavior, where future context is unavailable. What public buyers and requests show [AssemblyAI’s benchmark documentation](https://www.assemblyai.com/benchmarks) lists full talks and licensed accented-speech conditions among its evaluation sources. That is first-party evidence that real model developers evaluate more than one controlled read-speech domain. It is not evidence of a purchase from Fiund or an open procurement request. A [public request for longer multi-speaker audio](https://www.reddit.com/r/datasets/comments/1u2eq3a/free_english_audio_datasets_for_transcription/) reports that many available examples were too short for the poster’s transcription and diarization tests. Such requests motivate useful content and sourcing questions. They do not establish search-volume estimates or prove that every requester has a commercial budget. Benchmark families to compare [LibriSpeech](https://www.openslr.org/12) is a foundational read-speech reference, but its usual segmented evaluation does not reproduce every spontaneous long-form condition. [Earnings-22](https://arxiv.org/abs/2203.15591) supplies 119 hours across 125 files of accented English earnings calls. The source domain and recording structure should remain visible when using it as an evaluation control. [SPGISpeech](https://huggingface.co/datasets/kensho/spgispeech) and [GigaSpeech](https://huggingface.co/datasets/speechcolab/gigaspeech) are substantial speech resources with important access restrictions. GigaSpeech’s audio gating terms limit use to noncommercial research even though its metadata displays an Apache tag. Do not choose a commercial evaluation source from a license badge alone, and do not call all source hours independently human-referenced without checking the release. Define the unit, context and failure policy The scoring unit should match the product input: an episode, a meeting, a call or another complete recording. Specify whether the model receives speaker counts, language hints, reference prompts or segment boundaries. These can be legitimate inputs, but they change the question the benchmark answers. Failed outputs should remain visible. Decide how timeouts, empty transcripts, truncated results and undecodable files are recorded. Silently excluding difficult files can make a system look better by reducing the test set after the fact. Report both coverage and quality, with a stable denominator and an explanation of any predeclared exclusions. Reference text must be independent and policy-driven A transcript used for evaluation should have a documented creation and review process. If it starts from automatic recognition, identify how it was corrected and how uncertain regions were handled. A polished editorial transcript that removes repetitions is not the same target as literal spoken-word recognition. Define conventions for fillers, partial words, punctuation, capitalization, numbers and named entities. Preserve the unnormalized reference as well as the scoring form. Otherwise a later normalization change can obscure whether a result improved because the model changed or because its errors were made equivalent to the reference. Keep audio and text versions linked. Word error rate needs a reproducible policy Word error rate compares substitutions, deletions and insertions with the number of reference words. Reporting the number without the tokenization and normalization policy leaves important ambiguity. [NIST’s SCTK repository](https://github.com/usnistgov/SCTK) is a primary reference for established speech-scoring tools; record the exact tool/version and configuration used in an experiment. The basic arithmetic is straightforward: if a hypothetical reference contains 1,000 words and alignment finds 40 substitutions, 30 deletions and 20 insertions, WER is 9%. This is an illustrative calculation, not a result from any provider. A long hallucinated repetition can contribute many insertions, which is one reason complete-recording tests should not be replaced with cherry-picked short clips. Scores beyond one corpus-wide average Keep per-recording outputs and report slices that matter to the deployment. Duration, source domain, channel condition, accent coverage, speaker count and overlap can expose very different error patterns. A single micro-averaged score can be dominated by a large easy subset. Also consider the distribution of errors across recordings and whether a small number of catastrophic failures drives user harm. If confidence intervals are reported, preserve the grouping unit and method rather than treating individual correlated words as independent observations. These are recommended evaluation-design choices; no fixed slice target is a substitute for the actual product requirement. Timestamps and content alignment are separate from text accuracy A transcript can have good word recognition but poor timing. If the product supports search, captions or source citations, define an alignment metric or acceptance procedure separately. Inspect early and late points in a long recording to distinguish a fixed offset from accumulating drift. An edit can invalidate timestamps even when the spoken words remain correct. Keep references to the selected audio version and document any removed regions. Do not count a model’s timestamps as independently verified reference boundaries unless a separate review supports that claim. Text scoring, timing quality and speaker attribution should remain distinguishable in the report. Chunked, streaming and full-context systems If a system internally chunks long audio, document overlap windows, context carryover and deduplication. Repeated boundary text and dropped words at joins can be hidden when each chunk is scored separately. Evaluate the assembled output against the complete source if that is what a user receives. A streaming system also needs a timing definition: initial output, revised output and final output can differ. Decide whether latency and revision behavior are in scope and preserve the corresponding observations. Comparing a full-context offline model with a low-latency system can be useful, but the operating conditions should be clear rather than implied equal. A compact evaluation schema Package the benchmark so that another team can reproduce selection and scoring without private production access. Keep references, system hypotheses and metric results as separate artifacts tied by stable identifiers. Artifact Important fields Purpose Recording manifest recording_id, source_group, duration, audio_version Defines stable evaluation units Coverage labels language, domain, channel, speaker structure Supports predeclared slices Reference text, review status, normalization version Explains the target Timing/regions evaluation spans, uncertain/excluded regions Controls what is scored System run model/config, input hints, chunking, failures Records the actual test condition Score output alignment, per-file metrics, aggregation method Makes totals traceable Source rights and independent evaluation material A public research benchmark can be a useful control if its terms fit the project. A proprietary held-out source may be appropriate when the team needs a different domain or wants material with a more clearly documented availability history. New packaging alone does not prove the underlying recordings were absent from training. Ask for source dates, prior distribution and relationships to other releases where available. Treat contamination risk as an evidence question rather than a guarantee. The model may have encountered public speech or a related transcript. Keep all edits, segments and alternate encodings from a source recording in the same partition. Public pricing context and a benchmark budget [Defined.ai’s September 2024 announcement](https://defined.ai/press-room/defined-ai-launches-data-access-plan-dap-allowing-access-to-speech-datasets-at-affordable-prices) offered a 315-hour speech-data plan forEUR 50,000. That is a dated vendor program, not a current universal evaluation rate and not necessarily independently referenced long-form material. Current availability and scope require reconfirmation. A hypothetical evaluation project can instead budget its components explicitly. For 15 accepted hours, suppose placeholder source access isUSD 1,200, independent reference preparation isUSD 1,800, adjudication isUSD 900 and packaging/scoring review isUSD 600. The planning total isUSD 4,500. These invented assumptions illustrate cost structure; they are not a quote, a measured market average or a claim that all 15 hours can presently be sourced. Pilot acceptance before scaling Choose a small pilot that spans the intended duration and difficulty range. Include a complete long recording, a domain-vocabulary example, quiet speech, interruptions and a failure-prone boundary case. Review reference policy disagreements before preparing a much larger collection. Acceptance should verify that audio decodes, the version is stable, references follow the written policy, exclusions are explicit and the scoring procedure can be repeated. Keep an issue log tied to specific files and labels. A successful local scoring run does not by itself establish reference correctness, license suitability or the performance of an untested model. Reference yield and review effort Budget source duration and accepted reference duration separately. A recording may be usable for an unsupervised experiment but unsuitable for a high-confidence evaluation reference because important regions cannot be resolved. Establish how unclear words, overlapping speech and excluded content affect accepted volume before scaling. This prevents a large source-hour claim from becoming an unsupported reference-hour claim. Also separate transcription correction from independent adjudication. One reviewer can create a consistent transcript without revealing where another reviewer would disagree. A small second-pass sample can identify policy problems before they propagate across the collection. Track those issues by source condition and annotation rule, then update the written policy and version affected references. The aim is a reproducible target, not a cosmetic label of perfect accuracy. Discuss an evaluation brief with Fiund Email hello@fiund.com with model task, input duration distribution, languages, domains, speaker structure, reference requirements and intended uses. State whether the collection is for training or held-out evaluation and which source relationships must remain separate. Include the scoring and acceptance policy if one already exists. Fiund can discuss candidate source fit and a representative preparation plan. Availability, documentation, pricing and delivery are confirmed for the selected material. No page can guarantee a model result or that a public corpus is uncontaminated; those conclusions require evidence from the actual source and experiment. What duration counts as long-form? There is no universal threshold. Define the duration distribution and complete-source units that match the deployment, then report them explicitly. Can I evaluate with automatic transcripts? They can be useful candidate labels, but an evaluation reference needs a documented independent review policy. Otherwise the test may reward agreement with another recognizer’s errors. Is a low WER enough? Not for every product. Coverage failures, timestamp quality, speaker attribution, duration slices and catastrophic errors can matter separately. Can a new benchmark be called contamination-free? Not merely because it was packaged recently. Review the source’s prior availability and relationships to other releases; document uncertainty. What major ASR dataset deal price is established? No largest category-specific completed transaction was verified here. A dated vendor plan is useful pricing context but should not be relabeled as a transaction or market average. Research method: Fiund research desk; public sources, explicit worked-example labels and project-specific availability review. Editorial contact: hello@fiund.com. Sources Rev Speech Datasets official repository: https://github.com/revdotcom/speech-datasets Earnings-22 original paper: https://arxiv.org/abs/2203.15591 SPGISpeech maintainer dataset card: https://huggingface.co/datasets/kensho/spgispeech SpeechColab GigaSpeech official card: https://huggingface.co/datasets/speechcolab/gigaspeech OpenSLR LibriSpeech: https://www.openslr.org/12 AssemblyAI benchmark documentation: https://www.assemblyai.com/benchmarks Long recordings and diarization request: https://www.reddit.com/r/datasets/comments/1u2eq3a/free_english_audio_datasets_for_transcription/ Defined.ai Data Access Plan announcement: https://defined.ai/press-room/defined-ai-launches-data-access-plan-dap-allowing-access-to-speech-datasets-at-affordable-prices NIST SCTK official repository: https://github.com/usnistgov/SCTK