Your reading path
Start with the right questions.
From principle to practice
A compact evaluation schema
Open the worked exampleWhy long recordings deserve their own test
A system can perform well on isolated utterances while failing on a complete conversation. Segmentation may remove silence, supply clean boundaries or discard difficult overlap before recognition begins. If the deployment receives long audio, the benchmark should expose the same decision process rather than supplying perfect boundaries that the product will not have.
Long-form does not have one universal duration cutoff. Define a duration distribution that reflects the application and retain complete source units where possible. A corpus consisting only of five-minute excerpts can miss failures that appear later in an hour-long recording. The test should also distinguish file-based transcription from streaming behavior, where future context is unavailable.
What public buyers and requests show
AssemblyAI’s benchmark documentation lists full talks and licensed accented-speech conditions among its evaluation sources. That is first-party evidence that real model developers evaluate more than one controlled read-speech domain. It is not evidence of a purchase from Fiund or an open procurement request.
A public request for longer multi-speaker audio reports that many available examples were too short for the poster’s transcription and diarization tests. Such requests motivate useful content and sourcing questions. They do not establish search-volume estimates or prove that every requester has a commercial budget.
Benchmark families to compare
LibriSpeech is a foundational read-speech reference, but its usual segmented evaluation does not reproduce every spontaneous long-form condition. Earnings-22 supplies 119 hours across 125 files of accented English earnings calls. The source domain and recording structure should remain visible when using it as an evaluation control.
SPGISpeech and GigaSpeech are substantial speech resources with important access restrictions. GigaSpeech’s audio gating terms limit use to noncommercial research even though its metadata displays an Apache tag. Do not choose a commercial evaluation source from a license badge alone, and do not call all source hours independently human-referenced without checking the release.
Define the unit, context and failure policy
The scoring unit should match the product input: an episode, a meeting, a call or another complete recording. Specify whether the model receives speaker counts, language hints, reference prompts or segment boundaries. These can be legitimate inputs, but they change the question the benchmark answers.
Failed outputs should remain visible. Decide how timeouts, empty transcripts, truncated results and undecodable files are recorded. Silently excluding difficult files can make a system look better by reducing the test set after the fact. Report both coverage and quality, with a stable denominator and an explanation of any predeclared exclusions.
Reference text must be independent and policy-driven
A transcript used for evaluation should have a documented creation and review process. If it starts from automatic recognition, identify how it was corrected and how uncertain regions were handled. A polished editorial transcript that removes repetitions is not the same target as literal spoken-word recognition.
Define conventions for fillers, partial words, punctuation, capitalization, numbers and named entities. Preserve the unnormalized reference as well as the scoring form. Otherwise a later normalization change can obscure whether a result improved because the model changed or because its errors were made equivalent to the reference. Keep audio and text versions linked.
Word error rate needs a reproducible policy
Word error rate compares substitutions, deletions and insertions with the number of reference words. Reporting the number without the tokenization and normalization policy leaves important ambiguity. NIST’s SCTK repository is a primary reference for established speech-scoring tools; record the exact tool/version and configuration used in an experiment.
The basic arithmetic is straightforward: if a hypothetical reference contains 1,000 words and alignment finds 40 substitutions, 30 deletions and 20 insertions, WER is 9%. This is an illustrative calculation, not a result from any provider. A long hallucinated repetition can contribute many insertions, which is one reason complete-recording tests should not be replaced with cherry-picked short clips.
Scores beyond one corpus-wide average
Keep per-recording outputs and report slices that matter to the deployment. Duration, source domain, channel condition, accent coverage, speaker count and overlap can expose very different error patterns. A single micro-averaged score can be dominated by a large easy subset.
Also consider the distribution of errors across recordings and whether a small number of catastrophic failures drives user harm. If confidence intervals are reported, preserve the grouping unit and method rather than treating individual correlated words as independent observations. These are recommended evaluation-design choices; no fixed slice target is a substitute for the actual product requirement.
Timestamps and content alignment are separate from text accuracy
A transcript can have good word recognition but poor timing. If the product supports search, captions or source citations, define an alignment metric or acceptance procedure separately. Inspect early and late points in a long recording to distinguish a fixed offset from accumulating drift.
An edit can invalidate timestamps even when the spoken words remain correct. Keep references to the selected audio version and document any removed regions. Do not count a model’s timestamps as independently verified reference boundaries unless a separate review supports that claim. Text scoring, timing quality and speaker attribution should remain distinguishable in the report.
Chunked, streaming and full-context systems
If a system internally chunks long audio, document overlap windows, context carryover and deduplication. Repeated boundary text and dropped words at joins can be hidden when each chunk is scored separately. Evaluate the assembled output against the complete source if that is what a user receives.
A streaming system also needs a timing definition: initial output, revised output and final output can differ. Decide whether latency and revision behavior are in scope and preserve the corresponding observations. Comparing a full-context offline model with a low-latency system can be useful, but the operating conditions should be clear rather than implied equal.
A compact evaluation schema
Package the benchmark so that another team can reproduce selection and scoring without private production access. Keep references, system hypotheses and metric results as separate artifacts tied by stable identifiers.
- Artifact
- Recording manifest
- Important fields
- recording_id, source_group, duration, audio_version
- Purpose
- Defines stable evaluation units
- Artifact
- Coverage labels
- Important fields
- language, domain, channel, speaker structure
- Purpose
- Supports predeclared slices
- Artifact
- Reference
- Important fields
- text, review status, normalization version
- Purpose
- Explains the target
- Artifact
- Timing/regions
- Important fields
- evaluation spans, uncertain/excluded regions
- Purpose
- Controls what is scored
- Artifact
- System run
- Important fields
- model/config, input hints, chunking, failures
- Purpose
- Records the actual test condition
- Artifact
- Score output
- Important fields
- alignment, per-file metrics, aggregation method
- Purpose
- Makes totals traceable
Source rights and independent evaluation material
A public research benchmark can be a useful control if its terms fit the project. A proprietary held-out source may be appropriate when the team needs a different domain or wants material with a more clearly documented availability history. New packaging alone does not prove the underlying recordings were absent from training.
Ask for source dates, prior distribution and relationships to other releases where available. Treat contamination risk as an evidence question rather than a guarantee. The model may have encountered public speech or a related transcript. Keep all edits, segments and alternate encodings from a source recording in the same partition.
Public pricing context and a benchmark budget
Defined.ai’s September 2024 announcement offered a 315-hour speech-data plan forEUR 50,000. That is a dated vendor program, not a current universal evaluation rate and not necessarily independently referenced long-form material. Current availability and scope require reconfirmation.
A hypothetical evaluation project can instead budget its components explicitly. For 15 accepted hours, suppose placeholder source access isUSD 1,200, independent reference preparation isUSD 1,800, adjudication isUSD 900 and packaging/scoring review isUSD 600. The planning total isUSD 4,500. These invented assumptions illustrate cost structure; they are not a quote, a measured market average or a claim that all 15 hours can presently be sourced.
Pilot acceptance before scaling
Choose a small pilot that spans the intended duration and difficulty range. Include a complete long recording, a domain-vocabulary example, quiet speech, interruptions and a failure-prone boundary case. Review reference policy disagreements before preparing a much larger collection.
Acceptance should verify that audio decodes, the version is stable, references follow the written policy, exclusions are explicit and the scoring procedure can be repeated. Keep an issue log tied to specific files and labels. A successful local scoring run does not by itself establish reference correctness, license suitability or the performance of an untested model.
Reference yield and review effort
Budget source duration and accepted reference duration separately. A recording may be usable for an unsupervised experiment but unsuitable for a high-confidence evaluation reference because important regions cannot be resolved. Establish how unclear words, overlapping speech and excluded content affect accepted volume before scaling. This prevents a large source-hour claim from becoming an unsupported reference-hour claim.
Also separate transcription correction from independent adjudication. One reviewer can create a consistent transcript without revealing where another reviewer would disagree. A small second-pass sample can identify policy problems before they propagate across the collection. Track those issues by source condition and annotation rule, then update the written policy and version affected references. The aim is a reproducible target, not a cosmetic label of perfect accuracy.
Discuss an evaluation brief with Fiund
Email hello@fiund.com with model task, input duration distribution, languages, domains, speaker structure, reference requirements and intended uses. State whether the collection is for training or held-out evaluation and which source relationships must remain separate. Include the scoring and acceptance policy if one already exists.
Fiund can discuss candidate source fit and a representative preparation plan. Availability, documentation, pricing and delivery are confirmed for the selected material. No page can guarantee a model result or that a public corpus is uncontaminated; those conclusions require evidence from the actual source and experiment.
Questions buyers ask
Sources & research notes
Public documentation reviewed 2026-11-12. Provider descriptions and published terms are attributed below. Published facts are attributed to their sources. Worked files are labeled as illustrative or as an inspected editorial source; neither implies commercial dataset clearance.
- Rev Speech Datasets official repositorygithub.com · Reviewed 2026-11-12
Primary source reviewed November 12, 2026; check the current release and project-specific terms.
- Earnings-22 original paperarxiv.org · Reviewed 2026-11-12
Primary source reviewed November 12, 2026; check the current release and project-specific terms.
- SPGISpeech maintainer dataset cardhuggingface.co · Reviewed 2026-11-12
Primary source reviewed November 12, 2026; check the current release and project-specific terms.
- SpeechColab GigaSpeech official cardhuggingface.co · Reviewed 2026-11-12
Primary source reviewed November 12, 2026; check the current release and project-specific terms.
- OpenSLR LibriSpeechwww.openslr.org · Reviewed 2026-11-12
Primary source reviewed November 12, 2026; check the current release and project-specific terms.
- AssemblyAI benchmark documentationwww.assemblyai.com · Reviewed 2026-11-12
Primary source reviewed November 12, 2026; check the current release and project-specific terms.
- Long recordings and diarization requestwww.reddit.com · Reviewed 2026-11-12
Original public request; qualitative signal, not a verified purchase or search-volume measure.
- Defined.ai Data Access Plan announcementdefined.ai · Reviewed 2026-11-12
Primary source reviewed November 12, 2026; check the current release and project-specific terms.
- NIST SCTK official repositorygithub.com · Reviewed 2026-11-12
Primary source reviewed November 12, 2026; check the current release and project-specific terms.
