Speech instruction-tuning datasets: audio, instructions and target variants

Design speech instruction data with explicit task targets, transcript conventions, generation provenance, source grouping and reviewable evaluation splits.

A speech instruction-tuning dataset associates an audio source with a task instruction and a target response. The same source may support verbatim transcription, normalized transcription, translation or a grounded question, but each target has a different meaning and review requirement. Count both unique audio and derived task examples. Preserve the original recording relationship so generated targets do not masquerade as independent recordings or human-verified facts.

Core record

Audio segment + instruction + target + provenance

Volume accounting

Unique source duration and task rows separately

Target quality

Task-specific review, not a single confidence score

Download

Illustrative multi-task JSONL, with no audio

Choose the behavior before multiplying examples

A speech language model may be asked to repeat words, explain a spoken instruction, translate a sentence or describe nonverbal sound. These tasks can use the same waveform but cannot share an undefined target field. State the input modality, response language, permitted context and expected output format for every task. A training instruction should describe a behavior that can be evaluated, not a vague request to understand the clip.

Begin with a small task inventory. For each task, identify what evidence must be audible, which transformations are allowed and what an abstaining response looks like. If a question can only be answered using an episode description or outside knowledge, mark that source explicitly. An audio-only example should not quietly depend on a transcript or website paragraph that the model will not receive at inference time.

Derived records can be valuable, but they change the denominator. Ten prompts over one minute of audio are ten task examples over one unique minute, not ten minutes of additional acoustic coverage. Report both counts and retain the common source identifier when comparing a purchase or a model experiment.

What current speech research establishes—and what it does not

NVIDIA's [Granary-v2 project](https://research.nvidia.com/labs/nemotron/Granary-v2/) documents multiple transcript targets and instruction-oriented speech processing. It is a useful technical reference for separating target conventions and curation stages. Its published data and processing descriptions do not establish that an unrelated source has equivalent coverage, quality or permitted uses. Examine the particular release and target subset before using its statistics in a procurement comparison.

Microsoft's [SpeechT5 project](https://github.com/microsoft/SpeechT5) provides a historical example of speech-text tasks in a shared model framework. The repository is now archived; its research value does not make it an actively maintained production dependency. The important design lesson for a dataset brief is to name the input-output task and the associated preprocessing rather than label every row simply speech instruction data.

Use [dataset documentation](https://arxiv.org/abs/1803.09010) to record where supervision came from: source transcripts, manual annotations, automatic transcription, model-generated questions, translations or combinations of these. A generated label can be a candidate annotation. Its presence is not evidence that a human checked the audio, that it captures all disfluencies or that a downstream answer is grounded.

Keep verbatim, normalized and semantic targets separate

A verbatim transcript aims to preserve spoken material under an explicit convention. A normalized transcript may remove repetitions, change number formatting or improve punctuation. A summary intentionally compresses meaning. None is a universal gold transcript for all tasks. Write the convention alongside the field, and keep separate target versions rather than overwriting the earlier representation.

For code-switching, names and uncommon terms, preserve the difference between what was heard and how the project elects to write it. A normalization rule can be consistent while still unsuitable for another task. Define whether hesitation sounds, partial words, background speech and laughter belong in the target. If a convention suppresses these events, do not later use that target to evaluate their recognition.

The downloadable example uses the authored sentence “Well, I, I think we should start tomorrow.” A normalized target removes the hesitation and repetition. Both are constructed examples, with no associated recording. The example demonstrates schema behavior; it provides no evidence that any acoustic alignment or transcription accuracy has been measured.

Target

Preserves

Needs its own review

Verbatim transcript

Words and specified disfluencies

Audibility, omissions and convention

Normalized transcript

Agreed readable meaning

Meaning changes and forbidden edits

Grounded answer

A supported proposition

Evidence sufficiency and abstention

Translation

Meaning in another language

Bilingual adequacy and named entities

Use explicit evidence for questions and answers

A spoken question-answering record needs the question, target answer, source interval and an answerability decision. Evidence may be a local phrase, several turns or the entire recording. Distinguish an answer explicitly spoken from a reasonable inference. If an inference is allowed, describe the reasoning boundary so reviewers can apply it consistently.

Create genuinely unanswerable cases. In the example, the proposed starting day is available in the sentence, while the speaker's city is not. An unknown answer should not be manufactured from an assumed biography or a model's general knowledge. Decide whether the target is an empty value, a specific abstention phrase or a structured status before building the evaluation set.

Check questions for accidental disclosure. A prompt such as “Why did the speaker choose tomorrow?” presupposes both a decision and a reason that may not exist in the source. Reviewers should verify the question itself, not merely whether the model-generated answer sounds plausible. Keep rejected questions and their reasons in the quality report so the acceptance denominator is inspectable.

A minimal multi-task record and downloadable example

The [illustrative instruction-target JSONL](/downloads/speech-instruction-targets.jsonl) contains four rows sharing one fictional source. It explicitly marks the audio URI as absent and the targets as authored or derived from authored text. It is a planning artifact, not a corpus of recorded speech. Replace the example identifiers only after a real source and its intended use have been reviewed.

In a production schema, preserve the source hash, exact segment boundaries, instruction version, target version, creation method, reviewer decision and split group. The target-generation model version can matter as much as the audio-processing version. A later regeneration should create a new annotation version without pretending the original waveform changed.

Store private permission documents outside a public training manifest. Use an evidence reference that an authorized reviewer can resolve. The public record should explain the relationship and review status without including participant contact details, signatures or private contracts.

{
  "example_status": "constructed_not_recorded",
  "source_id": "demo-source-001",
  "segment_id": "demo-segment-001",
  "task_id": "transcribe_verbatim",
  "instruction": "Transcribe every spoken word and hesitation.",
  "target": "Well, I, I think we should start tomorrow.",
  "target_basis": "authored_example",
  "review_status": "illustrative_not_reviewed",
  "split_group": "demo-source-001",
  "audio_uri": null
}

Avoid leakage across task variants and source families

Assign train, development and test groups at the source-family level before generating alternate tasks. If a verbatim transcript enters training and a question over the same recording enters evaluation, the evaluation may reward memorized content. Randomly splitting the expanded task rows does not prevent this relationship.

Include neighboring segments, repeated intros, alternate edits, translations and recurring recording sessions in the grouping review. A common host may be acceptable across splits for one experiment and a confound for another. State the evaluation question and the independence constraint rather than declaring a split universally leakage-free.

Run a task-by-source accounting report after expansion. Compare unique sources and duration across splits, identify duplicate targets and inspect unusually similar questions. Keep a small, separately reviewed holdout that does not rely on the same automatic generation process as the bulk training annotations. That makes it easier to distinguish model quality from artifacts of the annotation pipeline.

Pilot acceptance should test each transformation

Choose a pilot that contains both routine examples and known difficult material: overlapping speech, partial words, names, code-switching, long pauses and ambiguous questions when those conditions are in scope. Review the original audio and target together. A clean-looking transcript alone cannot establish that it represents the recording correctly.

Score task validity separately from transcript accuracy. A record can have a good transcript and an unsupported question; it can have a supported answer but the wrong output language. Report failure reasons by task, source and creation method. Review disagreements need a documented resolution rule rather than an unexplained average score.

Set acceptance rules before batch production. Define how rework is delivered, whether failed rows are replaced and whether source hours remain counted after every derived target is rejected. Keep the accepted task inventory and source inventory reconciled at each release. This is more useful than a single aggregate claim of high-quality instruction data.

Request a quote with the expansion factor visible

A proposal can price source access, segmentation, task generation, human review and release preparation separately. Ask whether the charged unit is a unique recorded hour, a task row, an accepted question or a reviewed target minute. Do not compare these units as interchangeable. A larger expansion factor may increase annotation work while adding no new speakers or recording conditions.

For an illustrative scope, 100 unique hours with three candidate task variants would create up to 300 task-hours of target material before rejection. That arithmetic describes an authored planning scenario, not an available Fiund collection or a market price. The accepted output depends on evidence sufficiency and the task-specific rejection policy.

Send Fiund the intended model behavior, input/output schema, target languages, unique-source requirement, proposed task mix and review criteria. We can review the source and preparation requirements for a proposal. Use the [commercial speech comparison](/resources/commercial-speech-dataset-comparison/) to separate catalog access from new annotation work, and the [speaker metadata guide](/data-types/speaker-demographic-speech-data/) when speaker coverage is part of the experiment.

Are generated questions automatically grounded?

No. Check that the question is answerable from the allowed input and that the answer has sufficient evidence. Keep generation and review provenance separate.

Can one clip support several tasks?

Yes, when the targets are appropriate, but report unique audio and derived task rows separately and keep related variants in the same split group.

Does the download contain training audio?

No. It contains explicitly constructed records with absent audio URIs, intended for schema and procurement review.

Research method: Fiund research desk; public sources, explicit worked-example labels and project-specific availability review. Editorial contact: hello@fiund.com.

Sources
Granary-v2: multilingual speech curation and target variants: https://research.nvidia.com/labs/nemotron/Granary-v2/
SpeechT5 original implementation and task documentation: https://github.com/microsoft/SpeechT5
Datasheets for Datasets: https://arxiv.org/abs/1803.09010