Conversational AI data: connect speech to the right target Map conversational audio to ASR, speaker turns, dialogue state and grounded outcomes. Includes a correction-sequence example and a task-label decision table. Conversational AI data can mean recorded speech, transcripts, speaker turns, dialogue state, responses or verified outcomes. These targets are related, but one does not automatically supply another. Start with the behavior being trained or evaluated, preserve the context it needs, and define the evidence behind its labels. This guide explains the task distinctions through a short correction sequence; it does not assume that any conversation archive includes task-success annotations or is ready to train every kind of voice model. First choice Which behavior is the model expected to learn? Source unit A conversation with traceable turns and context Common mismatch A plausible response is not a verified outcome Keep separate Speech text, speaker attribution, state and actions A conversation is not one universal training example For speech recognition, the target is what was said under a stated transcription convention. For diarization, the labels describe who spoke when. For a task-oriented agent, the target might be the current request state, an allowed tool action or an evidence-supported response. A collection can support the first while lacking the others. This distinction affects source selection. A panel discussing travel contains travel vocabulary, but it is not automatically a dataset of successful bookings. A customer-service exchange can contain a request and an apparent resolution, yet the audio alone may not establish what happened in the underlying system. Public research illustrates different target designs The [SpokenWOZ paper](https://arxiv.org/abs/2305.13040) describes a task-oriented spoken-dialogue benchmark with cross-turn challenges. Its annotations are part of that dataset’s design. They should not be presumed to exist in a podcast archive just because both sources contain human conversation. The [Spotify Podcast Dataset paper](https://arxiv.org/abs/2004.04270) describes episodes with ASR transcripts and varied formats. This offers a useful contrast: an automatic text layer can make an archive searchable without establishing human-reviewed state or outcome labels. Read the target definitions before comparing dataset sizes. A worked correction sequence This is an authored illustration, not a real customer call or recorded dataset sample. A user says, “Book Friday—sorry, Monday.” The other speaker asks, “Monday morning?” The user replies, “After two.” The same three turns produce different correct targets depending on the task. Task Target for this illustration Evidence needed ASR Retain the audible Friday-to-Monday correction under the chosen convention The exact audio and a reviewed text reference Speaker attribution Keep the two participant identities consistent across the turns Supported speaker or channel evidence Dialogue state Requested day becomes Monday; time constraint becomes after 2:00, with AM/PM and date context unresolved Turn order, interpretation rules and any relevant context Response generation A suitable next response may clarify the actual date and available appointment The allowed task policy and available information Task success A booking is completed only if an appropriate external record supports it A tool result, booking record or other defined outcome evidence Why cleaning the transcript can change the task A readable summary such as “The user wants Monday afternoon” is not a verbatim ASR reference. It removes the initial request, the correction and the clarifying question. That may be a useful summary target, but it should have its own type and lineage instead of replacing the original text. The [Transformers ASR guide](https://huggingface.co/docs/transformers/en/tasks/asr) provides an audio-to-text training example. In a conversation project, define any additional label layers explicitly. Stable turn and segment IDs let an engineer connect the transcript, speaker annotation and state update without pretending they are the same artifact. Use label contracts to handle uncertainty In the example, “Monday” lacks a calendar date unless other context supplies it. A labeler should not invent a date from the current time or assume that a request has been accepted. Represent the missing field and state the evidence required to resolve it. Similar issues arise with names, quantities, negation and repairs. Write rules for ambiguous references, incomplete turns, overlapping words and unresolved outcomes. Include an adjudication route for disagreements rather than force every case into a clean success label. The difficult examples are often the most useful test of whether the annotation specification is actually actionable. Diagnose a pipeline in stages If a voice agent books the wrong day, inspect the stages separately. Did the recognizer miss “sorry”? Did the state updater fail to replace Friday? Did a tool call use the stale value? Did the system say “done” without a successful result? One end-to-end failure can have several causes. A useful evaluation packet retains inputs and outputs at the relevant stages, with access and handling agreed for the project. It reports which layer was reviewed and which was generated automatically. Otherwise an apparently improved response model can conceal a worse transcript or an unsupported outcome label. Select sources for the conversational evidence they retain Complete exchanges are often more useful than isolated utterances when the target depends on a correction, question or earlier constraint. Keep source boundaries and related editions visible. A collection of short clips may be appropriate for recognition but unable to answer a long-context dialogue question. Existing recordings, commissioned scenarios and generated conversations offer different evidence. Label the origin. A scripted scenario can cover a difficult policy branch but should not be described as a natural customer outcome. Evaluate on appropriate independent examples rather than treating a plausible script as empirical product validation. Specify the handoff before expanding the corpus A first brief should say which target layers are required and what will count as an accepted example. Fiund can discuss relevant sources and preparation against that brief; current availability and scope remain project-specific. Define the task, context window and selected source unit. List separate transcript, speaker, state, action and outcome fields as needed. Provide one worked positive example and one ambiguous or unanswerable case. State which labels are human-reviewed, automatic or inferred from another system. Group related sessions, speakers and derivatives to protect evaluation independence. Define acceptance by evidence and task behavior, not conversation hours alone. Is conversational audio enough to train a task-success model? Not necessarily. The recording may contain requests and responses without verified outcomes. Define the outcome evidence and how it is linked to the conversation. Can a cleaned transcript replace the original? It can serve a separate readability or summary purpose, but it may erase corrections, partial words or uncertainty needed by another task. Retain the relationship between the source and each text version. Do more conversation hours guarantee better agent behavior? No. The target labels, source fit, independent evaluation and failure coverage matter. A controlled pilot should establish whether the proposed material addresses the actual error. Research method: Fiund research desk; public sources, explicit worked-example labels and project-specific availability review. Editorial contact: hello@fiund.com. Sources SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agents: https://arxiv.org/abs/2305.13040 The Spotify Podcast Dataset: https://arxiv.org/abs/2004.04270 Transformers automatic speech recognition task guide: https://huggingface.co/docs/transformers/en/tasks/asr