Full-duplex speech datasets: turn timing, overlap and interaction evidence

Specify an interaction problem, not merely conversational audio

A full-duplex model may need to acknowledge a speaker without taking the floor, stop when interrupted or answer promptly after a question ends. These behaviors depend on more than a transcript with speaker names. A transcript can omit pauses, shorten repetitions and hide overlapping utterances, all of which can matter to interaction timing.

Write separate requirements for recognition, speech generation and interaction. A corpus suitable for acoustic transcription may lack source-isolated tracks or sufficiently precise turn labels. A corpus useful for turn timing may have limited subject diversity. Neither property guarantees that the collection suits a voice agent's operational domain.

This page focuses on speech interaction signals. For telephone-agent actions, tool calls, escalation and task outcomes, use the Call Datasets site rather than treating task success as an acoustic label. A natural conversation can be excellent timing material without representing a resolved support case or a completed transaction.

Natural sessions, meeting corpora and generated evaluations

Kyutai's Moshi repository identifies its framework as full-duplex spoken dialogue. It provides a primary reference for why simultaneous listening and speaking need explicit representations. Its model code, model weights and any underlying training-source terms should be reviewed as separate artifacts; a repository license does not automatically license every source recording used in research.

The AMI Meeting Corpus offers another research reference for synchronized multi-participant recordings and annotations. Meeting conversation differs from two-person remote dialogue in microphone arrangement, social setting and turn structure. Use the differences to define a pilot, rather than treating all conversational hours as equivalent.

Sierra Research's voice interaction metrics documentation distinguishes response latency, turn-yield latency and other interaction measurements in a generated benchmark setting. Those measurements are useful definitions to examine. The benchmark's simulated timing assumptions are not universal acceptance thresholds for natural human recordings. Record which evidence comes from a real session and which comes from a generated evaluation.

Inspect channels and the common clock

Ask how the tracks were captured. One file per microphone can still contain substantial cross-talk, and a stereo room recording can contain both voices in both channels. Inspect overlap regions and quiet periods, not only isolated clean speech. Record whether a track is source capture, a mixdown or a model-separated derivative.

A shared timeline needs an origin, sample-rate convention and an offset policy. Independent local devices may start at different times or drift during a long session. A fixed initial shift does not prove alignment remains correct at the end. Retain the measured offsets and any time-varying correction applied by the preparation process.

For edited material, document whether pauses were shortened or turns moved. Concatenating separate monologues creates a sequence, but it does not recover authentic response timing. Retain an unedited reference when that is within the approved source scope. Report the duration before and after editing so a buyer can identify which representation supports interaction research.

Define event labels with context and uncertainty

A short “right” can be a backchannel, agreement, a request for the floor or part of a longer interrupted phrase. Labeling it solely by duration loses the interaction context. Specify a review window before and after the event, and allow an uncertain or unresolvable state when the source does not establish intent.

Define interruption operationally. For example, a project may require an attempted floor takeover plus an observable reaction, while another counts any overlapping onset. Neither taxonomy should be silently substituted for the other. Store the observed onset/offset separately from the interpretive label so a later taxonomy revision can reuse the timing evidence.

Treat laughter, breath and non-speech acknowledgments explicitly. Decide whether they create a turn, extend one or constitute a separate event. A clear codebook should include positive cases, counterexamples and a disagreement procedure. Keep the codebook version on each annotation batch; otherwise the same event label can change meaning across deliveries.

Worked timing example: acknowledgment is not response latency

The interaction-event JSONL contains a deliberately constructed session. Speaker A ends at 2.00 seconds; B has a short acknowledgment from 1.20 to 1.45 and begins a later response at 2.35. The response gap is 350 milliseconds under the stated onset rule. The earlier acknowledgment is a separate overlapping event, not proof that the question was answered before it ended.

These numbers are illustrative, not measurements from a recording. They demonstrate why an event type and evidence interval belong together. A real release should also identify the source track, clock basis, timing precision and review decision. If the timing comes from automatic alignment, record that method and its validation, rather than presenting every timestamp as manually verified.

Response latency alone is insufficient. A system could respond quickly by interrupting every speaker. Review premature responses, missed turn yields, appropriate acknowledgments and whether speech stopped when required. Use a task-specific set of metrics with clearly defined denominators rather than selecting the lowest latency number as the best interaction.

Build evaluation groups around sources and interactions

Keep adjacent segments of the same dialogue together when independence requires it. Repeated speakers, shared prompts and alternate exports can connect examples even when filenames differ. If the test is meant to assess unseen voices, group by verified participant identity where available. If identity cannot be resolved, state that limitation rather than claiming a speaker-disjoint evaluation.

Stratify the pilot by interaction condition: long turns, rapid exchanges, overlap, noisy capture, backchannels and ambiguous floor changes. Report what fraction of the sample is scorable for each label. A clean two-person prompt exercise may not test the behavior of an informal group conversation.

Use naturally occurring events and intentionally elicited scenarios transparently. An elicited interruption may be useful for a targeted stress test, but it has a different distribution from an incidental interruption in everyday conversation. Mark the collection method in the manifest and avoid reporting the combined count as an unqualified rate of natural interruption behavior.

Acceptance tests for a full-duplex pilot

Review a pilot on its original timeline. Verify that the audio decodes, the number of channels matches the manifest, track offsets are documented and annotated intervals fall within source duration. Then inspect source-specific problems: clipped onsets, noise gates suppressing acknowledgments, echo cancellation altering overlap and edits that remove gaps.

Score timing and event labels separately. An event may have correct boundaries and an uncertain interpretation, or the correct class with imprecise onset. Define the acceptable tolerance for the intended model and obtain reviewer agreement on ambiguous cases before scaling. A tolerance should be a project decision, not an invented industry standard.

Ask for an error log and rework policy. A record with missing capture provenance may be excluded from timing analysis while remaining useful for a content task. Preserve that distinction in acceptance reports. Do not discard useful recordings wholesale or imply they meet every task merely because one layer passed.

Pricing and a practical acquisition brief

Compare fees against unique shared-session hours, participant-track hours and accepted labeled events separately. Two isolated tracks from one hour of dialogue are two track-hours but only one session-hour. A per-hour quote must identify its denominator, whether silence is retained and whether label review is included.

Record the requested channel isolation, language, spontaneous versus prompted interaction, overlap conditions and target label set. Recruitment, capture setup, synchronization review and annotation may be distinct line items. A public catalog rate is not a prediction of a bespoke full-duplex project's cost or feasibility.

Send Fiund the target interaction behavior and a draft event schema. We can review source fit and the work needed for a proposal, with actual availability and permission scope confirmed separately. For speaker attributes see the demographic metadata guide; for purchasing units and vendor-specific list prices use the commercial comparison.

Questions buyers ask