Code-switching speech datasets for ASR evaluation

Define the switching behavior before selecting data

Code switching can occur across turns or inside one utterance. Borrowed terms, proper names and abbreviations can create ambiguous boundaries, so the benchmark needs an operational definition. A buyer asking for bilingual data should specify which behavior it wants rather than assuming every mixed vocabulary example qualifies.

A two-person conversation can also involve one monolingual speaker in each language. That is a useful interaction but a different recognition condition from a single person switching repeatedly. Preserve these patterns in the manifest. An aggregate language list cannot describe the switch structure or its difficulty.

Natural, prompted and synthetic source classes

Natural conversation captures switching that arose during the interaction. Prompted speech can control coverage but may reflect the collection instructions. Synthetic text or audio can add examples with a known construction method. These source classes should be compared explicitly instead of silently merged.

Synthetic assembly can introduce artifacts at boundaries, while prompted reading can change prosody and vocabulary. Those are experimental properties to document, not reasons to reject every such resource. A held-out set intended to represent actual user conversation should describe how its source behavior matches that target.

Public benchmarks worth comparing

CS-Dialogue’s original paper reports 104 hours of spontaneous Mandarin-English dialogue from 200 speakers. ASCEND’s official card exposes language, session, topic and speaker fields for another Mandarin-English resource, withCC BY-SA 4.0 metadata. Their current release terms and annotation details should be reviewed directly.

SwitchLingua’s paper describes a multilingual synthesis framework and an associated speech collection. It should not be summarized as wholly spontaneous dialogue. FLEURS and VoxPopuli provide broader multilingual controls, but multilingual breadth alone does not make them equivalent code-switching tests.

Evidence of a specific data need

A public original-bilingual-clip request illustrates the need for meaningful speech in both languages rather than a shared name alone. It is one qualitative request with its own conditions, not a representative survey or a large commercial procurement.

Potential users include multilingual recognizer developers, bilingual voice agents and speech-translation teams. The relevant benchmark differs by application. A recognizer may need literal words, a dialogue system may need turn context and a translation model may need a separate semantic target. No category page establishes a purchase commitment from a named company.

Reference scripts and normalization

A benchmark must decide whether words appear in native script, transliteration or another agreed representation. Keep the literal reference separate from any normalized scoring form. Proper names, borrowed terms and mixed abbreviations need examples in the policy so reviewers do not invent inconsistent rules.

Tokenization matters when one language is naturally scored at a different unit from another. Document the unit and normalization for each span and the way they combine into a mixed-language score. A result without that policy can be difficult to compare even when two teams use the same audio.

Language boundaries need a review convention

Not every token has an unambiguous language assignment. A widely borrowed term may be pronounced within the surrounding language, and a name can occur in several languages. Define how the benchmark handles those cases and preserve uncertain or shared-category labels where appropriate.

Span-level language labels should be tied to the selected audio/text version. If a transcript is corrected, update dependent language spans rather than leaving an old alignment. A single file-level mixed label can help filter candidates but does not describe where switches occur or whether the annotations are suitable for boundary-level analysis.

Coverage beyond a total-hour figure

Useful coverage measures can include language pair, switch direction, switch frequency, within-turn versus between-turn patterns, topic and speaker/session diversity. Count monolingual context separately from actual switching regions when it matters to the task. A long mixed-language file can contain only one brief switch.

Scoring should reveal, not erase, boundary difficulty

Report the exact scoring implementation and the treatment of scripts, punctuation, numbers and alternative spellings. A mixed error measure can be useful when languages use different natural units, but its construction should be explicit. Keep per-language and switching-region diagnostics where they answer the application question.

A semantically tolerant score can complement literal recognition scoring, but it should not silently replace it. A transcript that preserves approximate meaning may still fail a task requiring names or exact numbers. Choose the primary metric before inspecting results and retain enough outputs to explain substitutions, deletions and insertions.

A hypothetical metric example

Imagine a planning example with a reference containing 100 scored units after the agreed mixed-language tokenization. If an output hasfive substitutions, three deletions andtwo insertions, the illustrative error rate is 10%. Changing tokenization can change both the denominator and the alignment, so the policy must travel with the number.

This is arithmetic, not a measured result for any model. If the application depends on switch boundaries, report that slice separately rather than letting abundant monolingual context dominate. Do not compare a character-based rate from one protocol directly with a word-based rate from another as if they were identical units.

Audio context and source independence

Keep enough preceding and following speech to interpret the switch. Cutting exactly at a language boundary can create an easier artificial segment or remove important acoustic context. If the product receives complete conversations, evaluate complete source units or document why prepared excerpts are appropriate.

Group related segments, sessions and speakers according to the intended held-out question. Prompt templates and synthetic construction sources can also create relationships. Randomly splitting clips from one conversation is not evidence of generalization to new speakers or new switching behavior. Record unknown source links as limitations.

Provider and access comparisons

A public benchmark can supply a defined research reference under its current conditions. A custom collection provider can target specific language pairs and switching instructions. An archive source can supply naturally occurring behavior but may require more selection and review. The same bilingual label should not obscure those differences.

Ask whether the proposed source already exists, how switch behavior was established and what reference preparation is included. A research paper’s publication license does not itself settle all underlying recordings or participant permissions. Confirm the actual files, source terms and intended use before treating a resource as commercially available.

Pricing and a hypothetical pilot model

No reliable public average for commercially usable code-switching ASR evaluation data was established here. Source access, language-qualified transcription, boundary labeling and independent review need their own scope. A small public collection bounty should not be converted into an average per-hour benchmark price.

For a planning-only ten-hour pilot, invented placeholders might allocateUSD 1,000 for scoped source access, USD 1,600 for mixed-script reference preparation, USD 1,000 for independent language-boundary review andUSD 600 for packaging/scoring reconciliation: USD 4,200 total. These are transparent assumptions, not supplier prices or a Fiund quote.

Pilot acceptance should challenge the policy

Include natural switches, monolingual controls, a borrowed term, a proper name, an ambiguous boundary and an utterance with different script choices. Review the source class and full context before deciding how it should be labeled. A clean obvious bilingual sentence alone does not test the rubric.

Reconcile audio, text, language spans and scoring output. Document disagreements and revise examples before scaling. A pilot report should distinguish file validity, label quality, permission review and model results. Success in one layer does not establish the others, and no benchmark should be described as universally complete for every language pair.

Critical terms and release reproducibility

A mixed-language transcript can preserve broad meaning while getting a product name, number or short command wrong. If those errors matter to the application, define an additional critical-term review rather than assuming a semantically tolerant metric captures them. Keep the literal reference and the semantic interpretation separate so readers can understand what improved.

Package the release with a stable source manifest, annotation policy, normalization rules and split definition. Record which source and reference versions were scored and how uncertain language spans were handled. A later text correction should have a version identifier rather than silently changing the benchmark. This makes repeated experiments comparable without implying that every source relationship or possible spelling has been perfectly resolved.

Switch frequency and accepted volume

Report both complete source duration and the regions that demonstrate the required switching behavior. A long recording with one borrowed term should not silently satisfy a brief for sustained bilingual interaction. Preserve monolingual context when useful,but explain how acceptance counted it. This keeps a headline hour total tied to observed behavior rather than a broad file-level label.

Discuss a code-switching brief with Fiund

Email jaeden@fiund.com with language pairs, switching patterns, domains, recording conditions and the intended task. State whether natural dialogue is essential or prompted material can fit. Include reference scripts, normalization and held-out requirements if already defined.

Fiund can discuss candidate sources and a bounded preparation plan. Availability, permissions, pricing and delivery are confirmed for selected material. No largest publicly priced code-switching dataset transaction was verified here; research scale, funding rounds and broad speech partnerships should not be relabeled as a corpus sale.

Questions buyers ask