Synthetic versus real speech: how to test the training mix Compare recorded and generated speech with a controlled augmentation experiment, real held-out evaluation, provenance records and realistic cost accounting. Synthetic speech can help target lexical gaps and create controlled training variations, but it is not automatically a substitute for the recordings a product must handle. The useful comparison is a controlled experiment on unseen real speech, with the source, generator, voice permissions and training budget documented. Generated audio can still involve contractual, identity or privacy questions. There is no universal best synthetic-to-real ratio, guaranteed accuracy improvement or reliable industry-wide cost multiplier. Decision Does the added material reduce a specific real-world error? Evaluation anchor Unseen real speakers and deployment conditions Lineage Generator, version, prompts and source relationships Distinguish Generated speech, voice conversion and acoustic augmentation Define what “synthetic” means in your experiment Text-to-speech generates an audio rendition from text. Voice conversion modifies aspects of an existing recording. Acoustic augmentation can alter an existing waveform or feature representation without creating a new independent speaker or original utterance. These operations should not be grouped into a single undifferentiated synthetic-hours number. If one recorded sentence produces twenty variants, the result is still related to one underlying example. Keep a parent ID. Otherwise a random split can place related versions in training and evaluation and make the new collection look more independent than it is. Count original and derived material separately. What published research can tell a buyer [Ogun, Colotte and Vincent’s augmentation study](https://arxiv.org/abs/2503.08954) evaluates TTS- and voice-conversion-based additions to particular ASR systems. It finds that the variation introduced and the experimental setup matter; naive mixing is not necessarily optimal. That supports testing a proposed intervention, not promising the paper’s improvement percentages for a commercial collection. [SpecAugment](https://research.google/pubs/specaugment-a-simple-augmentation-method-for-automatic-speech-recognition/) provides another example of targeted augmentation, operating on feature inputs. A useful sourcing decision asks which variation is missing: vocabulary, pronunciation, speaker characteristics, room acoustics, interaction timing or something else. A method addressing one gap may leave the others unchanged. An illustrative three-arm trial The following is an authored experimental design, not a reported Fiund result. Start from one fixed baseline checkpoint and keep evaluation speakers, devices and source sessions outside all training arms. Allocate the same training-compute budget so the comparison is not simply a longer run. Arm Training change Question it tests Recorded baseline Existing accepted real training set only What does the current source already support? Lexical augmentation Same real set plus generated rare product names and correction phrases Does targeted lexical exposure reduce the intended term errors? Acoustic augmentation Same real set with documented noise or feature variations Are failures better explained by capture conditions? Interpret results at the level of the intended gain For an address-entry recognizer, aggregate word error rate can conceal the important mistake: transcribing the wrong street number. Keep the broad recognition metric, but also review target-term accuracy, correction handling and hallucinated words in silence. Report the counts behind those slices so a small number of examples is visible. If generated names improve the target-term slice while errors increase for unseen real accents, the experiment reveals a trade-off. It does not justify the headline “synthetic wins.” Inspect the generated pronunciation and training balance, then repeat a bounded test. Keep the real evaluation fixed while making that decision. Preserve natural events that generation may miss Actual speech can contain breath, hesitation, partial words, overlapping speakers, self-correction, device artifacts and genuine uncertainty. A generated sentence may sound convincing while simplifying those events. Whether that matters depends on the model task: transcription of prepared narration and management of interruptions in a voice agent are different objectives. For interactive data, inspect complete exchanges rather than a playlist of isolated answers. Mark which turns were authored, generated, edited or recorded. A polished scripted dialogue is not evidence of natural turn-taking, and a transcript used as a synthesis prompt should not be presented as independent human ground truth. Generated audio still needs a source and permission review Check the actual generator and model terms, permitted output uses and any restrictions relevant to training another system. Also examine whether the material imitates an identifiable person or derives from recordings with limited permissions. “No human performed this exact output” does not answer those questions. The [Copyright Office’s digital-replica report](https://copyright.gov/ai/Copyright-and-Artificial-Intelligence-Part-1-Digital-Replicas-Report.pdf) addresses realistic depictions of people, including voices. It is a policy report rather than universal project clearance. Separately, the [EDPB AI opinion](https://www.edpb.europa.eu/system/files/documents/2024-12/edpb_opinion_202428_ai-models_en.pdf) cautions against automatic assumptions of model anonymity. Ask a qualified reviewer to assess the intended use and applicable rules. Compare accepted cost rather than generation speed A useful budget includes source access, generation, failed renders, quality review, metadata, rights review and the experiment itself. Measure how much accepted material remains after removing mispronunciations, unsupported speaker labels, unstable timing and duplicates. A fast generation pipeline may still produce little task-relevant coverage. Preserve separate cost lines for recorded speech and its generated derivatives. If one source permission or prompt changes, the lineage record should identify the affected outputs. Do not describe a compute-only estimate as a market price for a commercially usable training corpus. A release checklist for mixed-source speech The handoff should let another engineer reproduce the source distinction and evaluation boundary. Unknown fields should remain unknown rather than be populated with invented demographic or recording attributes. Tag each example as recorded, generated, converted or otherwise augmented. Record generator/model version, prompts or prompt-set IDs and relevant source parent IDs. Document text normalization, pronunciation choices, audio preparation and rejected-output reasons. Keep related originals and derivatives in the same split group. Report the real held-out trial, slice counts, training conditions and observed limitations. Confirm the project’s actual permitted uses before redistribution or expansion. Can synthetic speech replace all human recordings? That conclusion requires evidence for the actual task and deployment conditions. A useful test keeps unseen real speech as the evaluation anchor and checks interaction and acoustic conditions that generation may omit. Does generated speech come with a free reference transcript? The synthesis prompt supplies intended text, not proof of what was rendered accurately. Check pronunciation, omitted words, normalization and failed outputs before treating that text as an accepted reference. Is synthetic speech always cheaper? Not on an accepted, task-relevant basis. Compare generation, rejected material, review, documentation and experiments alongside source costs. A compute-only price omits much of the work. Research method: Fiund research desk; public sources, explicit worked-example labels and project-specific availability review. Editorial contact: hello@fiund.com. Sources An Exhaustive Evaluation of TTS- and VC-based Data Augmentation for ASR: https://arxiv.org/abs/2503.08954 SpecAugment: A Simple Augmentation Method for Automatic Speech Recognition: https://research.google/pubs/specaugment-a-simple-augmentation-method-for-automatic-speech-recognition/ Copyright and Artificial Intelligence, Part 1: Digital Replicas: https://copyright.gov/ai/Copyright-and-Artificial-Intelligence-Part-1-Digital-Replicas-Report.pdf Opinion 28/2024 on personal data in AI models: https://www.edpb.europa.eu/system/files/documents/2024-12/edpb_opinion_202428_ai-models_en.pdf