What is it?
Audio generation is the use of AI to create new sound.
Inputs may include text, audio, music structure, a voice reference, or other conditions. Outputs may include speech, music, sound effects, or ambient audio.
07 · Generative & Multimodal AI · Topic 05
What is Audio Generation?
Audio generation uses AI models to create new sound, speech, music, or other audio.
Think of it like...
Think of it like giving a digital sound studio a description.
The studio uses your request and sound clues to produce audio matching it.
How it works
1 · ConditionGive an inputText, audio, structure, or another condition.
2 · EncodeRepresent the inputTurn the condition into information the model can use.
3 · GenerateCreate a representationThe model builds new audio information.
4 · DecodeMake a waveformA decoder turns it into playable audio.
5 · OutputReturn the resultSpeech, music, effects, or ambience.
Real-world examples
Text-to-speech
Input: written text
Output: spoken narration
Music generation
Input: style, mood, or prompt
Output: a song or instrumental track
Sound effects
Input: “rain on a window”
Output: matching sound or ambience
What it is NOT
Audio Generation ≠ Speech Recognition
Generation: creates new audio.
Recognition: turns speech into text or meaning.
Audio Generation ≠ TTS
Generation: includes many kinds of sound.
TTS: one speech-generation use case.
Audio Generation ≠ Music Generation Only
Generation: covers speech, effects, and ambience too.
Music generation: focuses on musical audio.
Related concepts
Generative AI → Audio Generation → Speech · Music · Effects
Audio generation is one modality of generative AI.
Explore next
Remember this
Audio generation creates new sound rather than only analyzing existing audio.
Video
Video
A visual explainer for Audio Generation is not available yet.