Develop a speech-capable generative AI application | AI-103 | Episode 17

Voice-enabled generative AI is essentially a two-way conversion pipeline: speech → text lets an application understand spoken input, while text → speech turns generated responses back into audio. Azure AI Foundry provides specialized models for both inference tasks.

Know which model solves which problem:

A transcription model such as GPT-4o-mini-transcribe accepts audio and returns text. A TTS model such as GPT-4o-mini-tts performs the reverse and can also follow instructions affecting characteristics such as tone.

The implementation pattern is straightforward: deploy the appropriate model in Foundry → create an authenticated Azure OpenAI client → call the corresponding audio API → handle text or binary audio output. Streaming is useful for TTS because audio bytes can be consumed as they arrive rather than waiting for the complete response.

# Speech → Text
with open("speech.wav", "rb") as audio:
    text = client.audio.transcriptions.create(
        model="gpt-4o-mini-transcribe",
        file=audio
    )

# Text → Speech
with client.audio.speech.with_streaming_response.create(
    model="gpt-4o-mini-tts",
    voice="alloy",
    input="Hello from Azure AI"
) as audio:
    audio.stream_to_file("speech.mp3")

Remember the direction: Transcribe = audio in, text out. TTS = text in, audio out. The audio side is binary data, so applications must correctly read input files or stream/write generated audio.