Develop a speech agent with the Azure Speech MCP server | AI-103 | Episode 19

Azure Speech MCP gives a Foundry agent speech-to-text (STT) and text-to-speech (TTS) as remotely callable tools. The key idea: the agent doesn't implement speech processing itself—it selects and invokes the Speech MCP tool when needed.

Architecture

User → Foundry Agent → Speech MCP Server → Azure Speech
↕
Blob Storage

Know the responsibilities:

• Agent: interprets the request and selects the appropriate MCP tool.

• Speech MCP: exposes STT and TTS operations.

• Azure Speech: performs transcription or speech synthesis.

• Blob Storage: persists audio input/output.

Required configuration

The Speech MCP tool needs access to:

• Foundry resource → provides the Azure AI capabilities
• Model with tool calling – required so the agent can invoke MCP tools.
• Azure Storage Account + Blob Container → persists input/output audio
• X-Blob-Container-Uri → container URI including a SAS token, granting the MCP server the required Blob permissions
• Azure Speech MCP Server – added from the tool catalog and connected to the agent.

Key distinction: the model reasons about when speech is needed; the MCP tool performs the specialized speech operation.

Client-side mental model

Once the Speech MCP tool is configured on the agent, the application mainly invokes that agent:

project = AIProjectClient(endpoint, DefaultAzureCredential())
openai = project.get_openai_client()

response = openai.responses.create(
    input="Generate speech saying: Hello Azure",
    extra_body={
        "agent_reference": {
            "name": "speech-agent",
            "type": "agent_reference"
        }
    }
)

print(response.output_text) 
# Example: "Speech generated: https://.../audio.wav"

Key distinction: the client does not directly call Speech MCP here. It calls the Foundry agent, and the agent decides when to invoke its configured speech tool.