AI-103: Develop AI Apps and Agents on Azure

Building an AI solution goes beyond calling a model. The focus is on creating production-ready AI applications and agents with Microsoft Foundry that can use enterprise data, interact with tools, process different content types, and collaborate to complete real tasks.

Core capabilities to know

Exam focus: Understand not just what these capabilities do, but when and why you would use them together in an Azure AI solution.

Think in solution flows

A useful mental model for AI-103 is:

User → AI App/Agent → Model → Data + Tools → Action/Response

For more complex solutions:

User → Orchestrator → Agent A + Agent B + Agent C → Tools/Data → Result

An agent therefore isn't simply a chatbot. It combines a model's reasoning capabilities with instructions, knowledge, and tools so it can perform useful work.

Preparing effectively

The course assumes working knowledge of Python, REST APIs/SDKs, Azure fundamentals, and generative AI concepts. Hands-on practice is important: build applications in Microsoft Foundry, connect models to data, add tools to agents, experiment with multimodal inputs, and create multi-agent workflows.

Key takeaway: Think beyond prompts and models. AI-103 is about assembling the components required for an end-to-end AI solution:

Models → Grounding → Tools → Agents → Orchestration → Production

Related Snipps

• What Microsoft Foundry Provides for Building AI Agents — Core Foundry capabilities for tools, conversations, models, security, storage, and monitoring.
• Microsoft Foundry SDK vs. Foundry Tools SDKs — Understand platform-level and specialized SDKs when developing Foundry solutions.
• Building Production-Ready AI Agents with Microsoft Agent Framework Course — Explores agent architecture, tools, workflows, memory, monitoring, and deployment.

Before building an AI application or agent, understand how Microsoft Foundry organizes models, tools, knowledge, and development resources. These relationships form the foundation for everything that follows.

Foundry architecture at a glance

Think of the structure as:

Foundry Resource → Project → Models + Agents + Tools + Knowledge

The Foundry resource is the underlying Azure resource and infrastructure boundary. A project lives within that resource and organizes the models, agents, tools, and knowledge used by an AI solution. The resource is the foundation; the project is the development workspace.

Microsoft Foundry provides access to generative models alongside Foundry Tools for specialized AI capabilities such as Language, Speech, Translation, and Document Intelligence. These services complement models when an application needs capabilities such as speech recognition or structured information extraction.

Know which SDK fits

Key distinction: use the OpenAI SDK when targeting a model directly; move toward the Foundry SDK when working with the broader agentic platform, including tools and grounding.

For development, Visual Studio Code with the Microsoft AI Toolkit is the recommended combination presented in the course. In the Foundry portal, Discover is primarily for finding models, tools and templates, while Build is where deployed resources are configured and tested.

Remember the six Responsible AI principles

Fairness • Reliability & Safety • Privacy & Security • Inclusiveness • Transparency • Accountability

These are not an afterthought: they influence grounding, prompts, guardrails, UX, evaluation, and ongoing operation of the solution.

Memory model: Resource hosts → Project organizes → Model reasons → Knowledge grounds → Tools act → Responsible AI governs.

Related Snipps

• What Microsoft Foundry Provides for Building AI Agents — Explains models, tools, knowledge, security, monitoring, and managed agent capabilities.
• Building Production-Ready AI Agents with Microsoft — Covers agents, tools, orchestration, monitoring, and production-oriented architecture.
• Choosing the Right Platform for Building AI Agents — Places Foundry and Azure within the broader range of agent-development approaches.

Choosing a model is more than picking the most capable option. In Microsoft Foundry, the practical lifecycle is Select → Deploy → Evaluate: balance model capability, performance, cost, data-location requirements, and measured output quality.

1. Select the model

Use the Model Catalog and Leaderboard to narrow candidates by capabilities, supported languages, context window, fine-tuning support, and benchmarks.

Key trade-off: a larger model may deliver better results, while a smaller model can offer higher throughput and lower cost.

2. Choose the deployment

Know these deployment choices:

• Global → broadest capacity and potentially highest throughput

• Data Zone → processing stays within a geographic zone such as the EU or US

• Regional → maximum control over the processing region, but capacity is constrained to it

• Standard → usage/token-based; suitable for general workloads

• Provisioned → predictable, guaranteed throughput

• Batch → high-volume, non-interactive processing where latency is less important

• Developer → lightweight testing of fine-tuned models

Remember: Global Standard is the general-purpose choice highlighted for obtaining the largest available quota.

3. Evaluate before production

Model performance isn't just speed. Evaluate quality, relevance, fluency, and groundedness.

Manual evaluation: run representative prompts and edge cases against models side-by-side.

Automated evaluation: use a larger prompt dataset, expected behavior, and evaluators to systematically score responses and safety.

High-value distinctions to remember:
Throughput = speed/capacity of responses · Fluency = natural, linguistically correct output · Groundedness = whether responses align with known information

Mental model:
Requirements → Catalog/Benchmarks → Deployment → Test Dataset → Evaluators → Compare → Improve

A chat application becomes much easier to design once you understand three decisions: which endpoint to use, which API to call, and where conversation state is maintained.

Endpoint & API choices

Key concept: LLMs are inherently stateless. Chat history must therefore be supplied or referenced. With Chat Completions, your application maintains and resends the message array. With Responses, server-side context can be continued by passing the previous response identifier.

Core pattern

response = client.responses.create(
    model="my-deployment",
    instructions="You are a helpful assistant.",
    input=user_input,
    previous_response_id=last_response_id
)

print(response.output_text)
last_response_id = response.id

For authentication, prefer Microsoft Entra ID over embedded API keys. DefaultAzureCredential is useful because the same application code can obtain credentials across local development and Azure-hosted environments.

Also know the configuration controls: temperature influences response variability, while max tokens constrains output size.

For applications performing other I/O while waiting for model responses, use the asynchronous client + await to avoid blocking execution.

Remember: responses.create() generates a response; response.output_text retrieves its text; response.id can connect the next turn.

Generative AI models are powerful, but their trained knowledge is limited. Tools extend models beyond text generation, allowing them to access real-time information, take actions, ground responses in facts, extend functionality, and build intelligent workflows.

Know the Tools

Remember: current information → web_search · uploaded/private documents → file_search · calculations/code → code_interpreter · application-specific actions → function

Responses API

Tools are provided through the tools collection. The model can determine which available tool is appropriate for a request.

response = client.responses.create(
    model=model_name,
    input="Answer the user's request using the available tools.",
    tools=[
        {"type": "code_interpreter", "container": {"type": "auto"}},
        {"type": "web_search"},
        {"type": "file_search", "vector_store_ids": [vector_store.id]}
    ]
)

print(response.output_text)

Core flow: User → Responses API → Model → Tool → Result → Model → Response

For file_search, documents are stored in a vector store and prepared for semantic retrieval:

Files → Chunking → Embeddings → Vector Store → Retrieval → Model

This lets the model answer using relevant document content rather than relying only on its trained knowledge. Uploaded company policies or private documents → File Search + Vector Store.

Function Calling

Functions are different because the application executes the function, not the model. The model identifies the required function and returns a function-call request:

User → Model → Function Call → Application → Function → Result → Model → Response

The application executes the requested code and returns its result. This process can run in a loop when multiple tool calls are needed.

Key distinction: built-in tools extend the model with predefined capabilities; function calling connects the model to your own application logic and actions.

Related Snipps on Snippset

• What Microsoft Foundry Provides for Building AI Agents — Overview of Foundry capabilities for models, agents, tools, data, security, and monitoring.
• Microsoft Foundry SDK vs. Foundry Tools SDKs — Explains the difference between platform-level Foundry development and specialized AI service SDKs.
• Building Production-Ready AI Agents with Microsoft Agent Framework Course — Covers tools, memory, workflows, orchestration, monitoring, and production-ready AI agents.

Getting better results from a generative AI model does not automatically mean fine-tuning it. The key is identifying what is wrong with the output and choosing the least complex technique that solves it.

The decision you should remember

Start with prompt engineering. System instructions can define the model's role, constraints, tone and expected output. Examples and few-shot prompting can further improve consistency.

RAG = give the model knowledge

Retrieval-Augmented Generation is appropriate when the required information is too large, specialized or dynamic to include directly in the prompt:

Question → Vectorize → Retrieve relevant chunks → Question + context → LLM → Answer

The important distinction is that RAG does not retrain the model. Relevant external information is retrieved and added to the model's context at runtime.

Fine-tuning = change behavior

Fine-tuning trains a base model using many examples of desired input/output behavior. A supervised dataset commonly contains conversations such as:

{"messages":[
  {"role":"user","content":"Suggest a destination."},
  {"role":"assistant","content":"Absolutely! What kind of trip interests you?"}
]}

Microsoft Foundry supports different customization methods depending on the selected model, including supervised fine-tuning and Direct Preference Optimization (DPO). Model and region must support the chosen method.

Key distinction: Fine-tuning is comparatively expensive and time-consuming, produces a new fixed model that must be deployed, and must be repeated when training requirements change.

Remember

Prompt → instructions. RAG → knowledge. Fine-tuning → behavior.

When uncertain, try prompting first, use RAG for missing knowledge, and reserve fine-tuning for behavior that prompting cannot reliably achieve.

Generative AI is non-deterministic: the same system can produce unexpected or harmful outputs. Responsible AI therefore needs to be engineered into the complete application lifecycle—not added as a final check.

The core lifecycle

MAP → MEASURE → MITIGATE → DEPLOY & MONITOR ↻

Defense in depth

Think of safety as an AI request/response pipeline:

Input → UX Controls → System Prompt + Grounding → Guardrails → Model → Guardrails → Output

UX controls limit the attack surface, for example through input or conversation limits. System prompts and grounding constrain model behavior, but prompts alone are not a security boundary.

Microsoft Foundry guardrails add enforcement around the model and can intervene before input reaches the model and after output is generated. Controls can target:

Jailbreaks · Hate · Violence · Sexual content · Self-harm · Protected material · Groundedness · PII

Guardrails have configurable blocking thresholds, allowing stricter policies for higher-risk applications.

Model refusal ≠ Guardrail blocking

This distinction matters:

Model refusal: Request → Model → "I can't help with that."
The model received and processed the request.

Guardrail blocking: Request → Guardrail → BLOCKED ⛔ → Model
The unsafe request never reaches the model.

What to remember

There is no single safety control. Combine UX restrictions, system instructions, grounding, guardrails, appropriate model selection, evaluations, and production monitoring.

Treat responsible AI like security engineering: identify → test → defend → monitor → repeat.

AI agents add an orchestration layer above an LLM: model + instructions + tools + conversation state work together in an agentic loop to complete multi-step tasks rather than simply return a response.

Core Architecture

Application → Foundry Project → Agent → Model + Instructions + Tools

Microsoft Foundry Agent Service provides a managed runtime for conversation state, tool calling, and agent lifecycle. Agents can invoke built-in tools such as File Search for grounded retrieval and Code Interpreter for Python-based analysis, plus APIs and custom functions.

Build & Integrate

Typical flow:

Create project → Deploy model → Define agent → Configure instructions/tools → Test → Consume from application

The Foundry portal is useful for prototyping; SDK/code-based configuration improves repeatability, version control, and CI/CD.

from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential

project = AIProjectClient(
    endpoint=PROJECT_ENDPOINT,
    credential=DefaultAzureCredential()
)

openai = project.get_openai_client()

Key Distinctions to Remember

Project endpoint ≠ model endpoint. When working through Foundry, AIProjectClient connects to the Foundry project endpoint. The project client can provide an OpenAI-compatible client for Responses, Conversations, and related operations.

Conversation state can be server-side. Conversations are durable objects containing messages, tool calls, and tool outputs, allowing the same conversation to continue across requests.

Tools determine agent capabilities. File Search provides grounding/RAG; Code Interpreter executes analysis; custom functions/APIs allow agents to take actions.

Managed Agent Service vs. direct model API: the managed runtime handles infrastructure such as conversation management, tool execution/orchestration, and agent lifecycle, while application code operates at a higher abstraction.

Exam mental model: know what belongs to the project, agent, model, conversation, and tool layers—and which SDK/client or endpoint your application is actually targeting.

AI agents become much more powerful when they can act on external systems, not just generate answers. Custom tools let a Foundry agent use application logic, databases, APIs, calculations, and workflows.

The Core Tool-Calling Pattern

Prompt → Agent → function_call → App executes tool → Result → Agent → Answer

A custom function tool has a name, description, and parameters. The agent uses these definitions to determine when a function is needed and what arguments to provide.

# 1. Create the agent
agent = project_client.agents.create_version(...)

# 2. Ask the agent
response = openai_client.responses.create(
    conversation=conversation.id,
    input="What's the weather in Zurich?",
    extra_body={"agent": agent}
)

# 3. Check whether the agent wants to use a function
for item in response.output:
    if item.type == "function_call":

        # YOUR application executes the function
        result = call_function(item.name, item.arguments)

        # Return the result to the agent
        send_function_result(item.call_id, result)

Key concept: the LLM does not execute your local function. It returns a function_call containing the requested function and arguments. Your application dispatches and executes it, then returns the result so the agent can continue reasoning.

Choose the Right Tool

Remember

Agent = decides what to call → Application = executes it → Agent = uses the result

One prompt can trigger multiple function calls, allowing an agent to combine several operations before generating its final response.

Model Context Protocol (MCP) standardizes how AI agents discover and invoke external tools. An MCP server publishes its available tools; an MCP client discovers them and makes them usable by an agent.

Remote MCP — direct integration

If the MCP server is remotely accessible, it can be registered directly with the Foundry agent using MCPTool.

# Define the remote MCP server
mcp_tool = MCPTool(
    server_label="docs",
    server_url="https://.../mcp",
    require_approval="always"
)

# Give it to the agent
agent = project_client.agents.create_version(
    ...,
    tools=[mcp_tool]
)

The agent can discover and use tools from that server. If approval is required:

for item in response.output:
    if item.type == "mcp_approval_request":
        approval = McpApprovalResponse(
            approval_request_id=item.id,
            approve=True
        )

💡 Key point: Remote MCP can be registered directly with the agent. Approval can provide a control point before a requested MCP tool is executed.

Local MCP — your application is the bridge

A cloud-hosted agent cannot directly reach an MCP server running on your machine. Your application therefore connects to the server as the MCP client.

# MCP SERVER — expose tools
@mcp.tool()
def get_inventory(product):
    return ...

# MCP CLIENT — discover & call tools
session = ClientSession(...)

tools = await session.list_tools()
result = await session.call_tool(...)

# AGENT — expose tools as functions
agent_tool = FunctionTool(...)

💡 Key point: Local MCP requires your application to perform the MCP communication. The discovered tools are exposed to the agent as FunctionTools.

How to remember it

Ask one question: Can the agent reach the MCP server directly?

Remote MCP: Yes → register it with MCPTool.
Agent → MCPTool → Remote MCP Server

Local MCP: No → your application bridges the connection.
Agent → FunctionTool → Your App → Local MCP Server

⭐ In short: Remote = agent talks to MCP directly. Local = your application acts as the bridge.

An LLM cannot inherently access your private or newly created data. RAG (Retrieval-Augmented Generation) solves this by retrieving relevant information at runtime and supplying it as context for a grounded, citation-backed response.

Foundry IQ = managed knowledge for agents

Foundry IQ provides reusable knowledge bases backed by Azure AI Search. Multiple agents can use the same knowledge base, while agentic retrieval can select sources, execute searches, rank results, and return relevant content with citations.

Know the three source patterns:

Key trade-off: indexed data gives fast, optimized retrieval; direct/real-time access favors freshness and avoids maintaining another indexed copy.

Instructions control retrieval

Connecting knowledge isn't enough. Effective agent instructions define three critical behaviors:

1. When to retrieve → e.g. always consult the knowledge base for product questions.

2. How to cite → require attribution to retrieved sources.

3. What to do when retrieval fails → say the information wasn't found rather than inventing an answer.

# Mental model — not full SDK code

agent.instructions = """
For product questions:
1. Search the knowledge base.
2. Cite retrieved sources.
3. If nothing relevant is found, say so.
"""

response = agent.run("Which tents are weatherproof?")
# Agent → Foundry IQ → source → grounded answer + citations

Remember: LLM = reasoning/generation; Foundry IQ = managed knowledge layer; knowledge base = reusable collection of sources; instructions = rules governing when and how that knowledge is used.

Building an agent is only half the job. Publishing turns a tested Foundry agent into something users can access through Microsoft 365 Copilot and Teams.

The publish pipeline

Agent Version → Publish → Agent Endpoint → Azure Bot Service → M365 / Teams

When publishing to Microsoft 365/Teams, Foundry handles several integration steps:

• validates the selected agent version and metadata

• creates/configures an Azure Bot Service for channel routing

• generates and submits the Teams app manifest

• enables the Activity protocol for message exchange

• configures authorization based on the selected publishing scope

Azure Bot Service is the bridge between Microsoft 365/Teams and the Foundry agent.

Publishing scope = access model

With organization-wide publishing, the agent becomes discoverable to users in the tenant after approval in the Microsoft 365 admin center.

Identity & versions

A published agent has a dedicated Entra agent identity and a stable endpoint. Resources accessed using that identity need the appropriate RBAC permissions.

Also distinguish saving/versioning from publishing: agent versions can evolve independently from what users currently consume.

Remember the flow:
Version → Publish → Bot Service → Activity protocol → M365/Teams

And the key mapping: Shared → RBAC; Tenant → tenant access + admin approval.

Build Agent-Driven Workflows in Microsoft Foundry

When an AI solution becomes multi-step, don't turn one agent into a logic engine. Use a workflow to combine LLM reasoning with deterministic orchestration: agents decide what something means; workflow logic decides what happens next.

Core model: Workflow = Executors (nodes) + Edges

• Agent node → invokes an agent

• Variables → carry state/data between nodes (Local.*)

• If / Switch → deterministic branching

• For-Each → process collections without duplicating nodes

• Edges → route execution between nodes

• Structured output (JSON) → turns probabilistic agent output into predictable data for downstream logic

• Human-in-the-loop → pause, clarify, approve, or escalate

Typical patterns are Sequential, Human-in-the-loop, Group chat, and Fan-out/Fan-in.

Think in workflows

A support flow could be:

Tickets → For-Each → Triage Agent → {category, confidence} → If confidence → Route

The agent classifies the ticket and returns structured JSON. The workflow then evaluates confidence: low confidence requests clarification; Billing escalates to a human; other categories continue to a Resolution Agent.

Key distinction: use an agent for semantic reasoning; use workflow nodes/edges for loops, routing, state, and business rules.

Calling it from code

The workflow can remain server-side while application code simply starts a conversation and invokes it:

project = AIProjectClient(endpoint, DefaultAzureCredential())
openai = project.get_openai_client()

conversation = openai.conversations.create()

response = openai.responses.create(
    conversation=conversation.id,
    extra_body={
        "agent_reference": {
            "name": "support-workflow",
            "type": "agent_reference"
        }
    },
    input="Process the tickets"
)

print(response.output_text)

Remember: Agent = reasoning → Structured JSON = contract → Workflow = orchestration → Edges = routing → For-Each = iteration → Human-in-loop = controlled escalation.

Microsoft Agent Framework provides a code-first abstraction for building agents across models, providers, tools, conversations, and workflows. It brings together concepts from Semantic Kernel and AutoGen behind a more consistent agent programming model.

Core model to remember: Provider → Agent → Tools → Run. A provider connects the agent to an underlying model; the agent combines instructions + tools; and run() executes the interaction. Built-in capabilities include file search, web search, conversation management, and workflow orchestration.

from agent_framework import Agent, tool

@tool
def submit_claim(subject: str, body: str):
    print(subject, body)

agent = Agent(
    client=model_client,
    instructions="Create expense claims using the available tool.",
    tools=[submit_claim]
)

result = await agent.run("Submit my expenses")

The important part is what is missing: no manual function-call dispatch loop. The framework can recognize the model's tool request, invoke the registered function, return its result, and continue execution automatically.

Key concepts

• Agent abstraction → allows different model/chat providers behind a common interface.

• @tool → exposes Python functions as agent tools; metadata can be inferred instead of manually constructing tool schemas.

• Tool approval → controls whether execution requires approval.

• Agent thread → manages conversation state and stored messages.

• Authentication → prefer token-based authentication; local development may use Azure CLI credentials, while hosted applications typically require an appropriate workload identity such as managed identity.

Remember: the framework handles much of the plumbing between LLM → agent → tool → result, letting application code focus on agent behavior rather than dispatch infrastructure.

A single agent can handle one task well; multi-agent orchestration coordinates specialized agents to solve larger tasks together. The Microsoft Agent Framework provides orchestration patterns that can be implemented directly in application code.

Know which orchestration pattern fits which execution flow:

Key distinction: Sequential and concurrent workflows have largely predictable structures. Handoff, group chat, and especially magnetic orchestration introduce increasingly dynamic agent selection and execution.
A simple sequential workflow illustrates the core programming model:

# 1. Define specialized agents
summarizer = chat_client.as_agent(...)
classifier = chat_client.as_agent(...)
action     = chat_client.as_agent(...)

# 2. Define their execution order
workflow = SequentialBuilder(
    participants=[summarizer, classifier, action]
).build()

# 3. Run the orchestration
await workflow.run(user_input)

Here, the summarizer → classifier → action order is explicit, and each agent runs in turn.
Remember: first define the agents and their capabilities, then select an orchestration pattern matching the required execution and collaboration model.

Need to identify a document’s language, extract people and places, or remove personal data? Azure Language in Foundry Tools provides purpose-built NLP capabilities without requiring an LLM.

Core capabilities to know

Key decision: prefer these specialized tools when you need predictable, focused text analysis. Compared with general-purpose LLMs, they can also be more appropriate where cost and latency matter. An LLM is better suited when the task requires broader reasoning or flexible natural-language generation.

Python mental model

from azure.identity import DefaultAzureCredential
from azure.ai.textanalytics import TextAnalyticsClient

client = TextAnalyticsClient(
    endpoint=endpoint,
    credential=DefaultAzureCredential()
)

docs = ["Alex visited Seattle last week."]

language = client.detect_language(docs)
entities = client.recognize_entities(docs)
pii = client.recognize_pii_entities(docs)

print(language[0].primary_language.name)
print([(e.text, e.category) for e in entities[0].entities])
print(pii[0].redacted_text)

Remember: endpoint + credential → TextAnalyticsClient → analysis method → structured result. Authentication can use credentials such as Microsoft Entra ID; document operations can also be performed in batches.

Scenario clues: “extract people, places and dates” → NER. “Remove customer details before publishing” → PII detection/redaction.

An LLM can identify entities or PII itself—but an agent can instead delegate these tasks to a specialized Azure Language tool through MCP. This separates agent reasoning from deterministic NLP processing.

Architecture: Prompt → Agent → discover/select MCP tool → Azure Language → tool result → final response

What to know

Azure Language MCP Server exposes Azure Language capabilities as tools that an agent can dynamically discover and invoke. Core capabilities include PII detection, language detection, and Named Entity Recognition (NER); additional Language capabilities are also exposed through MCP.

The important distinction is:

• Agent/LLM: reasons about the request and chooses an appropriate tool.

• MCP: standardizes tool discovery and invocation.

• Azure Language: performs the specialized NLP operation.

Tool selection is not hard-coded. The MCP server advertises available tools and their descriptions; the agent matches the user's intent to those descriptions. Good agent instructions further guide when Azure Language should be used.

Minimal mental model

# Agent is already configured with Azure Language MCP
response = openai_client.responses.create(
    input="Find and redact PII in this text...",
    extra_body={"agent": {"name": "text-agent"}}
)

print(response.output_text)

Behind this simple call:

Agent
 └─ discovers MCP tools
     └─ selects PII tool
         └─ Azure Language analyzes text
             └─ result returns to Agent

Watch for approval: MCP tool calls can require user/application approval. Either handle the approval request in code or configure appropriate tools for automatic approval.

Remember: MCP exposes tools; the agent selects them; Azure Language executes the NLP task.

Voice-enabled generative AI is essentially a two-way conversion pipeline: speech → text lets an application understand spoken input, while text → speech turns generated responses back into audio. Azure AI Foundry provides specialized models for both inference tasks.

Know which model solves which problem:

A transcription model such as GPT-4o-mini-transcribe accepts audio and returns text. A TTS model such as GPT-4o-mini-tts performs the reverse and can also follow instructions affecting characteristics such as tone.

The implementation pattern is straightforward: deploy the appropriate model in Foundry → create an authenticated Azure OpenAI client → call the corresponding audio API → handle text or binary audio output. Streaming is useful for TTS because audio bytes can be consumed as they arrive rather than waiting for the complete response.

# Speech → Text
with open("speech.wav", "rb") as audio:
    text = client.audio.transcriptions.create(
        model="gpt-4o-mini-transcribe",
        file=audio
    )

# Text → Speech
with client.audio.speech.with_streaming_response.create(
    model="gpt-4o-mini-tts",
    voice="alloy",
    input="Hello from Azure AI"
) as audio:
    audio.stream_to_file("speech.mp3")

Remember the direction: Transcribe = audio in, text out. TTS = text in, audio out. The audio side is binary data, so applications must correctly read input files or stream/write generated audio.

Azure AI Speech provides specialized APIs for adding speech-to-text (STT) and text-to-speech (TTS) to applications. Compared with using a general-purpose generative model for speech, dedicated Speech services provide more predictable latency, cost, audio formats, voices, logging, and output—but they do not reason about the meaning or intent of the content.

Core SDK pattern

Both directions start with SpeechConfig, which connects the SDK to the Azure/Foundry Speech resource using an endpoint + key or Entra ID credential.

Speech → Text:
SpeechConfig + input AudioConfig → SpeechRecognizer → recognize_once_async() → text + result metadata

Text → Speech:
SpeechConfig + output AudioConfig → SpeechSynthesizer → speak_text_async() → audio + result metadata

import azure.cognitiveservices.speech as speechsdk

speech = speechsdk.SpeechConfig(
    subscription=KEY, endpoint=ENDPOINT)

# Speech → Text
audio = speechsdk.audio.AudioConfig(filename="input.wav")
recognizer = speechsdk.SpeechRecognizer(speech, audio)
text = recognizer.recognize_once_async().get().text

# Text → Speech
audio = speechsdk.audio.AudioOutputConfig(filename="output.wav")
synth = speechsdk.SpeechSynthesizer(speech, audio)
synth.speak_text_async("Hello Azure!").get()

Remember the responsibilities

SpeechConfig = service connection + speech settings
AudioConfig = where audio comes from or goes
SpeechRecognizer = audio → text
SpeechSynthesizer = text → audio

For finer TTS control, use SSML (Speech Synthesis Markup Language) instead of plain text. SSML can control voice, speaking style, pronunciation, pauses, pitch, rate, and how specific values are spoken.

💡 Key distinction: Change the voice through SpeechConfig; use SSML when you need detailed control over how speech is delivered.

Azure Speech MCP gives a Foundry agent speech-to-text (STT) and text-to-speech (TTS) as remotely callable tools. The key idea: the agent doesn't implement speech processing itself—it selects and invokes the Speech MCP tool when needed.

Architecture

User → Foundry Agent → Speech MCP Server → Azure Speech
↕
Blob Storage

Know the responsibilities:

• Agent: interprets the request and selects the appropriate MCP tool.

• Speech MCP: exposes STT and TTS operations.

• Azure Speech: performs transcription or speech synthesis.

• Blob Storage: persists audio input/output.

Required configuration

The Speech MCP tool needs access to:

• Foundry resource → provides the Azure AI capabilities
• Model with tool calling – required so the agent can invoke MCP tools.
• Azure Storage Account + Blob Container → persists input/output audio
• X-Blob-Container-Uri → container URI including a SAS token, granting the MCP server the required Blob permissions
• Azure Speech MCP Server – added from the tool catalog and connected to the agent.

Key distinction: the model reasons about when speech is needed; the MCP tool performs the specialized speech operation.

Client-side mental model

Once the Speech MCP tool is configured on the agent, the application mainly invokes that agent:

project = AIProjectClient(endpoint, DefaultAzureCredential())
openai = project.get_openai_client()

response = openai.responses.create(
    input="Generate speech saying: Hello Azure",
    extra_body={
        "agent_reference": {
            "name": "speech-agent",
            "type": "agent_reference"
        }
    }
)

print(response.output_text) 
# Example: "Speech generated: https://.../audio.wav"

Key distinction: the client does not directly call Speech MCP here. It calls the Foundry agent, and the agent decides when to invoke its configured speech tool.

Azure Voice Live provides a managed speech-to-speech layer for building low-latency voice agents. Instead of manually chaining speech-to-text → LLM → text-to-speech, Voice Live exposes a real-time, bidirectional session that can listen and respond with streaming audio.

How it works

Client ⇄ WebSocket ⇄ Voice Live ⇄ Foundry Agent / Model

1. Connect to Voice Live and authenticate.

2. Configure the session — model/agent, modalities, voice, audio format and turn detection.

3. Stream microphone audio into the session.

4. Handle server events and immediately play streamed response audio.

5. Use VAD (Voice Activity Detection) to detect when the user starts/stops speaking.

# Conceptual SDK flow
connection = await connect(endpoint, credential, agent)

await connection.session.update(
    modalities=["text", "audio"],
    input_audio_format="pcm16",
    turn_detection="server_vad"
)

async for event in connection:
    if event.type == "input_audio_buffer.speech_started":
        stop_playback()          # user interrupts / barge-in

    elif event.type == "response.audio.delta":
        play(event.delta)        # streamed agent audio

Key concepts to remember

• Full-duplex / real-time: audio flows bidirectionally through a persistent connection.

• Authentication: Microsoft Entra ID is recommended; Voice Live also supports API-key authentication.

• PCM16 vs G.711: PCM16 = uncompressed/high fidelity; G.711 = smaller, speech-oriented encoding.

• input_audio_buffer.speech_started is especially important: with server VAD it signals detected user speech and can be used to stop current playback when the user interrupts.

• Voice Live can additionally provide noise suppression, echo cancellation, turn detection, voices, avatars and function calling.

Mental model: Connect → Configure Session → Stream Audio → Handle Events → React to Interruptions.

Need multilingual text or real-time conversations? Azure provides purpose-built translation APIs for both text and speech. The key is knowing which SDK object performs which job.

Text Translation — know the difference

Translation changes language; transliteration changes script.

client = TextTranslationClient(endpoint, credential)

# English → French
result = client.translate(
    body=["Hello world"],
    to_language=["fr"]
)
# → "Bonjour le monde"

The source language can be auto-detected or supplied with from_language. translate() can also target multiple languages in one request.

Speech Translation — remember the pipeline

Audio → TranslationRecognizer → translated text → SpeechSynthesizer → Audio

• SpeechTranslationConfig → credentials + source/target languages
• AudioConfig → microphone/file input or audio output
• TranslationRecognizer → recognizes and translates incoming speech
• SpeechSynthesizer → converts translated text back to speech

config = SpeechTranslationConfig(endpoint, credential)
config.speech_recognition_language = "en-US"
config.add_target_language("fr")

recognizer = TranslationRecognizer(config, audio_in)
result = recognizer.recognize_once()

# result.translations["fr"] → translated text

Key decision

One target + streaming/live output → event-based synthesis
Multiple target languages → manual synthesis

Event-based synthesis supports one translation target; for multiple languages, iterate through the translated results and synthesize each separately.

Remember: Translate = language, Transliterate = script, TranslationRecognizer = speech → translated text, SpeechSynthesizer = text → speech.

Azure Content Understanding converts unstructured documents, images, audio, and video into structured, application-ready data. The central concept is the analyzer: a reusable configuration defining how content is processed and what information is returned.

Core architecture

Input → Analyzer → AI processing → Structured output

An analyzer combines:

• Base analyzer → modality-specific foundation, e.g. document, image, audio, or video.

• Field schema → defines the information and data types to return.

• Models/configuration → controls AI-powered processing.

• Output → content, structured fields, grounding and confidence information.

Key distinction: the schema defines WHAT to extract; the analyzer defines HOW that schema is applied repeatedly.

Prebuilt vs. custom

Use a prebuilt analyzer when the scenario matches an existing type, such as invoice or receipt. Build a custom analyzer when application-specific fields are required.

Invoice
 ├─ VendorName: string
 ├─ Total: number
 └─ LineItems: array<object>
      ├─ Description: string
      └─ Quantity: number

Fields can use different generation methods:

extract → retrieve information from the source
classify → select from predefined categories
generate → derive new information from the content

For example, quantities can be extracted from invoice rows while TotalQuantity can be generated from those values.

Multimodal processing

The important pattern stays the same across modalities: define schema → build analyzer → analyze content → consume structured results.

Studio, Foundry & applications

Microsoft Foundry is the broader AI development platform; Content Understanding provides multimodal analysis capabilities within that ecosystem. Content Understanding Studio is the specialized experience for designing, testing, and evaluating analyzers.

Production applications normally use the API/SDK directly:

client = ContentUnderstandingClient(endpoint, DefaultAzureCredential())

# Build reusable analyzer
client.begin_create_analyzer(
    analyzer_id="invoice-analyzer",
    analyzer_definition=schema
).result()

# Analyze new content
result = client.begin_analyze_binary(
    analyzer_id="invoice-analyzer",
    binary_input=document
).result()

Content can be supplied as binary data or a downloadable URL, and analyzer operations typically follow an asynchronous begin_* → poll → result pattern.

Remember

Prebuilt analyzer → ready-made scenario
Base analyzer → foundation for customization
Schema → fields, types and extraction behavior
Analyzer → reusable processing configuration
Grounding → where extracted information came from
Confidence (0–1) → application decision signal

Typical decision flow:
Choose modality/analyzer → customize schema if needed → build → analyze → inspect fields → validate using grounding/confidence.