Learning by Patrik

Develop a vision-enabled generative AI application | AI-103 | Episode 22

A vision-enabled generative AI app combines visual context with natural-language instructions. The key is not a separate image-analysis pipeline: a multimodal model can receive text and images together and reason across both.

Architecture

Multimodal model → Text + Image → Visual reasoning → Text response

Know this	What it means
Model	Must support text + image input
Prompt	Combine instructions and image in the same message
Remote image	Pass an accessible image URL
Local image	Convert it to a Base64 data URI
API	Responses API supports multimodal input
Output	Read the generated text from `response.output_text`

Microsoft's current Responses API uses input_text and input_image; images can be supplied through a URL or Base64 data URI.

response = client.responses.create(
    model="multimodal-model",
    input=[{
        "role": "user",
        "content": [
            {"type": "input_text", "text": "Describe this image."},
            {"type": "input_image", "image_url": image_url}  # URL or data:image/...;base64,...
        ]
    }]
)

print(response.output_text)
# → "The image contains ..."

What to remember

Multimodal ≠ image generation. Here, the model understands visual input and reasons about it alongside text. Typical tasks include describing/captioning images, answering questions about visual content, and extracting information from images.

For image processing, detail can be low, high, or auto: lower detail reduces token usage and latency; higher detail captures finer visual information.

Mental model:
text + image → multimodal model → reason over both → response

Multimodal

Vision

Responses

Images

Azure