Image understanding

Add vision to your workflow. Send images alongside a text prompt and get grounded text back you can crop, measure, or render on.

How it works

Send one or more images alongside text in a Responses API or chat completion request. Muse Spark reads the visuals and returns text. Provide each image one of three ways:

Public URL — a fully qualified http/https image link.
Base64 data URL — the image bytes inline, no hosting required.
Uploaded file — a file_id from the Files API.

Use it for:

Describing scenes: generate detailed descriptions of what appears in an image.
Answering questions: respond to specific queries about objects, people, or actions in an image.
Extracting information: pull text, data, or key elements from charts, diagrams, or documents.
Analyzing content: identify objects, understand relationships, and categorize visual information.
Localizing objects: report where objects are as coordinates you can crop, measure, or draw overlays with. See perception grounding.

Image understanding with the Responses API

Send images as input_image content blocks inside a user message. The image_url field is a plain string — a public URL or a base64 data: URL — or set file_id to reference an image uploaded through the Files API.

python
import os
from openai import OpenAI
client = OpenAI(
    base_url="https://api.meta.ai/v1",
    api_key=os.environ["MODEL_API_KEY"],
)
response = client.responses.create(
    model="muse-spark-1.3",
    input=[
        {
            "type": "message",
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": "What is in this image?",
                },
                {
                    "type": "input_image",
                    "image_url": "https://upload.wikimedia.org/wikipedia/commons/thumb/c/c9/Avocado_Hass_-_single_and_halved.jpg/1280px-Avocado_Hass_-_single_and_halved.jpg",
                },
            ],
        },
    ],
)
print(response.model_dump_json(indent=2))

typescript
import OpenAI from 'openai';
const apiKey = process.env.MODEL_API_KEY;
if (!apiKey) {
  throw new Error('MODEL_API_KEY is not set');
}
const client = new OpenAI({
  baseURL: 'https://api.meta.ai/v1',
  apiKey,
});
const response = await client.responses.create({
  model: 'muse-spark-1.3',
  input: [
    {
      type: 'message',
      role: 'user',
      content: [
        {
          type: 'input_text',
          text: 'What is in this image?',
        },
        {
          type: 'input_image',
          image_url: 'https://upload.wikimedia.org/wikipedia/commons/thumb/c/c9/Avocado_Hass_-_single_and_halved.jpg/1280px-Avocado_Hass_-_single_and_halved.jpg',
        },
      ],
    },
  ],
});
console.log(JSON.stringify(response, null, 2));

python
import json
import os
import requests
response = requests.post(
    "https://api.meta.ai/v1/responses",
    headers={
        "Authorization": f"Bearer {os.environ['MODEL_API_KEY']}",
        "Content-Type": "application/json",
    },
    json={
        "model": "muse-spark-1.3",
        "input": [
            {
                "type": "message",
                "role": "user",
                "content": [
                    {
                        "type": "input_text",
                        "text": "What is in this image?",
                    },
                    {
                        "type": "input_image",
                        "image_url": "https://upload.wikimedia.org/wikipedia/commons/thumb/c/c9/Avocado_Hass_-_single_and_halved.jpg/1280px-Avocado_Hass_-_single_and_halved.jpg",