← Back to blog

Image to Prompt API — Turn Any Image into a Stable Diffusion Prompt

You found a reference image with lighting, pose, and color choices you want to study, but you do not know what prompt might describe it. The image-to-prompt API analyzes visible content with a vision model, then compiles the description into a prompt package for Stable Diffusion, ComfyUI, or AUTOMATIC1111 — positive prompt, negative prompt, recommended settings, and integration guidance.

This post walks through the real endpoint (POST /api/prompt-director/from-image) with working curl and Python examples, the MCP tool (image_to_prompt) for agents, and honest notes on pricing and limits. No fake SDKs — just the HTTP API and the MCP tools that are actually shipped.

What the endpoint returns

The endpoint is the vision half of the Prompt Director family. It runs a vision caption, cleans it into a dense visual description, then feeds that through the same prompt compiler used by /api/prompt-director. You get a package shaped like this:

{
  "success": true,
  "version": "1.0",
  "medium": "anime",
  "aspect_ratio": "portrait",
  "framing": "portrait",
  "model_family": "illustrious",
  "subject_count": 1,
  "strategy": "single subject, direct generation",
  "positive_prompt": "1girl, long black hair, ...",
  "negative_prompt": "lowres, bad anatomy, bad hands, ...",
  "recommended_settings": {
    "width": 832, "height": 1216,
    "steps": 28, "cfg": 5.0,
    "sampler": "Euler a", "scheduler": "normal", "seed": -1
  },
  "integrations": {
    "comfyui": { "positive_node": "CLIP Text Encode (Prompt)", "...": "..." },
    "stable_diffusion_webui": { "endpoint": "/sdapi/v1/txt2img", "...": "..." }
  },
  "warnings": [],
  "source": {
    "mode": "from-image",
    "image_description": "...",
    "request_used": "...",
    "vision_prompt": "...",
    "user_direction": "..."
  },
  "billing": { "describe_image": { "...": "..." } }
}

The source block gives you the raw vision description and cleaned request_used, so you can inspect how the service constructed the prompt. The integrations block provides ComfyUI and AUTOMATIC1111 handoff guidance; review settings before using them in a generation workflow.

Quick start: one curl call

Pass a public image URL as JSON — no upload needed for anything already hosted:

curl -X POST https://agentmediatools.com/api/prompt-director/from-image \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/reference.png",
    "medium": "anime",
    "direction": "same pose, soft light"
  }'

If you do not have a hosted URL, upload the file directly as multipart form data. The route accepts a file field named image:

curl -X POST https://agentmediatools.com/api/prompt-director/from-image \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -F "image=@./reference.png" \
  -F "medium=realistic" \
  -F "direction=golden hour, cinematic"

For uploaded files the limit is 8 MB; for anything larger, host the image and pass url instead. The API key goes in the standard Authorization: Bearer header — agent keys start with mt_ and are created in the toolbox.

Python example with requests

The same call in Python, using the URL path:

import requests

API = "https://agentmediatools.com/api/prompt-director/from-image"
HEADERS = {
    "Authorization": "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
}

payload = {
    "url": "https://example.com/reference.png",
    "medium": "anime",
    "aspect_ratio": "portrait",
    "framing": "portrait",
    "direction": "keep the pose, change the background to a night city",
}

r = requests.post(API, headers=HEADERS, json=payload)
r.raise_for_status()
pkg = r.json()

print(pkg["positive_prompt"])
print(pkg["negative_prompt"])
print(pkg["recommended_settings"])

And the multipart upload version for local files:

import requests

r = requests.post(
    "https://agentmediatools.com/api/prompt-director/from-image",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    files={"image": open("reference.png", "rb")},
    data={"medium": "realistic", "direction": "studio lighting"},
)
r.raise_for_status()
pkg = r.json()
print(pkg["positive_prompt"])

Request fields worth knowing

Field	Default	Notes
url / image_url	—	Public image URL (JSON body), or multipart image file (max 8 MB)
medium	anime	anime or realistic; changes the vision prompt and model family
direction	—	Your re-style intent, e.g. same pose, soft light. Also accepted via style or request
aspect_ratio	portrait	portrait, square, or landscape
framing	—	close-up, portrait, full-body, wide-shot
subject_count	1	1–6; the compiler warns when prompting alone is unreliable (3+)
target_model	auto	auto, illustrious, pony, sdxl, flux
interface	comfyui	comfyui, automatic1111, forge
avoid	—	Extra negatives to emphasize
advanced	false	Optional DeepSeek specialist pass for +1 credit
max_tokens	256	Vision caption length, clamped 64–512

The optional Advanced Optimizer pass

Set "advanced": true and, when you have an active API key, the service runs the compiled package through a DeepSeek specialist-and-critic optimizer (the same one used by /api/prompt-director/advanced). The advanced pass is billed only on success — if the optimizer fails, you get the standard package plus an advanced_error field and no extra credit is charged. Without an API key the standard package is returned with advanced: false and a note.

Using it from an agent via MCP

For agents, the same workflow is exposed as MCP tools. image_to_prompt wraps this exact endpoint — it accepts image_url, medium, direction, aspect_ratio, framing, subject_count, style, avoid, target_model, interface, and advanced, and POSTs to /api/prompt-director/from-image:

{
  "name": "image_to_prompt",
  "arguments": {
    "image_url": "https://example.com/reference.png",
    "medium": "anime",
    "direction": "same pose, soft light"
  }
}

If you only need a caption — no prompt package — the sibling describe_image MCP tool (REST: POST /api/describe-image) returns plain description, task_type, and prompt fields using the same vision backend. A good agent pattern: describe_image for quick understanding, image_to_prompt when you are about to generate.

Pricing and honest limits

This is a paid tool with no free daily quota: each call bills 2 flexible credits for the vision pass, and the optional Advanced Optimizer adds 1 credit on success. The package compilation itself is free — you are paying for the vision model. A few limits worth knowing:

• Uploads capped at 8 MB — use a hosted URL for larger images.
• Vision prompt is clamped — vision_prompt max 500 chars, direction max 1000 chars.
• One subject by default — multi-subject scenes (3+) generate warnings recommending regional conditioning and inpainting instead of prompting alone.
• No identity preservation — the vision pass is instructed to describe visual content only, not to recreate or preserve a specific character's identity.

A complete workflow

Here is a realistic end-to-end chain: an agent receives a reference image, reverse-engineers it into a prompt package, then generates a new image with the same composition:

The value is a consistent starting point: the vision model describes visible details and the compiler normalizes them for a target workflow. The result is still an interpretation, not the original prompt or a guarantee that another model will reproduce the source image.

Related: Prompt Director API → · AI image generation → · Background removal →