Most agent stacks are blind by default. A model can read a JSON payload, call a scraper, and summarize a PDF, but the moment the useful information lives in a JPEG — a product photo, a dashboard screenshot, a scanned receipt — the loop breaks and someone has to look at it manually.
The Describe Image endpoint closes that gap with one POST: send a public image URL, get back a plain-text description, or ask a specific question about what the picture shows. This post covers the real endpoint (POST /api/describe-image), the shipped describe_image MCP tool, working curl and Python examples, and the limits you should design around. No invented SDKs — just the HTTP API that exists.
Every call runs a vision model (Moondream, served through FAL) and returns text. The task_type field decides how you use it:
| task_type | Default prompt | Good for |
|---|---|---|
caption | "Describe this image in detail." | Alt text, catalog descriptions, tagging, search indexing |
query | "What is in this image?" (or your own prompt) | Yes/no checks, reading a label, comparing a screenshot against an expectation |
Caption mode is the fastest path to a usable string. Query mode is where agents get interesting: instead of dumping a paragraph of scenery, you ask "Does this screenshot show a red error banner?" and get an answer you can branch on.
You need an agent key. Create one from the Agent Hub (the guided path at /first-win walks through key creation), then export it so it never lands in shell history:
export AMT_API_KEY="your-agent-key"
curl -s -X POST https://agentmediatools.com/api/describe-image \
-H "Authorization: Bearer $AMT_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://agentmediatools.com/images/og-blog.jpg","task_type":"caption","max_tokens":128}'
The response is small and boring on purpose — easy to feed straight into an agent's context window:
{
"success": true,
"description": "A dark editorial banner with gold serif lettering...",
"task_type": "caption",
"prompt": "Describe this image in detail.",
"model": "moondream",
"billing": { "tool": "describe-image", "free": false, "credits_charged": 2, "plan": "pro" }
}
Note the billing block. It tells you what was actually charged and which plan was used, so an agent can log real cost per call instead of guessing from a price table.
Switch to query and pass a prompt. Keep it short and closed-ended; vision models answer vague questions vaguely:
curl -s -X POST https://agentmediatools.com/api/describe-image \
-H "Authorization: Bearer $AMT_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://agentmediatools.com/icons/icon-512.png",
"task_type": "query",
"prompt": "What colors dominate this logo, and is there any text in it?",
"max_tokens": 96
}'
| Field | Notes |
|---|---|
url | Required for JSON calls. Must be reachable from the public internet. image_url is accepted as an alias. |
image | Multipart file alternative: curl -F "image=@photo.jpg". Uploads are capped at 8 MB. |
task_type | caption (default) or query. Anything else silently falls back to caption. |
prompt | Optional. Truncated to 500 characters server-side, so front-load the instruction. |
max_tokens | Clamped to 32–512, default 128. Captions longer than ~200 tokens rarely help an agent. |
If the image is not hosted anywhere, send it as multipart form data. The route reads the file, detects the mime type, and inlines it for the vision model:
curl -s -X POST https://agentmediatools.com/api/describe-image \ -H "Authorization: Bearer $AMT_API_KEY" \ -F "image=@/path/to/screenshot.png" \ -F "task_type=query" \ -F "prompt=Does this page show a login form? Answer yes or no."
Anything above 8 MB is rejected with a clear error — host large images somewhere and pass the URL instead. That is also the better pattern for agents: a URL is a single string you can retry without re-uploading bytes.
Same endpoint, wrapped in something you can drop into a pipeline step. This version batches a list of URLs, stops on the first hard failure, and keeps going on individual image errors:
import os
import requests
API = "https://agentmediatools.com/api/describe-image"
HEADERS = {"Authorization": f"Bearer {os.environ['AMT_API_KEY']}"}
def describe(image_url, prompt=None, task_type="caption", max_tokens=128):
payload = {"url": image_url, "task_type": task_type, "max_tokens": max_tokens}
if prompt:
payload["prompt"] = prompt
r = requests.post(API, headers=HEADERS, json=payload, timeout=120)
if r.status_code == 402:
raise RuntimeError(f"Out of credits: {r.json().get('error')}")
r.raise_for_status()
return r.json()["description"]
urls = [
"https://agentmediatools.com/images/og-blog.jpg",
"https://agentmediatools.com/icons/icon-512.png",
]
for url in urls:
try:
print(url, "->", describe(url, "What is the main subject? One sentence.", "query", 64))
except requests.HTTPError as e:
print("skipped", url, e)
Run the loop sequentially. Firing twenty concurrent vision calls at once mostly buys you rate-limit errors and a surprising bill; a ThreadPoolExecutor with three workers is plenty for catalog-sized batches.
The MCP server ships a matching tool, so Claude, Hermes, or any MCP client can use the same capability with no HTTP layer of its own. The tool is named describe_image and takes:
image_url — required, public image URLprompt — optional instruction or questiontask_type — caption or querymax_tokens — 32–512, default 128Two practical differences from the REST API: MCP calls pass a URL only (no file upload — put the bytes somewhere reachable first), and the tool returns the raw JSON as text content, so the agent sees description alongside billing. If you are wiring up a client for the first time, the MCP setup guide covers the connection side.
Let the agent decide between caption and query. Cheap triage: run query with "Is this image a screenshot of a webpage? yes/no" first, then spend a caption call only on the images that matter.
Loop over every image in your CMS, call the endpoint in caption mode, write description into the alt attribute, and flag anything under 40 characters for human review. This is the single highest-value use: it makes existing content accessible and indexable without a designer in the loop.
You already have a screenshot tool. Take a shot of the checkout page after a deploy, then ask a visual question: "Is there any visible error message or red banner?" If the answer starts with "yes", fail the pipeline. It will not catch layout regressions a pixel-diff would, but it catches the embarrassing ones — blank pages, broken sign-in modals, unstyled HTML — in about a second of model time.
User-submitted images are expensive to review. A query call that classifies "is this a photo of a document, a product, or a person?" lets you route to OCR, to a catalog queue, or to a human, before spending OCR or moderation credits on the wrong thing. Pair it with OCR extraction when the answer is "document" — vision describes, OCR reads characters.
This is the part most posts skip. From the shipped billing code:
402 with an error explaining the cost, plus a billing block containing the price and a buy_credits link — not a generic auth error, so your agent can report something useful.analyze_image tool, which reads the file rather than looking at it.If your goal is to understand an image, this endpoint is the right size: one call, one string. If your goal is to regenerate that image, use image to prompt, which runs the same vision pass and then compiles a positive/negative prompt package for Stable Diffusion or ComfyUI. A sensible chain is describe_image for triage, then image_to_prompt only on the images you actually want to rebuild — you pay the vision cost twice, so let the cheap decision come first.
The most common implementation mistake is asking for a 512-token description and pasting it into a tool result. Vision text is verbose, and a verbose caption crowds out the rest of your agent's context for no gain. Start at 64 tokens for yes/no questions, 128 for captions, and only raise it when a downstream step genuinely needs the detail. Your token budget — and your bill — will thank you.
Drop in an image URL and get a caption, or call POST /api/describe-image from your agent with an API key.