← Back to blog

Generate an XML Sitemap for Any Site with One API Call

October 22, 2026 · Eric · 7 min read

Every crawler you install locally solves the same problem twice: once for the headless browser, once for the queue and dedupe logic. Then it solves it a third time when the browser updates and your scraper stops launching. For a one-off task — audit a new client site, bootstrap a sitemap for a static deploy, hand an agent a list of URLs to read — that is a lot of infrastructure for a file that is a few hundred lines of XML.

This post documents the sitemap generator I run at POST /api/agent/sitemap/generate. You send a URL and a page budget. You get back the crawled page list and the finished <urlset> XML as a string, ready to write to disk. Below: the exact request and response shape, how the per-page credit math works, a Python script that saves and validates the output, a scheduled refresh, and the handful of things this crawler deliberately does not do.

The whole endpoint in one table

ItemValue
Method + pathPOST /api/agent/sitemap/generate
Body{"url": "https://example.com", "max_pages": 50}
max_pagesOptional. Defaults to 50, clamped to 5–500
Returnspages[] (url, title, priority, change_freq), sitemap_xml, summary, billing
Cost1 credit per 10 requested pages, minimum 1
Free tier1 crawl per day before credits apply
MCP toolNone — this one is REST only

That last row matters if you are wiring an assistant. Several tools on this platform have first-class MCP equivalents; the sitemap generator is not one of them yet, so an agent calls it as a plain HTTP tool (which is exactly what a fetch or requests tool does inside any tool-calling loop).

Step 1: the first call, no key required

A single crawl per day is free, so you can test the response shape before you think about billing. The request needs nothing but a JSON body:

curl -sS -X POST https://agentmediatools.com/api/agent/sitemap/generate \
  -H "Content-Type: application/json" \
  -d '{"url":"https://agentmediatools.com","max_pages":25}'

The response is one JSON envelope. Trimmed to two pages:

{
  "success": true,
  "domain": "agentmediatools.com",
  "base_url": "https://agentmediatools.com/",
  "elapsed_ms": 4180,
  "summary": {
    "pages_crawled": 25,
    "pages_discovered": 31,
    "limit_reached": true
  },
  "pages": [
    {
      "url": "https://agentmediatools.com",
      "title": "Agent Media Tools — 100+ tools for AI agents",
      "lastmod": null,
      "priority": "1.0",
      "change_freq": "daily"
    },
    {
      "url": "https://agentmediatools.com/pricing",
      "title": "Pricing",
      "lastmod": null,
      "priority": "0.8",
      "change_freq": "weekly"
    }
  ],
  "sitemap_xml": "\n\n  \n    https://agentmediatools.com\n    daily\n    1.0\n  \n",
  "sitemap_size_bytes": 6912,
  "billing": {
    "tool": "sitemap-generator",
    "free": true,
    "free_used": 1,
    "free_daily_limit": 1,
    "free_remaining": 0,
    "credit_cost_if_over": 3,
    "plan": "free"
  }
}

Two fields do most of the work for a pipeline: sitemap_xml is the finished document as a string (not a temp file URL), and summary.limit_reached tells you whether the crawl stopped because it ran out of budget or because it ran out of links.

Step 2: read the response correctly

pages_crawled vs pages_discovered. The first counts pages that returned HTML and were added to the sitemap. The second counts every URL the crawler dequeued, including pages that turned out to be non-HTML or failed to load. A gap between the two is normal — it usually means you have PDFs, images, or a few dead links in your navigation.

lastmod is always null. The crawler does not send conditional requests and most servers do not return a reliable Last-Modified, so the XML omits <lastmod> entirely rather than inventing a timestamp. If lastmod accuracy matters to you, that is a job for a build step that knows your content dates, not a crawler guessing.

Priority is positional, not editorial. The root gets 1.0 / daily; every other page gets 0.8 / weekly. Treat those as placeholders to edit, not as a recommendation from Google (which has said it ignores both tags anyway).

Errors arrive inside the envelope. A missing url is a real 400 with {"success": false, "error": "url required"}. Most other failures — DNS failure, a site that blocks the request, a crawl that dies partway — come back as HTTP 200 with success: false and an error string. Always branch on success, never on the status code alone.

Step 3: save and validate the sitemap in Python

This script crawls, writes sitemap.xml, then re-parses it with the standard library so a broken document fails loudly instead of shipping to production:

import xml.etree.ElementTree as ET
import requests

resp = requests.post(
    "https://agentmediatools.com/api/agent/sitemap/generate",
    json={"url": "https://example.com", "max_pages": 100},
    timeout=300,
)
data = resp.json()

if not data.get("success"):
    raise SystemExit(f"crawl failed: {data.get('error')}")

xml_text = data["sitemap_xml"]
with open("sitemap.xml", "w", encoding="utf-8") as fh:
    fh.write(xml_text)

root = ET.fromstring(xml_text)
ns = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
locs = [el.text for el in root.findall("sm:url/sm:loc", ns)]

print(f"{len(locs)} URLs, {data['sitemap_size_bytes']} bytes, "
      f"limit_reached={data['summary']['limit_reached']}")
print("spend:", data["billing"])

The timeout=300 is not decorative. Crawling is sequential — one page at a time, 15 seconds of patience per request — so a 500-page budget is a multi-minute call. Give the client a real timeout or you will cancel work you already paid for.

Pass an agent key once you go past the free daily crawl:

headers = {"Authorization": "Bearer mt_your_key_here"}
resp = requests.post(URL, json=payload, headers=headers, timeout=300)

Step 4: do the credit math before you set max_pages

Billing is based on the budget you request, not on what the crawler finds: cost = max(1, ceil(max_pages / 10)). That means the size of the job is predictable in advance, and it also means a site with 12 pages still costs you the full 50-page budget if you ask for 50.

max_pagesCreditsGood for
51A landing page and its children
50 (default)5Most marketing sites and docs sections
20020Blog archives, product catalogs
500 (clamped max)50The largest site this tool will walk in one call

The free daily allotment for this tool is 1 crawl on the free tier and 50 crawls on a Pro key before credits apply. Pro is $4.99/month and includes 100 credits a month, so a monthly 200-page refresh fits comfortably inside the subscription. Calls also count toward your normal daily API allowance (10/day anonymous, 25/day on a free account, 1,000/day on Pro).

Step 5: refresh it on a schedule

A sitemap that is regenerated by hand is a sitemap that is wrong. Two cheap options:

Option A

Run the script above from cron or a CI job that also deploys the site, and commit the regenerated file. The crawl is deterministic enough that a failed run is obvious in the diff.

Option B

Let an agent own it. Register the endpoint as a tool, then use the durable scheduler described in durable cron tasks for agents to run a weekly refresh, and post the result to a webhook inbox — see webhooks for agent pipelines.

Either way, diff the URL set rather than the file. A new page appearing is a signal worth acting on; a byte-level diff is mostly noise from titles and ordering.

What this crawler does — and refuses to do

Being precise here saves you a support ticket and me an apology:

If you need rendered pages, that is a different tool — the browser automation covered in the AgentBrowse work handles JavaScript, and structured scraping with CSS selectors handles pulling fields out of a page once you have its URL list.

The natural follow-up: check the links you just collected

A sitemap tells you which URLs exist. It does not tell you whether the links on those pages still resolve. The companion route is POST /api/agent/links/check:

curl -sS -X POST https://agentmediatools.com/api/agent/links/check \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","max_links":50}'

It fetches one page, collects up to max_links unique links (default 50, clamped to 1–200), and probes each with an 8-second timeout. You get summary.total_checked, summary.broken, summary.valid, and a results[] array with status, status_text, redirect, and a broken boolean per URL. Cost is 1 credit per 25 links checked, minimum 1, with 1 free check per day.

Run it across the ten most important URLs in your sitemap and you have a real broken-link report for about a credit — no crawler install, no queue, no browser binary to keep patched. That combined workflow (sitemap, then link health, then DNS and TLS checks on whatever looks suspicious) is the smallest honest version of a site audit an agent can run end to end.

Try it

The first crawl of the day is free and needs no account, so the fastest way to judge this tool is to point it at your own site and read the pages[] array. If the list looks right, the XML is right — it is generated from the same data.

Generate a sitemap in the Toolbox

Run the same tools in the browser, or call them from your agent with an API key.

Open toolbox