← Back to blog

Web Page to Markdown API: Clean LLM Context From Any URL

October 4, 2026 · Eric · 7 min read

Hand a model a raw innerHTML dump and you have mostly paid for navigation menus, cookie banners, tracking scripts, and inline SVG. Hand it Markdown and you keep the structure that actually matters — headings, lists, links, code blocks — while dropping the parts that burn context for nothing.

Agent Media Tools ships two free REST endpoints for that job. POST /api/article-extract takes a URL and returns the readable article as Markdown, using the same class of readability algorithm behind browser reader modes. POST /api/html-to-md takes raw HTML you already have — an email body, a CMS export, a fragment your own agent fetched — and converts it. Neither needs a headless browser, an API key, or a package install.

Why Markdown and not plain text

It is tempting to strip tags and send flat text, because it is fewer tokens. That trade usually loses:

If you want to know the actual cost before you build a pipeline, the same platform exposes POST /api/token-count, which we will wire in below.

Endpoint 1: extract an article as Markdown

Send a URL, get back metadata plus the body as Markdown. No authentication is required for a normal daily allowance of calls.

curl
curl -s -X POST https://agentmediatools.com/api/article-extract \
  -H "Content-Type: application/json" \
  -d '{"url":"https://en.wikipedia.org/wiki/Markdown"}' | python3 -m json.tool

There is also a GET form, handy for a browser tab or a quick smoke test:

curl "https://agentmediatools.com/api/article-extract?url=https://en.wikipedia.org/wiki/Markdown"

The response carries exactly these fields (values shortened here):

FieldWhat it holds
successBoolean. On failure you also get an error string.
titleArticle headline as detected by the extractor.
authorByline, when the page declares one.
datePublished time, when the page declares one.
site_nameFrom the page's og:site_name tag.
excerptShort summary of the extracted body.
word_countWords in the extracted body.
reading_time_minutesEstimated at roughly 200 words per minute, floor of 1.
contentThe Markdown body. Headings are ATX (##), code blocks are fenced.
text_contentPlain-text version, capped at the first 10,000 characters.
source_urlThe URL you asked for.
toolAlways article-extract, useful for logging.
python
import requests

API = "https://agentmediatools.com/api"

doc = requests.post(
    f"{API}/article-extract",
    json={"url": "https://en.wikipedia.org/wiki/Markdown"},
    timeout=30,
).json()

if not doc.get("success"):
    raise SystemExit(doc.get("error"))

print(doc["title"])
print(doc["word_count"], "words ·", doc["reading_time_minutes"], "min read")
print(doc["content"][:400])

A few boundaries are worth knowing before you point this at a thousand URLs. The fetch runs server-side with a 15-second timeout and reads at most 2 MB of HTML, and it only accepts pages served as text/html — a PDF or JSON URL is rejected rather than half-converted. Failures come back as {"success": false, "error": "..."} with a 400 for a missing or non-HTML URL and a 500 if extraction itself blows up. Because the page is fetched, not rendered, JavaScript-only single-page apps can come back thin; for those, use the browser-backed tooling described in the AgentBrowse guide instead of fighting this endpoint.

Endpoint 2: convert HTML you already have

Sometimes there is no URL. You have HTML in hand: a newsletter body pulled from an inbox API, a CMS field, a template you are about to render, or a fragment your agent fetched through its own HTTP client. POST /api/html-to-md converts it without any network fetch at all.

curl
curl -s -X POST https://agentmediatools.com/api/html-to-md \
  -H "Content-Type: application/json" \
  -d '{"html":"

Release notes

Fixed the --fast flag.

  • Faster
  • Quieter
"}' \ | python3 -m json.tool

The body takes a single html string. The response returns:

FieldWhat it holds
successBoolean.
markdownThe converted document. ATX headings, fenced code blocks.
chars_inLength of the HTML you sent.
chars_outLength of the Markdown you got back.
ratiochars_out / chars_in, as a string — a quick size sanity check on your corpus.

In Python it is a one-liner in the middle of a pipeline:

html = newsletter_body_html  # from your own fetch or database
md = requests.post(f"{API}/html-to-md", json={"html": html}, timeout=30).json()["markdown"]

Pick the right one and stop guessing: URL in, Markdown out is article-extract; HTML in, Markdown out is html-to-md. If you are converting a whole PDF rather than a web page, that is a different endpoint — see the PDF processing guide.

Check the token budget before you build

Conversion is only half the job. The other half is knowing whether the result fits. POST /api/token-count estimates tokens for several model families at once, with a default set of gpt-4o, claude-3-5-sonnet, llama-3, and mistral — pass your own models array to override.

curl -s -X POST https://agentmediatools.com/api/token-count \
  -H "Content-Type: application/json" \
  -d '{"text":"paste a chunk here","models":["gpt-4o","claude-3-5-sonnet"]}' \
  | python3 -m json.tool

You get char_count, word_count, line_count, a tokens object keyed by model name, and cost_estimates_usd derived from the platform's built-in per-model rates. The response also carries an explicit note that these are BPE approximations — expect roughly a 5–10% spread against a vendor tokenizer — which is the right level of honesty for a budgeting call. Text is capped at 2,000,000 characters.

A working ingestion loop

Here is the whole pattern assembled: fetch a page as Markdown, split it on headings so each chunk stays coherent, then budget the prompt.

python
import re
import requests

API = "https://agentmediatools.com/api"

def page_markdown(url: str) -> dict:
    r = requests.post(f"{API}/article-extract", json={"url": url}, timeout=30)
    r.raise_for_status()
    doc = r.json()
    if not doc.get("success"):
        raise RuntimeError(doc.get("error"))
    return doc

def chunk_by_heading(markdown: str, max_chars: int = 1800) -> list:
    # Split before any ## or ### heading, keeping the heading with its section.
    parts = re.split(r"\n(?=#{2,3} )", markdown)
    chunks, buf = [], ""
    for part in parts:
        if buf and len(buf) + len(part) > max_chars:
            chunks.append(buf.strip())
            buf = part
        else:
            buf = f"{buf}\n{part}".strip()
    if buf:
        chunks.append(buf)
    return chunks

doc = page_markdown("https://en.wikipedia.org/wiki/Markdown")
print(f'{doc["title"]} — {doc["word_count"]} words, {doc["reading_time_minutes"]} min')

chunks = chunk_by_heading(doc["content"])
size = requests.post(f"{API}/token-count", json={
    "text": "\n\n".join(chunks),
    "models": ["gpt-4o"],
}, timeout=30).json()

print(len(chunks), "chunks ·", size["tokens"]["gpt-4o"], "estimated tokens")
for c in chunks[:3]:
    print(" •", c.splitlines()[0][:70])

Two practical notes. Keep the heading line in every chunk — a bare paragraph with no heading is much harder to route to the right document later. And if your chunks are large because a single section is long, split inside the section on paragraph boundaries rather than mid-sentence.

From an agent, with no code at all

Agent Media Tools runs a hosted MCP server at https://agentmediatools.com/api/mcp. It exposes the extractor as the extract_article tool, which takes one required argument, url, and returns the same Markdown plus metadata. So a Claude or Hermes agent can read a link properly — fetch, extract, summarise — inside a normal conversation. Setup is covered in the MCP setup walkthrough.

The raw-HTML conversion has no dedicated MCP tool; call POST /api/html-to-md directly from your agent's shell or HTTP tool when you are holding the markup yourself.

Limits and honest boundaries

Wrap up

Two endpoints cover most ingestion headaches: article-extract when you have a URL, html-to-md when you have markup, and token-count to keep the result inside a budget. Markdown in, structure preserved, no browser required.

Try it in the Toolbox

Run the same tools in the browser, or call them from your agent with an API key.

Open toolbox