Hand a model a raw innerHTML dump and you have mostly paid for navigation menus, cookie banners, tracking scripts, and inline SVG. Hand it Markdown and you keep the structure that actually matters — headings, lists, links, code blocks — while dropping the parts that burn context for nothing.
Agent Media Tools ships two free REST endpoints for that job. POST /api/article-extract takes a URL and returns the readable article as Markdown, using the same class of readability algorithm behind browser reader modes. POST /api/html-to-md takes raw HTML you already have — an email body, a CMS export, a fragment your own agent fetched — and converts it. Neither needs a headless browser, an API key, or a package install.
It is tempting to strip tags and send flat text, because it is fewer tokens. That trade usually loses:
## boundaries keeps each section semantically whole, which is the difference between useful retrieval and confident nonsense.If you want to know the actual cost before you build a pipeline, the same platform exposes POST /api/token-count, which we will wire in below.
Send a URL, get back metadata plus the body as Markdown. No authentication is required for a normal daily allowance of calls.
curl -s -X POST https://agentmediatools.com/api/article-extract \
-H "Content-Type: application/json" \
-d '{"url":"https://en.wikipedia.org/wiki/Markdown"}' | python3 -m json.tool
There is also a GET form, handy for a browser tab or a quick smoke test:
curl "https://agentmediatools.com/api/article-extract?url=https://en.wikipedia.org/wiki/Markdown"
The response carries exactly these fields (values shortened here):
| Field | What it holds |
|---|---|
success | Boolean. On failure you also get an error string. |
title | Article headline as detected by the extractor. |
author | Byline, when the page declares one. |
date | Published time, when the page declares one. |
site_name | From the page's og:site_name tag. |
excerpt | Short summary of the extracted body. |
word_count | Words in the extracted body. |
reading_time_minutes | Estimated at roughly 200 words per minute, floor of 1. |
content | The Markdown body. Headings are ATX (##), code blocks are fenced. |
text_content | Plain-text version, capped at the first 10,000 characters. |
source_url | The URL you asked for. |
tool | Always article-extract, useful for logging. |
import requests
API = "https://agentmediatools.com/api"
doc = requests.post(
f"{API}/article-extract",
json={"url": "https://en.wikipedia.org/wiki/Markdown"},
timeout=30,
).json()
if not doc.get("success"):
raise SystemExit(doc.get("error"))
print(doc["title"])
print(doc["word_count"], "words ·", doc["reading_time_minutes"], "min read")
print(doc["content"][:400])
A few boundaries are worth knowing before you point this at a thousand URLs. The fetch runs server-side with a 15-second timeout and reads at most 2 MB of HTML, and it only accepts pages served as text/html — a PDF or JSON URL is rejected rather than half-converted. Failures come back as {"success": false, "error": "..."} with a 400 for a missing or non-HTML URL and a 500 if extraction itself blows up. Because the page is fetched, not rendered, JavaScript-only single-page apps can come back thin; for those, use the browser-backed tooling described in the AgentBrowse guide instead of fighting this endpoint.
Sometimes there is no URL. You have HTML in hand: a newsletter body pulled from an inbox API, a CMS field, a template you are about to render, or a fragment your agent fetched through its own HTTP client. POST /api/html-to-md converts it without any network fetch at all.
curl -s -X POST https://agentmediatools.com/api/html-to-md \
-H "Content-Type: application/json" \
-d '{"html":"Release notes
Fixed the --fast flag.
The body takes a single html string. The response returns:
| Field | What it holds |
|---|---|
success | Boolean. |
markdown | The converted document. ATX headings, fenced code blocks. |
chars_in | Length of the HTML you sent. |
chars_out | Length of the Markdown you got back. |
ratio | chars_out / chars_in, as a string — a quick size sanity check on your corpus. |
In Python it is a one-liner in the middle of a pipeline:
html = newsletter_body_html # from your own fetch or database
md = requests.post(f"{API}/html-to-md", json={"html": html}, timeout=30).json()["markdown"]
Pick the right one and stop guessing: URL in, Markdown out is article-extract; HTML in, Markdown out is html-to-md. If you are converting a whole PDF rather than a web page, that is a different endpoint — see the PDF processing guide.
Conversion is only half the job. The other half is knowing whether the result fits. POST /api/token-count estimates tokens for several model families at once, with a default set of gpt-4o, claude-3-5-sonnet, llama-3, and mistral — pass your own models array to override.
curl -s -X POST https://agentmediatools.com/api/token-count \
-H "Content-Type: application/json" \
-d '{"text":"paste a chunk here","models":["gpt-4o","claude-3-5-sonnet"]}' \
| python3 -m json.tool
You get char_count, word_count, line_count, a tokens object keyed by model name, and cost_estimates_usd derived from the platform's built-in per-model rates. The response also carries an explicit note that these are BPE approximations — expect roughly a 5–10% spread against a vendor tokenizer — which is the right level of honesty for a budgeting call. Text is capped at 2,000,000 characters.
Here is the whole pattern assembled: fetch a page as Markdown, split it on headings so each chunk stays coherent, then budget the prompt.
import re
import requests
API = "https://agentmediatools.com/api"
def page_markdown(url: str) -> dict:
r = requests.post(f"{API}/article-extract", json={"url": url}, timeout=30)
r.raise_for_status()
doc = r.json()
if not doc.get("success"):
raise RuntimeError(doc.get("error"))
return doc
def chunk_by_heading(markdown: str, max_chars: int = 1800) -> list:
# Split before any ## or ### heading, keeping the heading with its section.
parts = re.split(r"\n(?=#{2,3} )", markdown)
chunks, buf = [], ""
for part in parts:
if buf and len(buf) + len(part) > max_chars:
chunks.append(buf.strip())
buf = part
else:
buf = f"{buf}\n{part}".strip()
if buf:
chunks.append(buf)
return chunks
doc = page_markdown("https://en.wikipedia.org/wiki/Markdown")
print(f'{doc["title"]} — {doc["word_count"]} words, {doc["reading_time_minutes"]} min')
chunks = chunk_by_heading(doc["content"])
size = requests.post(f"{API}/token-count", json={
"text": "\n\n".join(chunks),
"models": ["gpt-4o"],
}, timeout=30).json()
print(len(chunks), "chunks ·", size["tokens"]["gpt-4o"], "estimated tokens")
for c in chunks[:3]:
print(" •", c.splitlines()[0][:70])
Two practical notes. Keep the heading line in every chunk — a bare paragraph with no heading is much harder to route to the right document later. And if your chunks are large because a single section is long, split inside the section on paragraph boundaries rather than mid-sentence.
Agent Media Tools runs a hosted MCP server at https://agentmediatools.com/api/mcp. It exposes the extractor as the extract_article tool, which takes one required argument, url, and returns the same Markdown plus metadata. So a Claude or Hermes agent can read a link properly — fetch, extract, summarise — inside a normal conversation. Setup is covered in the MCP setup walkthrough.
The raw-HTML conversion has no dedicated MCP tool; call POST /api/html-to-md directly from your agent's shell or HTTP tool when you are holding the markup yourself.
Two endpoints cover most ingestion headaches: article-extract when you have a URL, html-to-md when you have markup, and token-count to keep the result inside a budget. Markdown in, structure preserved, no browser required.
Run the same tools in the browser, or call them from your agent with an API key.
Open toolbox