Structured Web Scraping API: Extract Data from Any Page with CSS Selectors

Most web scraping APIs dump the entire page HTML and make you parse it. Others run a full headless browser that burns memory and takes seconds per page. Agent Media Tools takes a different approach: you send a URL and a JSON schema describing exactly what fields you want, and the API returns clean structured data — no browser, no heavy DOM dumps, no parsing on your end.

The /api/structured-scrape endpoint fetches a public URL, parses the HTML with Cheerio (a fast server-side jQuery), and runs your schema against the page. You can target elements by CSS selectors, extract attributes or text content, and even use built-in heuristics for common fields like titles, prices, emails, and headings.

Best of all: it works without a browser. No WebDriver, no Puppeteer, no Chrome binary eating your RAM. Just one HTTP call that returns JSON.

Why Skip the Browser?

Traditional scraping with a headless browser is powerful, but it also adds another browser runtime to operate. When the fields already exist in server-rendered HTML, a bounded HTTP fetch and selector pass is often the simpler path.

The structured-scrape API works with plain HTTP. It fetches the HTML directly, which means:

• Bounded requests with a 25-second timeout and 2 MB response limit
• No browser dependencies in your agent's runtime
• Predictable costs — free daily calls, then 2 flexible credits

The trade-off is that you only get the server-rendered HTML. Pages that rely heavily on JavaScript rendering won't produce the same DOM as a browser would. For most content-driven pages (product listings, documentation, news articles, directories), static HTML is all you need.

How the Schema Works

You define a JSON schema with a properties object. Each property can have:

If you omit a selector, the API falls back to heuristic matching — it recognizes common field names like title, description, price, email, headings, and links by scanning the page's meta tags, headings, and body text with regex patterns.

curl Examples

Let's scrape a product page. Here's a schema that extracts the title, price, and description:

curl -X POST "https://agentmediatools.com/api/structured-scrape" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/product",
    "schema": {
      "type": "object",
      "properties": {
        "title": {
          "type": "string",
          "selector": "h1"
        },
        "price": {
          "type": "number",
          "selector": ".price"
        },
        "description": {
          "type": "string",
          "selector": "meta[name=description]",
          "attr": "content"
        }
      },
      "required": ["title"]
    }
  }'

The API returns something like:

{
  "success": true,
  "url": "https://example.com/product",
  "data": {
    "title": "Premium Widget",
    "price": 29.99,
    "description": "A high-quality widget for everyday use."
  },
  "missing_fields": [],
  "required_missing": [],
  "page": {
    "title": "Premium Widget — Example Store",
    "description": "A high-quality widget for everyday use.",
    "headings": ["Premium Widget", "Features", "Specifications"]
  },
  "billing": { "charged": false, "tier": "free" }
}

Notice the billing field — when you're within your free daily quota, it tells you "charged": false. No surprises.

Extracting Multiple Items

Use "type": "array" or "multiple": true to collect all matching elements. Here's how you'd scrape all the links from a page:

curl -s -X POST "https://agentmediatools.com/api/structured-scrape" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "schema": {
      "properties": {
        "links": {
          "type": "array",
          "selector": "a[href]",
          "attr": "href"
        },
        "headings": {
          "type": "array",
          "selector": "h2"
        }
      }
    }
  }' | jq .data

Heuristic-Only Scrape

If you skip selectors entirely and use heuristic-recognized field names, the API extracts them from metadata and content analysis:

curl -s -X POST "https://agentmediatools.com/api/structured-scrape" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/blog/post",
    "schema": {
      "properties": {
        "title": {},
        "description": {},
        "author": {},
        "emails": {}
      }
    }
  }'

The heuristic engine recognizes title, description, author, headings, links, images, email/emails, price/amount/cost, and more — all without a single CSS selector.

Python Example

For Python agents and scripts, use the requests library:

import requests
import json

url = "https://agentmediatools.com/api/structured-scrape"

schema = {
    "type": "object",
    "properties": {
        "title": {"type": "string", "selector": "h1"},
        "price": {"type": "number", "selector": ".price"},
        "description": {"type": "string", "selector": "meta[name=description]", "attr": "content"}
    },
    "required": ["title"]
}

resp = requests.post(url, json={
    "url": "https://example.com/product",
    "schema": schema
})

data = resp.json()
print(json.dumps(data["data"], indent=2))
print(f"Missing fields: {data['missing_fields']}")

The endpoint accepts POST requests. Keep the schema in the JSON body so selectors and required fields are represented unambiguously.

Using It from an AI Agent (MCP)

If your agent runs the Agent Media Tools MCP server, the structured_scrape tool is available directly:

• MCP tool name: structured_scrape
• Required inputs: url (string), schema (JSON object)

Your agent can invoke it like any other tool — the MCP server sends the request to the backend, parses the response, and returns clean JSON. No HTTP setup needed.

There's also a Pipeline Builder preset called url-structured that wraps this endpoint into a reusable workflow step. You can chain it with other tools — scrape a page, then send the result to a webhook, save it as an artifact, or feed it into another pipeline stage.

Pricing and Limits

No API key is required for the anonymous daily allowance. After the allowance, authenticate with an account that has available flexible credits. Founding Pro is currently $4.99/month and includes a higher daily allowance plus monthly credits; see the pricing page for the current offer.

When to Use It vs. a Browser

What Gets Returned

Every response includes:

• data — Your extracted fields, coerced to the requested types
• missing_fields — Fields that couldn't be matched (so you know what to fix in your schema)
• required_missing — Required fields that came back empty
• page — Page metadata: title, description, and first 10 headings extracted automatically
• billing — Whether this call was charged or free

This is designed so your agent can inspect missing_fields and adjust its selector strategy on the next attempt — a simple self-healing loop.

Real-World Use Cases

• SEO monitoring: Scrape meta descriptions, titles, and headings from your own or competitors' pages daily.
• Price tracking: Extract prices and availability from product pages, store results as artifacts, and alert on changes.
• Directory building: Scrape listings and extract structured fields (name, email, phone) into a spreadsheet.
• Agent context gathering: Feed scraped page metadata into an LLM agent as structured context — much cleaner than dumping raw HTML into a prompt.
• Documentation indexing: Extract headings and descriptions from docs pages to build a custom search index.

Give It a Try

No signup needed. Just send a curl command and you'll get back structured JSON. The free tier gives you 3 calls per day — enough to prototype your schema and build your workflow.

For recurring usage, compare Founding Pro with flexible-credit top-ups. Response time depends on the destination page and network; use AgentBrowse's browser mode instead when a page requires JavaScript rendering.