Most APIs were designed for humans with a browser, not for LLMs with a function-calling loop. Your model can call POST /v1/items just fine — if it can figure out what the endpoint does, what it needs, and what comes back. Too often it can't, because the OpenAPI document that should tell it is missing descriptions, skips response schemas, or forgets to document authentication. The result is a "tool-calling API" that wastes tokens on hallucinated parameters and retries.
Agent QA is Agent Media Tools' answer: a deterministic, no-model quality gate for agent-facing APIs. Paste an OpenAPI document and get a 0–100 readiness score with a letter grade and a list of concrete fixes. Compare any spec against a saved baseline for a CI-friendly pass/fail gate. Run bounded read-only checks against live HTTPS endpoints, then schedule those runtime checks so endpoint failures surface early.
The scoring and runtime endpoints are public — no signup, no key. Scheduled monitors need a Founding Pro or Teams plan. Here's the whole workflow, with curl and Python examples that work today.
Every OpenAPI 3.x document gets scored against 16 deterministic checks in seven categories: foundation, authentication, tools, inputs, outputs, reliability, and support. A weighted rubric normalizes the result to a 0–100 score with a letter grade (A through F), so the output is stable and explainable — the same spec always produces the same score, and every finding carries a specific recommendation.
info.description that explains the API, and an HTTPS server URL.securitySchemes block and security requirements actually applied to operations.operationIds on every operation, and a summary or description per operation.requestBody schemas on mutating operations, documented responses, and machine-readable response schemas.Retry-After), and idempotency behavior for mutations.externalDocs or contact/support links.Those checks map directly to what an agent needs to call an API safely: a stable identifier, a description it can reason over, and schemas it can validate against. The tool is honest about its limits — it reviews documentation structure only. It never executes your operations, verifies credentials, or proves runtime security.
POST /api/agent-qa/openapi accepts an OpenAPI JSON object (up to 1 MB, up to 1,000 operations) and returns the full report. No URL is fetched and no key is required.
curl -s -X POST https://agentmediatools.com/api/agent-qa/openapi \ -H "Content-Type: application/json" \ -d @openapi.json | python3 -m json.tool
The report includes score, grade, a summary (title, path and operation counts, checks passed), and a findings array where each item has an id, a severity (important or improvement), and a concrete recommendation:
{
"score": 62,
"grade": "D",
"summary": {
"title": "Example API",
"operations": 4,
"passed_checks": 9,
"total_checks": 16
},
"findings": [
{
"id": "operation-ids",
"severity": "important",
"finding": "Every operation has a stable operationId.",
"recommendation": "Give every operation a unique, stable, verb-led operationId."
}
]
}
In Python:
import json
import requests
with open("openapi.json") as f:
spec = json.load(f)
resp = requests.post(
"https://agentmediatools.com/api/agent-qa/openapi",
json={"spec": spec},
)
data = resp.json()
print(f"Score: {data['score']}/100 ({data['grade']})")
for finding in data["findings"]:
print(f"- [{finding['severity']}] {finding['finding']}")
print(f" Fix: {finding['recommendation']}")
Fix the findings, re-run, and watch the number climb. Because the scoring is deterministic, a score of 80 today means the same thing as a score of 80 next quarter.
A single score is a snapshot. To catch regressions — someone removes a response schema, renames an operationId, or drops the security requirement — compare the current spec against a saved baseline with POST /api/agent-qa/compare. You can pass either raw OpenAPI documents or previously generated Agent QA reports, plus thresholds that form a pass/fail gate:
min_score — fail if the current score drops below this (default 0).max_score_drop — fail if the score drops more than this many points (default 0).fail_on_new_findings — fail when a previously passing check becomes a finding (default true).curl -s -X POST https://agentmediatools.com/api/agent-qa/compare \
-H "Content-Type: application/json" \
-d '{
"current_spec": {"openapi": "3.1.0", "info": {"title": "Example API", "version": "1.1.0"}, "paths": {}},
"baseline_report": {"success": true, "score": 85, "summary": {}, "findings": [], "checks": [], "limitations": []},
"thresholds": {"min_score": 80, "max_score_drop": 5}
}' | python3 -c "import json,sys; r=json.load(sys.stdin); print('GATE:', 'PASS' if r['gate']['passed'] else 'FAIL'); print('Delta:', r['comparison']['score_delta'])"
The response's gate.conditions explain exactly which threshold failed, and exports includes ready-made Markdown and JUnit XML reports. Write the JUnit output to a file for CI systems that support JUnit reports. The Agent QA page also links a ready-made GitHub Actions workflow you can download.
A minimal Python CI gate that fails the build:
import json
import sys
import urllib.request
payload = {
"current_spec": json.load(open("openapi.json")),
"baseline_report": json.load(open("agent-qa-baseline.json")),
"thresholds": {"min_score": 80, "max_score_drop": 5},
}
req = urllib.request.Request(
"https://agentmediatools.com/api/agent-qa/compare",
data=json.dumps(payload).encode(),
headers={"Content-Type": "application/json"},
)
result = json.load(urllib.request.urlopen(req))
open("agent-qa-junit.xml", "w").write(result["exports"]["junit"])
sys.exit(0 if result["gate"]["passed"] else 1)
Documentation can look perfect and the endpoint can still be down. POST /api/agent-qa/runtime runs 1–5 bounded checks against public HTTPS endpoints using only GET and HEAD — no credentials, no request bodies, no private networks, no mutations. Each test can assert an expected status code, a content type, and a supported JSON Schema subset for the response body. Evidence is bounded and redacted, responses are capped at 256 KB, and each request has an eight-second timeout. The result is useful regression evidence, not proof of security or availability.
curl -s -X POST https://agentmediatools.com/api/agent-qa/runtime \
-H "Content-Type: application/json" \
-d '{
"tests": [
{
"name": "health endpoint",
"method": "GET",
"url": "https://api.example.com/health",
"expect": {"statuses": [200], "content_type": "application/json"}
},
{
"name": "items list shape",
"method": "GET",
"url": "https://api.example.com/items",
"expect": {
"statuses": [200],
"schema": {
"type": "object",
"required": ["items"],
"properties": {"items": {"type": "array"}}
}
}
}
]
}'
The result is a gate plus per-test detail: status, latency, a small allow-listed header set, the individual checks, and a redacted response preview.
{
"gate": {"passed": true, "passed_tests": 2, "total_tests": 2},
"results": [
{
"name": "health endpoint",
"status": 200,
"latency_ms": 132,
"passed": true,
"checks": [{"id": "status", "passed": true, "message": "Received 200; expected 200."}]
}
]
}
The runtime endpoint is public and rate-limited; the same verifier powers the scheduled monitors below.
Manual checks are fine for a one-off audit. For ongoing coverage, create a monitor: a named set of 1–5 runtime tests with an interval from 15 to 10,080 minutes (weekly). The Agent Media Tools worker runs it on schedule, stores a history of the last 100 runs per monitor, and sends an email on the first failed run of an incident rather than repeating the same alert on every consecutive failure. Personal monitors require Founding Pro (up to 10), and Teams shares up to 100 monitors across five members with roles, history, and audit events.
Monitor endpoints use the same auth as the rest of the API — a Bearer agent key (mt_-prefixed, created in your account) or your session. Create one:
curl -s -X POST https://agentmediatools.com/api/agent-qa/monitors \
-H "Authorization: Bearer mt_..." \
-H "Content-Type: application/json" \
-d '{
"name": "customer-api-readiness",
"interval_minutes": 60,
"tests": [
{
"name": "health endpoint",
"method": "GET",
"url": "https://api.example.com/health",
"expect": {"statuses": [200], "content_type": "application/json"}
}
]
}'
Then list monitors, run one immediately, or pull its history:
curl -s https://agentmediatools.com/api/agent-qa/monitors \ -H "Authorization: Bearer mt_..." curl -s -X POST https://agentmediatools.com/api/agent-qa/monitors/1/run \ -H "Authorization: Bearer mt_..." curl -s https://agentmediatools.com/api/agent-qa/monitors/1/runs \ -H "Authorization: Bearer mt_..."
The core workflow is also available through MCP tools: analyze_openapi_readiness, compare_openapi_readiness, verify_api_runtime_readonly, list_agent_qa_monitors, create_agent_qa_monitor, and run_agent_qa_monitor. Configure the local Agent Media Tools MCP server with your API key for authenticated monitor tools (see the Claude MCP setup guide), then audit a spec from an MCP-compatible assistant. Monitor editing, deletion, and run-history retrieval remain REST/browser operations.
That's the full loop: score your spec, gate it in CI, prove the endpoints respond, and keep watching them on a schedule. Ten minutes of setup buys you the one thing agent integrations never have enough of — early warning.
Paste an OpenAPI document and get a readiness score and fix list instantly — no signup required.
Open Agent QA