For SEO research and AI projects
Minifetch provides composable extraction APIs that humans and AI Agents can autonomously discover, orchestrate, and pay for— making web pages simple to access. The API follows ethical scraping practices, respecting robots.txt and using rate limiting to avoid overloading target websites. You can pay-as-you-go with x402 micropayments in USDC, typically for SEO research and AI projects such as AI Agents, RAG pipelines, and building training datasets. There is no Minifetch account setup or monthly fee.
This documentation covers the the main endpoints available for developers. All endpoints return responses in JSON format.
This service uses the x402 protocol for micropayments with USDC on Coinbase's Base network or Solana. You can pay as you go—there is no Minifetch account setup or monthly fee. For payment implementation details, see the x402 protocol documentation.
You can use our homepage or visit the API endpoints directly in your browser. It's free to check if a URL is fetchable using the "Check URL" button on the Minifetch.com homepage. For paid endpoints, you'll be prompted to pay with your browser plugin wallet.
For programmatic access, we're building an API client library that handles x402 payments for you. Sign up for the waitlist and we'll notify you when it's ready.
We're also working on MCP (Model Context Protocol) integration so AI agents can use Minifetch directly. Stay tuned!
All API requests should be made to:
https://minifetch.com
Free
Call this endpoint before the extraction endpoints
This free endpoint checks if a URL is allowed to be fetched according to the website's robots.txt file. Use this before making paid requests to avoid spending extra on un-fetchable URLs. This ensures ethical scraping that respects robots.txt and site policies so you can sleep better at night. You can visit the API endpoint directly using your browser or click the "Check URL" button from the Minifetch.com homepage. The Minifetch API client library will also handle payment for you, sign up for our waitlist and we'll notify you when its ready.
| Parameter | Type | Required | Description |
|---|---|---|---|
| url | string | Yes | The URL to check against robots.txt |
GET https://minifetch.com/api/v1/free/preflight/url-check?url=https://example.com{
"success": true,
"queryParameters": {"url": ["https://example.com"]},
"results": [
{
"url": "https://example.com",
"allowed": true,
"message": "robots.txt not found, defaulting to allowed",
"crawlDelay": 1
}
]
}$0.002 USDC
SEO Research, Content Indexing, Building AI Training Datasets
This endpoint fetches and extracts rich structured metadata from a URL: title, description, og/twitter tags, json-ld, images, headings, response headers, and more. You can visit the API endpoint directly using your browser or click the "Go Fetch!" button from the Minifetch.com homepage. The Minifetch API client library will also handle payment for you, sign up for our waitlist and we'll notify you when its ready.
| Parameter | Type | Required | Description |
|---|---|---|---|
| url | string | Yes | The URL from which to extract metadata |
| includeResponseBody | boolean | No | If set to "true", includes the full HTML response body in the result |
GET https://minifetch.com/api/v1/x402/extract/url-metadata?url=https://example.com&includeResponseBody=true{
"success": true,
"queryParameters": {
"url": ["https://example.com"],
"includeResponseBody": true
},
"results": [
{
"metadata": {
"requestUrl": "https://example.com", // URL you requested
"url": "https://example.com", // Last URL in request chain, if there were redirects
"responseStatusCode": 200,
"responseHeaders": {},
"canonical": "",
"lang": "en",
"hreflang": [],
"charset": "utf-8",
"viewport": "",
"title": "Example Domain",
"favicons": [],
// Additional metadata fields, see "Example Metadata" section below
}
}
]
}$0.002 USDC
SEO, Link Validation & Analysis, Sitemap Building & Crawl Planning
Extracts all links from a URL categorized by type (internal/external/anchor) with SEO metadata. Detects image links, nofollow attributes, and analyzes external domain distribution. You can visit the API endpoint directly using your browser or click the "Options" button on the Minifetch.com homepage, select "Extract Links" then "Go Fetch!". The Minifetch API client library will also handle payment for you, sign up for our waitlist and we'll notify you when its ready.
| Parameter | Type | Required | Description |
|---|---|---|---|
| url | string | Yes | The URL from which to extract links |
GET https://minifetch.com/api/v1/x402/extract/url-links?url=https://example.com{
"success": true,
"queryParameters": {
"url": ["https://example.com"]
},
"results": [
{
"links": {
"requestUrl": "https://example.com",
"url": "https://example.com",
"links": {
"internal": [
{
"href": "https://example.com/about",
"text": "About Us",
"rel": []
}
],
"external": [
{
"href": "https://twitter.com/example",
"text": "Follow us",
"rel": ["nofollow", "ugc"]
},
{
"href": "https://github.com/example",
"text": "",
"rel": [],
"hasImage": true,
"imageAlt": "GitHub logo"
}
],
"anchors": [
{
"href": "#pricing",
"text": "Jump to pricing"
}
],
"summary": {
"totalLinks": 3,
"internalCount": 1,
"externalCount": 2,
"anchorCount": 1,
"nofollowCount": 1,
"uniqueExternalDomains": 2,
"topExternalDomains": [
{ "domain": "twitter.com", "count": 1 },
{ "domain": "github.com", "count": 1 }
]
}
}
}
}
]
}$0.001 USDC
AI Agents & RAG Pipelines
This endpoint fetches and extracts a light, token-efficient preview of a URL: title, description, and image (only). You can visit the API endpoint directly using your browser. The Minifetch API client library will also handle payment for you, sign up for our waitlist and we'll notify you when its ready.
| Parameter | Type | Required | Description |
|---|---|---|---|
| url | string | Yes | The URL from which to extract the preview |
GET https://minifetch.com/api/v1/x402/extract/url-preview?url=https://example.com{
"success": true,
"queryParameters": {
"url": ["https://example.com"]
},
"results": [
{
"metadata": {
"requestUrl": "https://example.com", // URL you requested
"url": "https://example.com", // Last URL in request chain, if there were redirects
"title": "Example Domain",
"description": "",
"image": ""
}
}
]
}$0.002 USDC
AI Agents & RAG Pipelines
Extracts a clean, LLM-ready, token-efficient content summary as markdown from a URL. Removes ads, nav, scripts. Much more efficient than raw HTML fetches. You can visit the API endpoint directly using your browser. The Minifetch API client library can also handle payment for you, sign up for our waitlist and we'll notify you when its ready.
| Parameter | Type | Required | Description |
|---|---|---|---|
| url | string | Yes | The URL from which to extract content |
| includeMediaUrls | boolean | No | If set to "true", includes image and video URLs in the result |
GET https://minifetch.com/api/v1/x402/extract/url-content?url=https://apnews.com/article/time-person-of-year-2025-77ec65c6792bc99ec2ce1919c5f421ea&includeMediaUrls=true{
"success": true,
"queryParameters": {
"url": ["https://apnews.com/article/time- ...etc"],
"includeMediaUrls": "true"
},
"results": [
{
"content": {
"requestUrl": "https://apnews.com/article/time- ...etc", // URL you requested
"url": "https://apnews.com/article/time- ...etc", // Last URL in request chain, if there were redirects
"summary": "# Time magazine names 'Architects of AI' as its person of the year... etc"
"mediaUrls": [
{
"url": "https://assets.apnews.com/19/66/bc546486408c8595f01753a9fbeb/ap-logo-176-by-208.svg",
"alt": "AP Logo"
},
... etc
]
}
}
]
}To see example responses, check these results:
Extract URL Metadata ($0.002 each) SEOAI
Extract URL Links ($0.002 each) SEOAI
Extract URL Preview ($0.001 each) AI
Extract URL Content ($0.002 each) AI
Minifetch implements several measures to ensure responsible and efficient operation:
These measures help us maintain an ethical scraping approach while providing reliable service.
Minifetch.com is the hosted version of a free and open-source npm package called url-metadata that has been serving the community for 10+ years.
During that time some users expressed a preference for a hosted service for various reasons, mostly technical hurdles with coding or accidentally triggering blocks from websites when using the open-source package.
For direct integration with your codebase, consider:
3.6 Million+ downloads over 10 years.
Docs are here.
Add to your project:
npm install url-metadata --save
If you have questions about the API or need assistance with implementation, use our feedback form and we'll get back to you.