Minifetch is a pay-per-URL SEO toolkit of composable extraction primitives for AI agents and automation pipelines. Run them as a full audit or call one at a time for a fraction of the price — and a fraction of the tokens. No subscription.
In the browser, we offer a free "Check URL" button (and API endpoint) to see if a URL is likely accessible or blocked by robots.txt before spending credits to fetch it.
We cannot guarantee a URL will be accessible just based on robots.txt. A site may block access for many reasons. Checking robots.txt first is a reasonable step before fetching to ensure success.
If you own the website in question, we have a tutorial to help you unblock Minifetch: How To Allow Minifetch to Fetch Your Pages
Websites can publish a robots.txt file that says "don't access these pages." We obey those rules. This means Minifetch might return an error that says "blocked by robots.txt", or a simple 502 status error instead of data to indicate your target URL was explicitly blocked. It also means we're operating ethically and transparently by respecting site owners' intellectual property. Minifetch does not charge users for explicitly blocked pages like this.
We also offer a free "Check URL" button (and API endpoint) to see if a URL is likely accessible or blocked by robots.txt before spending credits to fetch it. For full policies, check our Terms of Service.
In a browser, the "Go Fetch!" button on the homepage fetches and runs an SEO audit on the target URL you enter into the search box.
If you are signed in to your account, a successful "Go Fetch!" will be deducted from your credit balance. You can sign up for an account and get started for free by visiting the dashboard.
If you do not wish to sign up for an account, you will be prompted to use your browser plugin wallet to pay in USDC cryptocurrency on Coinbase's Base network or Solana. This service uses Coinbase's x402 protocol to facilitate these payments.
Either way you pay as you go. There is no Minifetch account setup or monthly fee. We do not charge for blocked pages or pages that error.
Yes, full API documentation is available →
We also provide API docs in LLMs.txt format for your AI agents.
For coders, the Minifetch API client on npm has a good README for getting setup quickly in your Javascript/ Typescript project.
We surface those in our audits and data, but we don't obey them. Here's why:
Not directly. The "Check URL" button checks robots.txt for the URL you enter, so bit.ly links will check bit.ly's rules, not the final destination page.
Solution: Expand the shortened URL first (paste it in your browser, copy the final URL), then use the final URL in Minifetch.
Minifetch only extracts publicly available metadata and content from pages accessible without authentication and Javascript execution.
What Minifetch does NOT do:
What Minifetch does NOT do currently but may offer in the future as an add-on:
Minifetch.com is the hosted version of a free and open-source npm package called url-metadata that has been serving the NPM community for 10+ years. During that time some users expressed a preference for a hosted service for various reasons, mostly technical hurdles with coding or accidentally triggering blocks from websites when using the open-source package.
minifetch-apiNEW — the official Minifetch API client for Javascript/ Typescript
npm install minifetch-api --save
url-metadata — the open-source npm package (10+ years, 4M+ downloads) that powers Minifetch
npm install url-metadata --save
Both from the same creator with 4 Million+ downloads over 10 years.
Add them to your project with confidence.
If you have questions or need assistance:
join our Discord server.
To leave feedback or join the bulk credits waitlist:
use our feedback form.
While you're at it, give us a follow on X.com (Twitter):
@minifetch