User FAQ

What does Minifetch actually do?

Minifetch provides composable extraction APIs that humans and AI Agents can autonomously discover, orchestrate, and pay for— making web pages simple to access. The API follows ethical scraping practices, respecting robots.txt and using rate limiting to avoid overloading target websites. You can pay-as-you-go with x402 micropayments in USDC, typically for SEO research and AI projects such as AI Agents, RAG pipelines, and building training datasets. There is no Minifetch account setup or monthly fee.

Explore the API →

What does "We ❤️ robots.txt" mean?

Websites can publish a robots.txt file that says "don't access these pages." We obey those rules. This means some pages might return "blocked by robots.txt" instead of data - but it also means we're operating ethically and transparently by respecting site owners' intellectual property.

For our users, we offer a free "Check URL" button (and API endpoint) to see if a URL is likely accessible or blocked by robots.txt before spending credits to fetch it.

What does the "Check URL" button do?

In the browser, we offer a free "Check URL" button (and API endpoint) to see if a URL is likely accessible or blocked by robots.txt before spending credits to fetch it.

We cannot guarantee a URL will be accessible just based on robots.txt. A site may block access for many reasons. Checking robots.txt first is a reasonable step before fetching to ensure success.

How do I pay for a "Go Fetch!"?

In the browser, the "Go Fetch!" button on the homepage extracts the data from the URL you enter into the search box. When you click "Go Fetch!" you'll be prompted to use your browser plugin wallet to pay in USDC cryptocurrency on Coinbase's Base network or Solana. You can pay as you go, there is no Minifetch account setup or monthly fee. This service uses Coinbase's x402 protocol to facilitate these payments.

Are there API Docs?

Full API documentation can be found here.

We also provide API docs in LLMs.txt format here for use by LLMs.

What about "noindex/nofollow" robots meta tags on individual pages?

We don't process those. Here's why:

• Robots meta tags in html pages tell search engines what to index
• We're not a search engine - we're an on-demand fetch service
• We work like a browser visiting a page, not a crawler indexing your site
• We respect robots.txt (crawling rules), but page-level noindex/nofollow meta tags control indexing, which we don't do

Can I check shortened URLs (bit.ly, t.co, etc.)?

Not directly. The "Check URL" button checks robots.txt for the URL you enter, so bit.ly links will check bit.ly's rules, not the final destination page.

Solution: Expand the shortened URL first (paste it in your browser, copy the final URL), then use the final URL in Minifetch.

How is Minifetch different from other scraping services?

• We always respect robots.txt (no sneaking around blocks)
• We respect rate-limits and cache to minimize server load
• We identify ourselves clearly with user-agent strings & ip address combo
• We use pay-per-request to prevent spam and abuse
• We respond to site owner & user feedback

What are Minifetch's service limitations?

What Minifetch does NOT do:

• Access authenticated or logged-in content
• Create accounts or log into user sessions
• Perform transactional actions (checkout, bidding, purchasing, form submissions)
• Bypass paywalls or access restricted content
• Ignore robots.txt directives

Minifetch only extracts publicly available metadata and content from pages accessible without authentication.

Can I monetize my content through Minifetch?

Yes! If you have premium content or APIs, we can work with you to:

• Set custom pricing for your URLs (via x402 protocol)
• Share revenue when users access your content through Minifetch
• Control access levels and pricing tiers

This lets site owners monetize access without managing payment infrastructure. Contact us thru our feedback form to discuss partnership opportunities.

Is Minifetch open-source?

Minifetch.com is the hosted version of a free and open-source npm package called url-metadata that has been serving the community for 10+ years.

During that time some users expressed a preference for a hosted service for various reasons, mostly technical hurdles with coding or accidentally triggering blocks from websites when using the open-source package.

Can I contact Minifetch?

Sure, just use our feedback form and we'll get back to you.