Site Owner FAQ

How do I verify it's really Minifetch.com?

Real Minifetch.com requests come from the following IP address: 138.68.38.1 AND we identify ourselves with this user-agent: minifetch/1.0 (+https://minifetch.com/site-owner-faq)
User-agents can be spoofed, so the IP address is your best verification.

Does Minifetch respect my robots.txt?

Yes, absolutely. We honor all robots.txt directives including crawl-delay and disallow rules. We follow both wildcard (*) rules that apply to all bots AND any specific rules for our user agent. If you don't want us visiting something on your site, you can either:

- Block all robots (including us):
User-agent: *
Disallow: /my-private-page
- Block only minifetch:
User-agent: minifetch
Disallow: /my-private-page
- To explicitly allow Minifetch while blocking other bots:
User-agent: minifetch
Allow: /

User-agent: *
Disallow: /
- Or to allow us with a crawl delay:
User-agent: minifetch
Crawl-delay: 5
Allow: /
Why allow Minifetch?

• Compliance-first: We check your robots.txt before accessing your content
• Transparent identity: We clearly identify ourselves via user agent
• Cost barrier: Micropayments deter abuse and spam
• Caching: We cache results to minimize repeat requests

We offer a free "Check URL" button (and API endpoint) to see if a URL is accessible or blocked by robots.txt before spending credits to fetch it. We cache that response for a while so our users aren't abusing your robots.txt file, either.

What does Minifetch actually do?

Minifetch provides composable extraction APIs that humans and AI Agents can autonomously discover, orchestrate, and pay for— making web pages simple to access. The API follows ethical scraping practices, respecting robots.txt and using rate limiting to avoid overloading target websites. You can pay-as-you-go with x402 micropayments in USDC, typically for SEO research and AI projects such as AI Agents, RAG pipelines, and building training datasets. There is no Minifetch account setup or monthly fee.

Explore the API →

How is Minifetch different from other scraping services?

• We always respect robots.txt (no sneaking around blocks)
• Rate-limit and cache to minimize server load
• Identify ourselves clearly in user-agent strings
• Use pay-per-request to prevent spam and abuse
• Respond to site owner & user feedback

What are Minifetch's service limitations?

What Minifetch does NOT do:

• Access authenticated or logged-in content
• Create accounts or log into user sessions
• Perform transactional actions (checkout, bidding, purchasing, form submissions)
• Bypass paywalls or access restricted content
• Ignore robots.txt directives

Minifetch only extracts publicly available metadata and content from pages accessible without authentication.

What about "noindex/nofollow" robots meta tags on individual pages?

We don't process those. Here's why:

• Robots meta tags in html pages tell search engines what to index
• We're not a search engine - we're an on-demand fetch service
• We work like a browser visiting a page, not a crawler indexing your site
• We respect robots.txt (crawling rules), but page-level noindex/nofollow meta tags control indexing, which we don't do

Can I monetize my content through Minifetch?

Yes! If you have premium content or APIs, we can work with you to:

• Set custom pricing for your URLs (via x402 protocol)
• Share revenue when users access your content through Minifetch
• Control access levels and pricing tiers

This lets site owners monetize access without managing payment infrastructure. Contact us thru our Feedback form to discuss partnership opportunities.

Is Minifetch open-source?

Minifetch.com is the hosted version of a free and open-source npm package called url-metadata that has been serving the community for 10+ years.

During that time some users expressed a preference for a hosted service for various reasons, mostly technical hurdles with coding or accidentally triggering blocks from websites when using the open-source package.

Can I contact Minifetch?

Sure, just use our feedback form and we'll get back to you.