Help Centre
Getting started, frequently asked questions, and known limitations for Robotsread.
Getting Started
Robotsread checks a single URL and tells you whether a search crawler can index it. It inspects all four signals that control indexability — robots.txt, meta robots tags, X-Robots-Tag response headers, and canonical links — and resolves them into one verdict with the specific rule that blocks the page, if any.
To check a URL:
- Go to the App page.
- Paste a full URL (including https://) into the input field.
- Click Check.
- The tool returns one of three verdicts: ✓ ALLOWED, ✗ BLOCKED, or ⚠ SOFT BLOCKED, along with a breakdown of each signal.
/api/check which does not exist. No actual URL fetching, robots.txt parsing, or verdict delivery occurs. This Help Centre describes the intended product.
Frequently Asked Questions
What does Robotsread actually check?
Robotsread checks the four signals that search engines use to decide whether a page can be indexed:
- robots.txt — the Disallow and Allow rules at the origin, scoped to the checked URL's path.
- Meta robots tag — an HTML <meta name="robots" content="..."> tag in the page's <head>.
- X-Robots-Tag header — an HTTP response header set by the server (often used for non-HTML content like PDFs).
- Canonical link — a <link rel="canonical"> pointing to a different URL, especially one on a different domain (cross-domain canonical acts as a soft block).
Each signal is evaluated independently, and the tool resolves the combination into a single verdict. If the page is blocked, the blocking rule (e.g., Disallow /wp-admin/ or meta robots content='noindex') is named.
The verdict says "BLOCKED" — what do I fix?
The verdict page shows exactly which rule caused the block. Look at the Signals section to find the blocking signal, then fix the corresponding source:
- Blocked by robots.txt: Edit your robots.txt file to remove or modify the Disallow rule that matches your page's path. Then test using the Google robots.txt testing tool.
- Blocked by a meta robots tag: Remove or change the noindex directive in the page's <head>.
- Blocked by an X-Robots-Tag header: Update your server configuration (for example, in your .htaccess or nginx.conf) to remove the X-Robots-Tag: noindex header for the affected pages.
After making the change, re-check the URL to confirm the fix. Note that search engines may not re-crawl immediately — the verdict reports what the rules currently say, not what Google will do.
What does "SOFT BLOCKED" mean?
A "soft block" means the page is technically accessible to crawlers (not blocked by robots.txt or a noindex directive), but a cross-domain canonical link — a <link rel="canonical"> pointing to a URL on a different domain — tells search engines to treat this page as a duplicate of another site's page. In practice, most search engines will not index the page and will attribute any ranking signals to the canonical target instead.
To fix this, either remove the cross-domain canonical link, or point it to the same page on your own domain (a self-referencing canonical).
Why can't you check pages hosted on Cloudflare?
Robotsread runs on Cloudflare Workers. Due to Cloudflare's own network architecture, a Cloudflare Worker cannot fetch a URL that is also hosted behind Cloudflare — the request loops and is dropped. This is a platform limitation of Cloudflare Workers, not a finding about the URL you submitted.
When you check a Cloudflare-hosted URL, the tool will report a fetch error. We display this caveat next to the URL input on the App page. There is no workaround available on our end.
The checker returned an error — what went wrong?
Errors come from a few possible causes:
- Invalid or malformed URL: Make sure the URL starts with https:// or http://. If you omit the scheme, the tool adds https:// automatically, but a typo in the domain will still fail.
- Non-2xx status code: If the page returns a 404, 403, 500, or any other error, Robotsread reports the status and does not attempt to grade an error page. The verdict will say "COULD NOT FETCH".
- Cloudflare-hosted site: As explained above, Cloudflare Workers cannot reach Cloudflare-hosted origins. You will see a fetch error rather than a verdict.
- Network or DNS failure: If the domain does not resolve or the server is unreachable, the tool reports the failure and marks every derived verdict as not established.
Does Robotsread store the URLs I check?
No. Robotsread does not have user accounts, does not store submitted URLs, and does not use cookies or tracking. Each check is processed in memory during the HTTP request and discarded immediately after the response is sent to your browser. There is no database, no log file, and no storage layer that retains your data. See the Privacy policy for details.
Known Limitations
Backend not yet built
The Robotsread API endpoint (/api/check) does not exist yet. The URL checker interface and verdict display shown on the App page are a frontend demonstration of the intended UX. Submitting a URL will produce an error because no backend processes requests. The core product — fetching a URL, parsing robots.txt, inspecting meta tags, and returning a verdict — is planned but not operational.
Single URL checks only
Robotsread checks one URL at a time. There is no batch checking, no bulk upload, and no API for programmatic access. A Pro tier with unlimited checks, a REST API, and batch URL checking is planned but not yet available.
No scheduled re-checks or monitoring
Robotsread does not offer scheduled re-crawls, alerts, or change monitoring. There is no email sender in this fleet — we cannot notify you if a page's indexability status changes. If you add or remove a blocking rule, you must re-check the URL manually.
Cloudflare Worker limitation
As described above, any URL hosted behind Cloudflare cannot be fetched from a Cloudflare Worker. This is a hard platform constraint with no workaround in the current architecture.
No real payment or subscription infrastructure
The demo checkout form on the Pricing page simulates a subscription for demonstration purposes. No payment processing, user accounts, or subscription management has been set up. The Pro plan ($12/month) is conceptual — it cannot be purchased, and no payment data is collected or stored.
No legal entity registered
This product is operated by an individual who has not yet registered a legal entity (company name, address, jurisdiction). The Privacy policy and Terms of service describe the operational reality of the tool but do not represent a binding legal agreement with a named entity. See the Terms and Privacy pages for details.
Contact Us
Support is not yet reachable by email. This product does not have a mailbox on any domain, and no email sender has been set up. If you write an email address you find on this site, it will not reach anyone.
The owner of this product has not yet established a formal contact method for customer support. Until that is done, here is what is available:
- This Help Centre — covers the most common questions about how the tool works and its current limitations.
- File a bug or feature request — once the owner establishes a contact method (email, issue tracker, or forum), it will be listed here.
Last updated: 6 August 2026