Bot Policy

About our crawler

SingURLs operates an automated crawler, identified by the user agent SingURLsBot/1.0 (+https://singurls.com/bot), to discover and monitor domains for our discontinued-product catalog. It fetches each homepage a small number of times to check reachability, and — for sites already reachable — renders the page once with a headless browser to extract structured, factual metadata (title, description, category, technology fingerprints). We discard the raw HTML and rendered DOM immediately after extraction; we never store a mirror or full copy of a crawled page.

What it respects

• We check and honor robots.txt before crawling a homepage.
• We do not crawl any page beyond a site's homepage.
• We do not attempt to bypass authentication, paywalls, or bot-detection challenges.
• We rate-limit our own crawl volume and re-check already-known domains on a fixed, moderate schedule rather than continuously.

How to block SingURLsBot

Add the following to your robots.txt to opt your site out of crawling entirely:

• User-agent: SingURLsBot
• Disallow: /

We check for this before every crawl. If your site is already listed and you'd like it removed regardless of robots.txt, see our Takedown page.

Contact

Questions or concerns about our crawler's behavior: bot@singurls.com.