CiteTheRecordBot

The crawler for Cite the Record, a UK public-interest project that keeps a cited, source-linked record of what Parliament, government and councils publish.

Identify it

User-Agent: CiteTheRecordBot/1.0 (+https://citetherecord.org/bots; crawler@citetherecord.org)

From: crawler@citetherecord.org

To confirm a request is from us, check it comes from 138.68.185.255 (IPv4 only), whose reverse DNS is crawler.citetherecord.org. Machine-readable list: citetherecordbot.json.

nslookup 138.68.185.255
nslookup crawler.citetherecord.org

How it behaves

It reads robots.txt first and obeys it, including Crawl-delay.
One request at a time per site, at least 5 seconds apart.
It stops contacting a site at its first block or challenge, and never tries to get around one.
GET requests only: no logins, forms, cookies or JavaScript.

Block or slow it

User-agent: CiteTheRecordBot
Disallow: /

Or email crawler@citetherecord.org and we'll stop crawling your site.

Report a problem

Email crawler@citetherecord.org with the time of the request and any firewall reference (for example x-azure-ref or cf-ray).