How this crawler behaves, and how to have your site removed.
Requests identify themselves:
MovesWatch/1.0 (+https://moves.watch/crawler)The User-Agent is always truthful. Where a site blocks datacenter TLS fingerprints and
its robots.txt permits the path, the TLS handshake may be made to look like a
browser's. The User-Agent above does not change.
| What the site does | How it is read | What happens |
|---|---|---|
robots.txt disallows the path | A machine-readable refusal | Never fetched. No override exists |
robots.txt is silent | No stated objection | Fetched |
| Datacenter IP or TLS block | A blunt filter, no stated intent | May be worked around, per above |
| 403 with a challenge page | An active refusal | Stops, recorded as blocked |
| CAPTCHA or JS challenge | An explicit refusal | Never solved. Recorded as blocked |
| Login wall | Not public data | Stops, recorded as requiring auth |
| An email from the site owner | A refusal | Removed permanently |
robots.txt and a CAPTCHA state an intent. A fingerprint check
does not, so it is read as a filter rather than as a refusal.
One request at a time per site, at least two seconds apart. Retry-After is
honoured. Three consecutive failures stop the crawl of that page. robots.txt is
re-read every 24 hours. Reads of the Internet Archive are slower still: one request per
second on a single connection, off-peak.
| What | Public | Signed in |
|---|---|---|
| Extracted prices, plans, and features | Yes | Yes |
| The changed lines around an edit (25 at most) | Yes | Yes |
| The full captured page | No | Yes, rate-limited, not indexed |
| Link to your page and to the Internet Archive copy | Yes | Yes |
Prices and plan names are facts, and facts are reportable. Marketing prose on the same page stays yours, which is why the public view carries the changed lines and a link to the original rather than the page itself.
| Request | Response |
|---|---|
| Stop crawling us | Honoured permanently, and recorded |
| Remove a specific capture | The stored page is deleted; the record that it existed, and when, remains |
| Remove all history | Considered case by case. Extracted facts are reportage and are not removed by default |
| Something here is wrong | Re-fetched and re-read, with the correction published alongside the original reading |
Write to crawler@moves.watch. A person answers within five business days.
Careers pages are the only surface that carries personal data in any volume, and the useful signal there is aggregate: how many roles of what kind, where, and when. Names, direct email addresses, phone numbers, and links to individual profiles are stripped before anything is written to disk, so none of them are stored at any point. Careers captures are kept for 24 months.