Policy
Crawler policy
How this crawler behaves, and how to have your site removed.
Requests identify themselves.
MovesWatch/1.0 (+https://moves.watch/crawler)
The User-Agent is always truthful. Where a site blocks datacenter TLS fingerprints and its
robots.txt permits the path, the TLS handshake may be made to look like a browser's.
The User-Agent above does not change.
A refusal and a filter are not the same thing.
| What the site does | How it is read | What happens |
|---|---|---|
robots.txt disallows the path | A machine-readable refusal | Never fetched. No override exists |
robots.txt is silent | No stated objection | Fetched |
| Datacenter IP or TLS block | A blunt filter, no stated intent | May be worked around, per above |
| 403 with a challenge page | An active refusal | Stops, recorded as blocked |
| CAPTCHA or JS challenge | An explicit refusal | Never solved. Recorded as blocked |
| Login wall | Not public data | Stops, recorded as requiring auth |
| An email from the site owner | A refusal | Removed permanently |
robots.txt and a CAPTCHA state an intent. A fingerprint check does not, so it is
read as a filter rather than as a refusal.
One request at a time, at least two seconds apart.
Live pages
One connection per site, two seconds between requests. Retry-After is honoured.
Three consecutive failures stop the crawl of that page. robots.txt is re-read every
24 hours.
Third-party sources
Slower still: one request per second on a single connection, off-peak. Shared infrastructure is not ours to saturate.
Facts are reportable. Your prose stays yours.
| What | Public | Signed in |
|---|---|---|
| Extracted prices, plans, and features | Yes | Yes |
| The changed lines around an edit (25 at most) | Yes | Yes |
| The full captured page | No | Yes, rate-limited, not indexed |
| A link back to your own page | Yes | Yes |
Prices and plan names are facts, and facts are reportable. Marketing prose on the same page stays yours, which is why the public view carries the changed lines and a link to the original rather than the page itself.
Ask, and it is done.
| Request | Response |
|---|---|
| Stop crawling us | Honoured permanently, and recorded |
| Remove a specific capture | The stored page is deleted; the record that it existed, and when, remains |
| Remove all history | Considered case by case. Extracted facts are reportage and are not removed by default |
| Something here is wrong | Re-fetched and re-read, with the correction published alongside the original reading |
A person answers within five business days.
Names never reach the disk.
Careers pages are the only surface that carries personal data in any volume, and the useful signal there is aggregate: how many roles of what kind, where, and when. Names, direct email addresses, phone numbers, and links to individual profiles are stripped before anything is written to disk, so none of them are stored at any point. Careers captures are kept for 24 months.