Only the addresses it is given
We do not crawl the web or follow links. Our reader opens the addresses a user lists for an audit.
Our crawler
ContentApexBot. Who we are and how our reader behaves.
If you saw ContentApexBot in your logs, someone used Content Apex to audit pages on your site. This page says how our reader introduces itself, how it reads pages and how to reach us.
Identity
How it introduces itself
Every request names our reader. It never pretends to be a person’s browser.
- Product token (use it in robots.txt)
-
ContentApexBot - Reading as a phone (the default)
-
Mozilla/5.0 (Linux; Android 10; K; compatible; ContentApexBot/0.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Mobile Safari/537.36 - Reading as a desktop
-
Mozilla/5.0 (X11; Linux x86_64; compatible; ContentApexBot/0.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36
The Chrome version in these strings is the browser we run; it changes when we update it. Each audit reads all of a site’s pages one way, as a phone or as a desktop, never both.
How it reads
Polite, and only what it is asked to read.
A real browser, the whole page
Each page is opened in a real browser and read from top to bottom, so content that loads as you scroll is read. Endless feeds are cut off and pagination is not followed.
One page at a time
One page at a time per site, across all our users, at least a second apart; the browser loads that page's own files alongside it. A 429 answer or a Retry-After header pauses us for as long as it asks, up to five minutes.
A refusal is final
If your site answers 401 or 403, shows a block page or asks for a human check, we stop and mark the page as blocked. We never retry around it and never try to solve a CAPTCHA.
No stealth
No disguised user agent, no fingerprint changes, no pools of addresses. The browser even reports itself as automated.
robots.txt
We read your robots.txt and report what it says.
Before reading a page we look up your robots.txt and record what it says about that page, for our reader and for Googlebot. Both answers appear in the audit report, because whether Google may crawl a page matters to an SEO audit.
Our reader does not crawl on its own. It opens only the addresses a person has asked it to audit, the way a browser or an SEO desktop tool does, so a robots.txt rule does not by itself stop a requested read today. To address our reader in robots.txt, use the product token ContentApexBot.
Contact
A question or a problem with our reader?
Write to us with your site’s address and, if you can, the time of the requests. A person reads every message.