Who asks for a missing address: Google bots, AI crawlers and scanners
The 404 list of Baza Helper says who asked for each address that does not exist: Google’s ad bot, Googlebot, other search engines, AI crawlers, SEO services, scanners or an ordinary visitor. You fix first the pages that cost you ads and search traffic, and you can ignore the noise. It is in Baza Helper > Redirects and 404, in the card Missing addresses (404). Redirect rules are described in Redirects and the 404 monitor.
What the list shows
Every row has a Who column with the kind of the visitor and a small mark, in the brand colour where there is a brand. The kind comes from the User-Agent of the request:
- Google (Ads and search): Google Ads bot and Googlebot.
- Other search engines: Bing, Baidu, DuckDuckGo and Applebot.
- AI crawlers: ChatGPT and OpenAI, Claude and Anthropic, Perplexity and other AI crawlers.
- SEO services: crawlers of SEO tools.
- Scanners and probes: tools such as sqlmap or nikto, and any request for a place where leaks and logins are looked for.
- Visitors: a visitor with a browser.
- Other: Facebook and Instagram, uptime and speed checks, other bots and scripts, and requests without a User-Agent.
A request for /.env, wp-config, xmlrpc.php, phpmyadmin or a backup file is shown as a probe whatever name it gives. Rows saved before this feature show Not recorded.
Filter and tiles
Open the filter icon and choose Who to see one group. The tiles above the table show how many requests and how many addresses each kind made in the chosen period, so you see at once whether the noise comes from scanners or from search engines.
Check that a search engine is genuine
A name in the User-Agent can be faked. For Google, Bing, Baidu and Apple the plugin checks the request after the page has been sent: a reverse DNS lookup of the visitor’s address must name a host of that search engine, and a lookup of that host must give the same address back. The row is then marked as verified, or as not genuine when somebody only pretends. A request that is not genuine never raises the alert below.
The alert for Google’s bots
If the Google Ads bot or Googlebot asked for a missing address in the last 7 days, a red alert Google asked for pages that do not exist appears above the table. Each line gives the address, who asked, how many times and when. The arrow button opens the add form with that address, and the bin button forgets the address.
This matters for ads: an ad that points to a missing page can be disapproved with the reason “landing page does not work”, and a search result that points to one can be dropped. Restore the page or add a redirect. Press Mark as seen to hide the alert until Google finds another missing page.
The same problem shows as a red warning item in the WordPress admin bar, so you see it on any screen.
Set it up
Open the card Settings of the Redirects and 404 tab. The options of this feature are:
- Note who asks for a missing address (Google, Bing, AI crawlers, scanners, visitors): on by default.
- Keep the network of the visitor (the address with its last part cut off): off by default. The list then keeps only the network, for example 203.0.113.0, never the full address.
- Warn in the admin bar when Google’s ad bot or Googlebot finds a missing page: on by default.
- Also send the administrator an e-mail (once an hour for one address): off by default.
Fix a missing page
- Open Baza Helper > Redirects and 404 and read the alert.
- If the page should exist, restore it. If it moved, press the arrow button to fill the add form with the old address and enter the new address in To.
- Press Add redirect, then Mark as seen.
