Technical SEO: robots.txt, XML sitemap and ownership

The Technical SEO tab (Baza Helper > Technical SEO) manages the files search engines read: robots.txt and the XML sitemap. Page titles, descriptions, canonical URLs and indexing stay in the page SEO panel of each post.

Technical SEO ownership

Choose the Technical SEO owner. Automatic: theme or detected SEO plugin lets Baza Helper defer to a theme or SEO plugin that already controls these files. Choose another option when you want Baza Helper to own them. When someone else owns technical SEO, the tab says so and the theme’s robots and sitemap changes are skipped.

robots.txt

Switch on Manage virtual robots.txt. WordPress builds robots.txt on the fly, and this setting adds your paths and the sitemap address to it.

• Additional Disallow paths: one path per line, for example /private/. A path typed without the leading slash gets one. The core wp-admin protection and admin-ajax access are always kept.
• Additional Allow paths: one path per line, to allow a path inside a folder you blocked. Do not block CSS, JavaScript or images.
• The file: shows whether the settings have an effect and has an Open robots.txt link.

robots.txt controls crawling, not indexing. To keep a page out of search results use the page’s noindex setting.

Why a change does not appear

• A real robots.txt file exists in the site folder. The web server serves it before WordPress runs, so the settings do not change it. Edit or delete the file to use the settings.
• A host, CDN or browser cache holds an old copy. Clear it after saving.
• Dev / staging mode overrides robots.txt with Disallow: / while it is active.

XML sitemap

Switch on Enable XML sitemap. WordPress generates paginated XML with real modification dates. Private, password-protected, noindex and non-canonical pages are left out, and the visibility setting in Settings > Reading is respected. sitemap.xml redirects to the canonical WordPress sitemap index.

• Post types in the sitemap: choose which content types are listed.
• Include author archives and Include public taxonomy archives add those pages.
• Open sitemap.xml shows the result.

Discouraged search engines

On a live site, if Discourage search engines from indexing this site in Settings > Reading is ticked, Baza Helper shows a warning with a link at the top of the dashboard and of this tab. Only administrators see it.