Adds a Robots.txt file that is configurable from /admin/settings/.
Adds a Robots.txt file that is configurable from /admin/settings/ and injects robots meta tag into all pages.
This module supports single site as well as multisites setups.
Install the module using composer:
composer require innoweb/silverstripe-robots dev-master
Then run dev/build.
On the SiteConfig (or Site is Multisites is installed) there is a setting in the CMS that lets you set the robots mode. The three options are:
The output of all three states is managed through templates and can be overwritten for an app or theme.
When switched to 'allow all' the module uses the template Innoweb/Robots/RobotsController_allow.ss
with the following default content:
<% if $GoogleSitemapURL %>Sitemap: {$GoogleSitemapURL}<% end_if %>
User-agent: *
Disallow: /dev/
Disallow: /admin/
Disallow: /Security/
The module checks whether the Google Sitemaps module is installed and injects the sitemap URL automatically.
It allows access to all pages and disallows access to development and security URLs by default.
When switched to 'disallow all' the module uses the template Innoweb/Robots/RobotsController_disallow.ss
with the following default content:
UserAgent: *
Disallow: /
This disallows all robots from accessing any page on the site.
This setting reveals a text field in the CMS where custom code can be entered.
The template contains the following code and doesn't add anything to the custom code entered:
$RobotsContent.RAW
A good standard robots.txt configuration for Silverstripe looks as follows. This is used as default when the module is switched to 'allow all':
Sitemap: https://www.example.com/sitemap.xml
User-agent: *
Disallow: /dev/
Disallow: /admin/
Disallow: /Security/
The module injects a robots meta tag into every page. The injection of the meta tag can be disabled using the following config, e.g. if the robots meta tag is managed manually in the template:
Page:
robots_enable_metatag: false
By default, all pages are set to index, follow
with the following exceptions:
test
or dev
-1
(see Google Sitemaps module)Additionally, for each page type a config value can be set to control the meta tag. By default, the following values are set:
Page:
robots_noindex: false
robots_nofollow: false
SilverStripe\CMS\Model\VirtualPage:
robots_noindex: true
robots_nofollow: true
SilverStripe\ErrorPage\ErrorPage:
robots_noindex: true
robots_nofollow: true
This can be customised for any custom page types as needed.
BSD 3-Clause License, see License
Module rating system helping users find modules that are well supported. For more on how the rating system works visit Module standards
Score not correct? Let us know there is a problem