This module provides simple robots.txt generation for Silverstripe, with various configuration
When a site is not in live mode (such as on a testing domain) it will respectively
block the entire domain, ensuring that (at least respectful) search engines will
refrain from indexing your test site.
This doesn't support complex rules (such as multiple rules for different user agents),
as it's designed to cater to the 99.9% of cases which won't require manual configuration.
composer require "tractorcow/silverstripe-robots": "3.0.*@dev"
sitemap.xmlin your site root.
Basic robots configuration can be found at _config/robots.yml
You can add a page or pattern to be blocked by adding it to the disallowedUrls configuration
--- Name: myrobotsconfiguration After: '#robotsconfiguration' --- Robots: disallowed_urls: - 'mysecretpage.html' - '_private' - 'Documents-and-Settings/Ricky/My-Documents/faxes/sent-faxes'
Also by default, any page with 'ShowInSearch' set to false will also be excluded. This
can be useful for hiding auxilary pages like "thanks for signing up", or error pages.
You can turn this off (if you really absolutely think you need to) using the below
--- Name: myrobotsconfiguration After: '#robotsconfiguration' --- Robots: disallow_unsearchable: false
By default the module will check for a sitemap file in
/sitemap.xml, or will assume
one is there if the googlesitemap module is installed. You can set a custom file location
using the below configuration.
--- Name: myrobotsconfiguration After: '#robotsconfiguration' --- Robots: sitemap: '/sitemap_index.xml'
Message or email me at [email protected] or, well, read the code!
Copyright (c) 2013, Damian Mooyman
All rights reserved.
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Module rating system helping users find modules that are well supported. For more on how the rating system works visit Module standards
Score not correct? Let us know there is a problem