Silverstripe FolderIndex

Silverstripe FolderIndex let you manage visibility of files for Search Engines on parent folder basis, by setting X-Robots-Tag header-tag noindex, nofollow, noimageindex, noarchive, nosnippet. By uncheck ShowInSearch under Details of a folder, headers 'll be sent on all files contained, inclusive files in sub-folders.

The module also adds NoFileIndex() to File. This can be used for checking in xml-sitemap, schema etc. and is used in CMS/Assets to indicate, if a file has X-Robots-Tag headers set per parent Folder or Parent/Parent/etc. If headers are set, it return the Folder-Object that blocks indexing, otherwise false. So far this module just integrates with Apache .htaccess.

Google will stop supporting noindex directive in robots.txt. This module aims to give some control in that area.
https://developers.google.com/search/blog/2019/07/rep-id

Requirements

• silverstripe/asset-admin: ^1.6 (just tested with that)

Installation

Composer is the recommended way installing Silverstripe modules.

composer require lerni/folderindex

Run dev/build

Getting started / Usage

• Uncheck the Indexing child files CheckBox of a folder under Permissions, save and rules in assets/.htaccess 'll be created accordingly.

Credits

Thanks to @digitall-it for inspiration and the Italian translation.

ToDo

• Hook-in to write assets/.htaccess rather than just dev/build?
• roles are falsely also set for sub-folders, which are blocked through parents
• show X-Robots-Tag alert-warning also in file section rather than just in the Details panel?