CNSearch 1.5.1
Function
To solve the problem of the indexing robots, you should create a file containing information about robot's behavior control to block robot's request to a web-server or its parts. This file must be located in the root directory.
This solution allows a robot to find algorithms describing its actions by requesting only one file. A file under name '/robots.txt' can be easily created on any existing web-server.
Such file name is chosen due to the following conditions:
- Name of a file must be the same for any operating system;
- File extension should not require any server re-configuration;
- Name of a file should be easy to remember and descriptive;
- Possibility of coincidence with existing files should be minimal.
