CNSearch 1.5.1
Using "Robots" Meta-tags
Besides the Exclusions standard, there is also a possibility to control search robots behavior by means of 'META' HTML-tag.
Unlike 'robots.txt' files which describes site indexing as a whole, 'META' tag controls indexing of a certain web-page. Besides, it is possible to cancel indexing not only of the document itself, but of the document links as well.
Indexing parameters should be defined in the 'content' field of a source code of each page. The following parameters can be used:
- NOINDEX - cancel document indexing;
- NOFOLLOW - cancel indexing of links contained in the document;
- INDEX - allow document indexing;
- FOLLOW - allow indexing of links contained in the document;
- ALL - equal to INDEX, FOLLOW;
- NONE - equal to NOINDEX, NOFOLLOW.
Default value: <meta name="Robots" content="ALL">.
Note: Values should not be separated by a comma.
Incorrect variant:
<META name="ROBOTS" content="noindex, nofollow">
Correct variant:
<META name="ROBOTS" content="none">
In this example the indexer allows analyzing a document without indexing its links:
<META name="ROBOTS" content="nofollow">
The name of the tag, as well as the names and values of fields are not register sensitive. As a matter of fact, the indexer checks only for 3 values: NOINDEX, NOFOLLOW, and NONE, because FOLLOW and INDEX are default values.
