> Main > Products and Services > Purchase > Company
English / Russian


CNSearch 1.5.1

© "CN-Software" Ltd. 2002-2008
Previous | Table of Contents  | Next


The structure and semantics of '/robots.txt" are as follows:

The file must contain one or several records separated by one or several lines (ending with CR, CR/NL, or NL). Each record must contain lines: "<field>:<optional_space><value><optional_space>".

Field <field> is register-independent.

Comments can be included in usual UNIX way: symbol '#' denotes start of a comment, end of line denotes end of a comment.

A record should start with one or more 'User-Agent' lines followed by one or more 'Disallow' lines. Unrecognized lines are ignored.



If '/robots.txt' is empty, do not correspond to the above-mentioned structure and semantics or is missing, then search robots act according to their settings.

Previous | Table of Contents  | Next