> Main > Products and Services > Purchase > Company
English / Russian

Documentation

CNSearch 1.5.1

© "CN-Software" Ltd. 2002-2008
support@cn-software.com
www.cn-software.com
Previous | Table of Contents  | Next

Structure

The structure and semantics of '/robots.txt" are as follows:

The file must contain one or several records separated by one or several lines (ending with CR, CR/NL, or NL). Each record must contain lines: "<field>:<optional_space><value><optional_space>".

Field <field> is register-independent.

Comments can be included in usual UNIX way: symbol '#' denotes start of a comment, end of line denotes end of a comment.

A record should start with one or more 'User-Agent' lines followed by one or more 'Disallow' lines. Unrecognized lines are ignored.

User-Agent:

Disallow:

If '/robots.txt' is empty, do not correspond to the above-mentioned structure and semantics or is missing, then search robots act according to their settings.


Previous | Table of Contents  | Next