CNSearch 1.5.1
Examples
Example 1:
# robots.txt for http://www.site.com
User-Agent: *
# this is an infinite virtual URL space
Disallow: /cyberworld/map/
Disallow: /tmp/ # these will soon disappear
The content of '/cyberworld/map/' and '/tmp/' is protected in this example.
Example 2:
# robots.txt for http://www.site.com
User-Agent: *
# this is an infinite virtual URL space
Disallow: /cyberworld/map/
# Cybermapper knows where to go
User-Agent: cybermapper
Disallow:
In this example, the search robot 'cybermapper' is granted full access, while the rest do not have access to content of '/cyberworld/map/'.
Example 3:
# robots.txt for http://www.site.com
User-Agent: *
Disallow: /
Access to the server is denied to any search robot in this example.
