|
Robots
Robots.txt
More detial or script code example
Disallow: /
disallow robots to read and search all file after the first "/"
often say every thing after your domain or
after your IP
Disallow:
It is empty after "Disallow:" then it means allow
---------- attention -----------
not all the r
阻止搜索引擎漫遊器讀取網站第一個"/"
之後的所有網頁或特定目錄 Description: Robots.txt shall be added at root, no other places
ex: http://----.com/robots.txt
User-Agent:*
Disallow: /
( disallow all robots to search all files)
------------- attention -----------
if you want allow all search engines
then suggest you :
way 1: To delete robots.txt because you do not need a robots.txt
way 2: To keep robots.txt BLANK so no errors fill in error log
when bots come.
Way 3: no robots.txt but create a customized error page
Do not recommend you to use
User-Agent:*
Disallow:
to allow all robots to read your files
Because not all robots follow the same rules
the better way to allow all robots to read your all sites
which is
Way 1: " to delete your robots.txt "
But without robots.txt will create errors when bots come.
Way 2: "to keep robots.txt with nothing in"
so you can avoid errors when bots come.
Way 3: no robots.txt but create a customized error page
Big Tips: read robots line by line each time
Robots.txt 是一個標準檔案，其功能是不讓
Googlebot 從您的 Web 伺服器下載資料。
同時也適用在其他一些主要的搜索引擎漫遊器。 
|
