Robots

Robots.txt

Disallow: /
disallow robots to read and search all file after the first "/"
often say every thing after your domain or
after your IP

Disallow:
It is empty after "Disallow:" then it means allow

---------- attention -----------
not all the r

阻止搜索引擎漫遊器讀取網站第一個"/"
之後的所有網頁或特定目錄

Robots.txt shall be added at root, no other places
ex: http://----.com/robots.txt

User-Agent:*
Disallow: /
( disallow all robots to search all files)
------------- attention -----------

if you want allow all search engines
then suggest you :
way 1: To delete robots.txt because you do not need a robots.txt
way 2: To keep robots.txt BLANK so no errors fill in error log 
when bots come.
Way 3: no robots.txt but create a customized error page

Do not recommend you to use 
User-Agent:*
Disallow:
to allow all robots to read your files

Because not all robots follow the same rules
the better way to allow all robots to read your all sites
which is

Way 1: " to delete your robots.txt "
But without robots.txt will create errors when bots come.

Way 2: "to keep robots.txt with nothing in" 
so you can avoid errors when bots come.

Way 3: no robots.txt but create a customized error page

Big Tips: read robots line by line each time

Robots.txt 是一個標準檔案，其功能是不讓
Googlebot 從您的 Web 伺服器下載資料。
同時也適用在其他一些主要的搜索引擎漫遊器。