About Robots.txt Generator
Robots.txt is the short form used by SEOs and tech-savvy webmasters to describe the robots exclusion standard. What this means is that the robots.txt instructs the search engine spiders, robots which areas of a website they should not visit. A simple, easy to use robots txt generator can be used to place these instructions in a website.
This standard was proposed in 1994 by Martijn Koster after a web crawler written by Charles Stross played havoc with Martijn’s site. Robots.txt has become the de facto standard which present day web crawlers follow and comply with. However spurious web crawlers that target websites to spread viruses and malware ignore robots.txt and visit the directories of websites that the robots.txt forbids crawlers from visiting. These malicious robots will not only ignore the robots.txt instructions but will visit the pages and directories that are forbidden to visit. That’s, how they spread malware and ruin sites.