Search engine crawlers are increasing my system load

Enter Your Query:
Use '%' for wildcards and quotes for "exact phrases"

Top Level » Apache Related Information

Search engine crawlers are increasing my system loadLast Modified: Oct 30, 2013, 2:40 pm
Since a search engine like google need to parse your website to determine what to search for, if your website has a lot of data, this can often cause a high load on your system if the crawl is done in a short amount of time.

By creating a robots.txt file in your public_html folder, you can instruct these crawlers to slow down.

A sample robots.txt might look like this:

User-agent: *
Crawl-delay: 300

Which tells all crawlers to wait 300 seconds before each request.

Without it, a cralwer might make multiple requests per second, thus increasing your system load.
Related Helpfiles
How to track which site is using the apache processes and causing load

2003 JBMC Software, Suite 173  3-11 Bellerose Drive, St Albert, AB  T8N 1P7  Canada.  Mon-Fri 9AM-5PM MST