Wikipedia:Spiders and bots policy

Search engine spiders

A huge portion of our traffic comes from referrals from search engines. Obviously, we want search engines to spider over the site and index us! These spiders are usually well-behaved; they take their time, obey the robots.txt (/robots.txt) file, and don't stress the server too much.

Other spiders

Other well-behaved spiders are also welcome.

But from time to time spiders are not friendly -- they don't obey the restrictions in robots.txt and overuse database-stressing dynamic functions, and/or request pages at so great a rate that they're preventing legitimate human users from getting at the database. If a spider is found to be causing that kind of trouble, the IP address will be banned from access to the site. Currently so banned are:

• 144.167.21.15
• 192.153.22.246
• 202.69.76.19

Bots that make edits to the wiki

Well-behaved bots are acceptable under some circumstances. Please see Wikipedia:Bots.