Bots, Crawlers and Spiders? Search Engine Robots Explained

Spiders and other creatures are crawling all over your website – but this is not an infestation you’ll want to stamp out.

The leading search engines have indexed billions of web pages, with more being added every day. The search engines use a variety of automated processes to do this. Typically, these are known as “bots,” “spiders” or “crawlers.”

Google has various specialist bots, including their web crawler, Googlebot, their picture spider, Imagebot, and even Googlebot-Mobile for mobile phones. The other top search engines, MSN and Yahoo!, have their own versions of bots and spiders.

Why should online publishers want to know how these robots and spiders crawl all over your website?

A search engine spider is your most important visitor. By allowing the bots easy access to your site and by making it easy for them to rummage around, you will gain extra points -- and more points mean higher search engine rankings.

If you want lots of relevant, qualified traffic from the search engines, then your site must be search engine friendly. Search engine spiders read like machines. They only read the website’s source code -- the HTML code with which the site was constructed.

The spiders only read text and cannot read JavaScript or Flash at all, so fancy animations on website pages are pretty much worthless from a search engine point of view. In addition, bots cannot see pictures (although this is being worked on) so it pays to make sure all images have ALT tags to help the crawlers understand and index the content. Search engine friendliness needs to be incorporated into a site from the initial planning stage.

The easiest way is to check out the search engine friendliness of a website is to use a text-mode based web browser such as Lynx. This will show what a purely text-based version of a site will look like – and what it looks like to the search engines (as an added benefit, it will also ensure that a site is accessible by visually impaired users). The most influential ‘blind’ visitors to your site are the search engine spiders. Do a search on “Lynx Viewer” for more information.

Although the look and experience of a site is important to visitors, website operators must remember that most visitors will come from the search engines -- so they need to make sure their site is accessible and attractive to spiders as well as to people. It’s a balancing act but one that can achieve huge success if done right. In the world of online publishing, great content is the key to keeping both visitors and spiders happy.