We’re not talking about our eight legged friends, we talking about the search engine spiders. Those little electronic pieces of clever coding that crawl around the web looking for the information search engines like. You will have also seen them being referred to as the ‘Google bots’ or web crawlers.
Spiders. Love them or loath them, they are here to stay, and they are the army that find the good stuff the search engines want to see. Put simply, a spider’s job is to trawl the web looking for the keywords that people are using, and most often want to see. If you site contains those keywords, the spiders will be your friend, and flag your site as one worth visiting. If you don’t cut it, they’ll simple crawl on by.
The most popular website is checked first, and this is how the master list of keywords is formed. Anywhere there is a link, the spiders will see it. While each of the search engines has their own algorithms and rules for searches, the basic principles are the same for all of them. e poe tegemine
Spiders can see the links, the text and the keywords. That’s it. They can see fancy graphics, or extravagant Flash animations, and they can’t click on links and browse through the menus on a website. When the spiders find your site, the information is stored in a database which forms the basis of the search engine and dictates your ranking. More keywords, more links and more visitors mean a higher ranking. Now we know that’s the simple way of putting it, many of you are experienced with SEO and understand it’s far more complicated than that. Since the advent of Google Penguin, its slightly more complicated than that, however, for this article, and in order to attract the spiders, the simple explanation is enough.
Catching the attention of the Spiders should be your goal, so knowing how to attract them is the key.
Here are some tips on how to make your website more Spider friendly.
Robots.txt File – Spiders are programmed to search for a robots.txt file. All well-built websites should contain this file because it’s the first thing Spiders are programmed to search for. This is the file that waves the initial flag and shouts, “Come over here! Here we are!” If your website does not have this file then you won’t be spotted at all.
Head Section -Once a spider has found you, the next thing they will do is look at your header. This should contain the descriptive title, Meta tags, and your chosen keywords. This is important, and you need to make sure this is done for every page on your site. Use the most descriptive keywords and choose them well. Doing some research will help you identify the right keywords to use for your website and business. Keep the title to less than 60 characters then the Spiders will read it all. Titles that are too long are not read fully. So, titles and headings should be keywords rich, concise and to the point. This is not the time to be clever with words.