How does SEO work?

Almost everywhere you look these days, web developers are being told that they have to learn about SEO, Search Engine Optimisation, as the secret of success. But how does it actually work?

How do search engines work?

When you enter a query into Google, the search is forwarded to an index server to see if there is already a result set to match the query. The index server will check with document servers to recover references to the actual sites of interest, bundle up the response and return the results to your browser. This process is very fast because the servers are optimised for the search process.

Thinking of this process backwards then, we want our website to contain the relevant information to produce a link on the document server so that we match the query. The question is, how can we do that most effectively?

One way is to make sure the domain name itself matches the query. If we are looking for nanotechnology links, then owning the domain www.nanotechnology.com will guarantee that we are at the top of the listings - you can try it and see. So clearly the domain name will certainly help. But that's rather obvious. What about the other ways of getting listed?

How do search engines get their data?

The major search engines all use computer programs called web crawlers or spiders to methodically traverse the web collecting data from websites. They are looking for data that will be useful matching the content of the site to the queries from users of their search engines, and they want the most efficient data.

They also want to be sure that the site is actually relevant, contains content that is useful, and that the site is recognised by others as being valuable. How do they do that?

Clearly the keywords that are entered in the search query are very important. If it is a precise product name, then that is wonderfully specific but most users will either not know precisely what they want, or will phrase the query with less detail. They might for example query "industries using nanotechnology".

For the spiders to extract the data, it must be present so one technique used by SEO companies is to identify the most productive keywords for your site's content and then make sure it appears in the webpages.

What data are they after?

Spiders will search for keywords in page names, headings, titles, and text, but also links and image tags. They will pick up the traditional areas for keywords also such as in the page Meta tags and Description though these are of less significance now. In the recent past, some web administrators in the quest for higher ratings, packed these areas with any and every keyword, and also littered their code with them. As a response, the search engines reacted to this attempt to manipulate ratings by banning those sites clearly attempting to skew the results.

But the spiders are also interested in how many links your site receives from outside. These are called inbound links and indicate how many other people think your site is worth reading. The more inbound links you have, the more popular is the site and by implication, the more useful is the content. Spiders therefore treat the number of inbound links as important.

But of course, it would be possible to generate all sort of links from any number of blog comments, forums, social networking sites, and so on. Generating a large number of links is not enough to satisfy the spider that the site is valuable. They also check for the quality of the links so links from sites that are themselves ranked highly, also rank higher than those from low-ranking sites.

There is also a check for the presence in established and reputable web directories such as DMOZ or Lycos, which vet the entries and validate the content before listing. Such a screening process means that a listing on these directories is worth more than an unvetted link from an otherwise unknown site. The spiders take this into account.

Some websites are constructed using content management systems which provide the actual copy for the webpages from a database. The content of the database is not visible to spiders and will therefore not be searched, so for sites built on content management systems (CMS) such as Joomla or Drupal or WordPress, the majority of the site content will be invisible to spiders. Some types of CMS provide facilities for the administrator to make sure that the URL of webpages generated from the database are search engine friendly, for example consisting of words rather than numerical codes, but this is a factor to bear in mind.

As well as the inbound links, outbound and internal links are also scanned so placing keywords in these is also useful.

How do search engines work out ranking?

This is a commercial secret. Regardless of how much inside information the SEO companies might claim, their understanding is based on careful observation and a certain amount of guesswork. Google does not let people know the details of its ranking algorithms so conjecture plays a big part.

The data from the spiders clearly get combined using some sort of formula but no-one outside of the search engine companies knows the precise details. That means no-one can be sure how effective the optimisation with be.

And we should realise that the effects will not be immediate. Search engine indexing takes place over a period of a week or so and therefore any improvement in keyword placement will not affect the rankings for at least a few weeks, and possibly hardly at all.

The bottom line?

SEO is a useful technique for doing your best to make your site friendly towards the indexing operation. Adding keywords might make a difference, possible a large difference if you have previously paid no attention to it. But more important will be the quality of your content. High quality content will encourage others to link to it and both the number and quality of inbound links count.

Not all site structures are completely transparent to spiders but that doesn't necessarily matter.

SEO should be considered an additional tool for web developers but it comes a long way behind ensuring high-quality useful content. There is an enormous amount of hype selling long on promises and delivering short on results. No-one except the search engine companies actually knows the algorithms for calculating ranking so we have to use some common sense and keep it in perspective.

If we run a commercial site, we should calculate the necessary increase in traffic and conversion rate to sales to justify the expenditure on an SEO company. We should also look at their track record to estimate the probability of them being able to deliver what they claim. Finally, we should estimate the benefit of putting that investment into improved content instead. If it still comes out as a benefit, then we might cautiously try the technique, with limited expenditure and trialling the results.

It's not magic. It's not a shortcut to web success. But it is a technique that helps get the site indexed properly which can benefit out sites.