Information Overload: Is Wikipedia the new Google?

Around the beginning of the 20th century, there was a clearly identifiable problem with information overload; although there was plenty of information available, it was practically impossible to search through it unless you already knew what you were looking for and where to find it. For example, Medelev's experiments on inherited traits were virtually ignored for decades, until being rediscovered in 1900, some forty years after they were published. This meant that although early studies in genetics were being carried out at around the same time that Darwin was working on his theory of evolution, it wasn't until nearly 60 years later that they were integrated into a common theory.

With the advent of computers and the World Wide Web, it became much easier to share, index and search huge amounts of information. In fact, most of the ideas that modern computers were built around were specifically formulated to deal with the problem of information overload. As a result, it used to be (and still is, for many) that if you wanted to find out something, you'd search for it with Google. Google would search it's index of thousands of websites, and try to guess what was the closest to the one you were looking for; one of the top four or five results would usually take you to a website that specialised in whatever subject you were looking for information about; more often than not, it would be someone's personal, labour-of-love page. No matter how mundane or obscure a subject seems to be, it seems that there's someone out there who is proud to be an expert in it.

Along with the results of its searches, Google also displayed (and still displays) "sponsored links"- websites that had paid to be shown alongside relevant searches. A great strength of this was the clear separation between the search service that Google was providing and its business interests- although Google were selling spaces in its search results, your search results were the closest match possible- not the highest bidding website. This was extended to selling advertising space on websites- website owners would allow Google to put relevant sponsored links on their site and a portion of the advertising fee would go to the website owners.

In extending its business interests outside of its own pages, something changed. Websites that ranked highly in Google's search results and benefited from the traffic that came from the search engine could be more profitable from their Google ads. If sites that made lots of money from their Google ads did well, then Google profited. The line between Google's search results and advertising schemes was starting to become blurred.

There is a massive commercial value in a site that is at the top of a Google search, a considerable amount of money is being spent on search engine marketing (ensuring that websites appear at the top of search engines) and an increasing number of websites which are simply link farms or Splogs are appearing. All of which aim to make you click on their links (which aren't necessarily the ones you are looking for), all serving to effectively dilute the results you get from a search engine.

Meanwhile, the kind of people who make websites for their particular passions tend to choose to spend their time and effort on improving their sites, rather than marketing or working on improving their Google rankings, and as a result will slide down the ranking and off the first page. Information overload is once again a problem.

As a result, I've found that if I'm looking for information- maybe something about a prominent public figure, an event in history or a brand I'm not familiar with, Google isn't my first port of call anymore. Rather than search the entire world wide web (or at least as much of it as Google had indexed), I search Wikipedia. More often than not, I find what I'm looking for.

Do I trust Wikipedia? Well, as much I trust anything I read on the internet, which is to say that I take it with a pinch of saltm, wouldn't invest my money based on what it says, and try to corroborate any information I do rely on with a second source (preferably from someone with an opposing point of view, who would be more likely to argue against it if it were false.) But it's in trying to find the second source online which highlights one of the problems with a Google search- you tend to find that quite often, in looking for something to verify a Wikipedia article, you simply find word-for-word copies of the same article on different websites; sometimes crediting the original Wikipedia article, more often not.

On the other hand, maybe Google will be able to react effectively to the problem and find a solution to separate the 'real' websites from the spam and link farms- I personally believe that if anyone can do it, then it's Google. But without having skilled people checking each website individually (and remember that Google prides itself on being a fully automated search tool), how do you tell the difference between a website built by a fan of a company or product and something built by the company themselves? It seems like a Herculean task at this stage- with over 8 billion pages indexed in September 2005, when they removed the number from their home page, and an estimated hundred trillion words now indexed.

Of course, the idea of Wikipedia taking over from Google is a flawed comparison; after all, it can be easier to find a particular Wikipedia article using Google rather than Wikipedia's own search tool. However, it does seem to me that the days of a search engines results being the starting point in a quest for information may soon be drawing to a close.

Maybe its the start of a new trend; if the increased prominence and popularity of the Wikipedia project brings more understanding and awareness of the Wiki format, maybe we'll see more specialised wikis appearing, devoted to more specialised subjects. Maybe alternative ways of searching for information will become more popular; although they still come under the Google umbrella, the Google Images, Maps and Froogle are all effectively specialised search functions (which specifically search for images, places and prices) offering alternative ways to look for specific information. Perhaps its simply that the world wide web has become too large to condense into a single meaningful index, and a more accurate starting point is needed?

Whatever happens, it looks like the problem of information overload is still going to be with us for a while.