Searching The Web – An Exercise in Futility

The internet is a pretty big place and with Facebook throwing its hat in the search ring with their trillion of connections made, it shouldn’t surprise anyone if a search engine doesn’t immediately deliver exactly what you’re looking for with your first search.

penguin2Google is often placed under the microscope when complaints about the web or search quality come up, but it seems exceedingly rare that anyone actually talks about how big a job it is to be a search engine. Using Facebooks example of having an index of a trillion connections made using their social software alone, it should be clear that the web is a huge place. An estimate of the size of the internet is somewhere over 100 trillion web pages and users and complainers are often quick to pass judgement on the search engines when they couldn’t find what they want. Google is the largest and most widely used search engine on the web, still holding onto more than 2/3 of the audience out there and even they don’t even try to get close to curating that massive amount of pages.

When you factor in that many pages on the web and an algorithm that sorts, ranks and tries to properly place every one that it crawls <em>and</em> that it can deliver your results pages in less than a half second it should really be amazing that it can be done at all. Constant updates and improvements to the algorithm that does the bulk of the work can alter the pages you see when you search, and even sometimes appears to completely break the results pages as was the impression when Panda and Penguin were integrated into the algorithm. As an exercise in just how massive an undertaking this can be, and how Google and the other search engines aren’t out to get you and your site specifically give this a go. Imagine you have 100 pennies in your possession all with a different year on them, after shaking them all up in a can pick out the one with the year of your birth on it, if you don’t pick out your year it goes back into the can. You might get it in the first few or it may take you 30 – 40 tries, now repeat that experiment 100,000,000 more times and you’ll have a sample of how much work the search algorithms do every time they perform your search.