In-Brief: An insightful and independent assessment of mobile search results in the U.K. (Google vs. Yahoo) by Mobile Commerce shows Yahoo wins hands-down. Result relevance is high, and so are CTRs, on search advertising. No Google-bashing here – but a strong case for big improvements.

Like everyone else, I have been immersed in news around Microsoft’s bid for Yahoo. I’ll circle back with my take – but in the meantime, here are some eye-opening stats that underline Yahoo’s mobile search prowess. Even if Yahoo’s bottom line fails to impress us, this informal test of the most popular search terms leaves me speechless.

How relevant and useful are the results delivered by the two search giants?

Well, this was tested systematically on two mobile phones. Results for 205 of the search terms proven to be most popular, based on the number of users who search for them, were rated on their relevancy to the keyword search term, the ability of the search engine to display the result wanted first (namely a WAP site since we’re on a mobile phone), and the quality of the result (does it work on a mobile phone?) Steve Page, CEO of Mobile Commerce, ran the numbers and Yahoo delivered the goods, providing the most relevant and usable results in 80 percent of the searches.

I asked Steve to put these results into perspective – and tell me what this means for the publishers who struggle to get their content discovered. Are they fighting a losing battle? Here an excerpt (and a couple of my own screenshots that show just how irrelevant search results can be…)

Q: Steve, why are the results significant?

A: It’s meant as constructive criticism and a wake-up call to Google and Yahoo – and all the other search engines – for a very good reason. In addition to the other business areas Mobile Commerce focuses on, we are also a publisher and we want to get discovered. We need Google and Yahoo algorithms to work [in mobile], and these numbers show that’s not the case. I would say the Yahoo algorithm is working pretty well and returning relevant and useful results that I can access on a mobile device the majority of the time.

Q: Both have their shortcomings. I checked, and a search for [the U.K.radio station] Radio One brings up Virgin first on Yahoo, and a search for “cinema” on Google – for some inexplicable reason – brings up the social network MOSH. Will it get worse before it gets better?

A: The results change all the time – however nothing makes me think that it’s going to get better soon. But it goes beyond just not delivering the most relevant results.

Google’s search engine, for example, does crawl our mobile site for a lottery service we offer, but the result doesn’t show up in the search results. Instead, it lists the Internet site for the service first. And, if you input in the search engine that you specifically want our mobile site, it will say that it can’t be found. We’ve optimized the site for mobile, it has been crawled by Google, but it simply doesn’t turn up [in the mobile Web results].

Q: Is it a flaw in the algorithm? Or more linked to the fact that mobile Web sites are like island nations? They are not connected to each other, so PageRank simply doesn’t deliver.

A: That’s part of the problem. The sites are linked – but don’t point to each other – so that part of the algorithm doesn’t work. But there’s also a quality control issue. These search engines don’t appear to have someone who is focused on making sure the results work on mobile or deliver an acceptable user experience.

Q: So we need a human element here? Are you saying that social search or even socially-assisted search has to be part of the mix in order to make sure the search results are quality search results?

A: Yes, because the current model doesn’t pay attention to that. The search engine [execs] say there are results – and that’s very good. But they need to examine them more closely because some are rubbish and can only discourage – not encourage – mobile search usage.

Q: Let’s move to advertising and the actual placement of ads in the search results. Last time we talked statistics we agreed that Yahoo gets 9X the CTRs that Google does. Where are we now?

A: Google’s ads used to be at the bottom [of the results] so users naturally didn’t see them or click on them. This has changed, and in many cases – not all – Google shows the ads at the top. But it took a while for them to change it.

Let me give you a real example. We bid on fifty-five different keywords that are horoscope related on Google, so that would be horoscopes, horoscope, Taurus, Aries, Zodiac, the stars, and lots of misspellings. And for the week of the 14^th to the 20th of January 2008; that gave us 850 page impressions. Google is telling us that our ad has been seen 850 times; so it’s been displayed 850 times. But for those sites that got the ad at the bottom, nobody will ever scroll down and see the ad. So you feel as though 850 people have seen your ad; in reality they haven’t.

Because the ad is at the bottom, we get about a 5 percent click-through rate. So we’ve had 850 impressions and we’ve had 38 – so 4.47 percent – people click on that ad. For Yahoo, it’s about double that.

Q: The ads are higher up on screen in Google than before. How about Yahoo?

A: We bid on Google and Yahoo because that’s where the volume is. The Google [bidding] system has historically been much better than the Yahoo system. However, the new Yahoo system [slated to launch end-February] looks as good as Google. It will be as easy as [Google] and the effect of having a good bidding platform means its makes it easier for advertisers to bid. And this should go a long way toward creating a larger and more robust marketplace.

          

This entry was posted on Friday, February 1st, 2008 at 9:27 pm and is filed under Content Discovery, Mobile Advertising, Mobile Research, Mobile Search. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.