In-Brief: Eric Chan shares his research and views on the state of recommendation. Based on a meticulous study of mobile search on the T-Mobile USA and Verizon portals (powered by white label mobile search provider Medio Systems), Eric concludes recommendation is work-in-progress. But, boy, the industry has a long way to go!

I took some time off yesterday to work on my presentation for Search Goes Mobile, an invitation-only workshop from April 28-30 that brings together senior search company execs, mobile operator representatives and top academics to discuss the challenges and opportunities that lie ahead as we create truly mobile mobile search. (More about this closer to the date.)

I also used the “downtime” from MSG for a long overdue mind-meld with Eric Chan, mobile evangelist, thought-leader, blogger and Adjunct Lecturer in the School of Computer Science at Carnegie Mellon University. Eric also founded Caboodle Networks, a company that developed the know-how and the patents to deliver recommendations based on the user’s context. He later sold the company to mobile search platform provider MCN, a move I reported here.

(I’m also proud to report Eric, who shares the MSG vision, will be a regular contributor to MSG, providing us with his take on content recommendation, key players and offers aimed at content cross-sell and upsell, as well as the delivery of relevant and useful mobile advertising. He has also agreed to collaborate with me and Peggy Albright as we develop a series of mobile search performance reports. I’m honored to have Eric on board and greatly look forward to what we can co-create. )

In January/February 2008 Eric compiled some remarkable data on recommendation, research he agreed today to share with MSG.

Before analyzing the results, which he gathered by testing the ability of the T-Mobile USA and Verizon portals to deliver search results and content recommendations, I wish to make it clear that this is not a judgment of T-Mobile USA/Verizon abilities. It is an objective look at service performance and plea to all companies to push the boundaries of what is possible. In fact, Eric commends both operators (and their search provider Medio) for improving content categorization and successfully tackling the issue of query ambiguity. Good progress, but there’s still a long way to go – for everyone…

The methodology: 100 searches with a primary focus on music (artists, genres, instruments and partial song titles). A special focus on the recommendation feature and how often the service suggested similar content based on an intelligent understanding of the query.

The results: No results for 8 percent of queries. That’s “very high” in Eric’s view. “Absolutely and unequivocally, that should be as close to zero as possible.” In fact, if there is no match, then that’s the optimal time to introduce a content recommendation – such as a song from a similar artist. “If recommendation works, then a service should never have to deliver ‘no results.’”

Formal recommendations offered for percent of queries. It’s a case of close-but-no-cigar, as far as Eric is concerned. This is because the recommendations lacked sophistication and a deeper understanding of what the terms meant. “They are based on matching text strings or word strings, not on semantics.”

The results: The lack of coordination/integration of search is notably high at 23 percent. “There is poor coordination between the Get It Now portal and the content management system.” As a whole, mobile search fails to function in an optimum way and the recommendation feature has clearly not been implemented. More importantly, Verizon literally delivers too much of a good thing, overwhelming users with a long list of results (a whopping 624 in response to the query “Frank Sinatra”.

The takeaway: “We’re still scratching the surface when it comes to recommendation. The companies – the ISVs (independent software vendors) – are still trying to do a lot of indexing and working on getting the search functionality right for the end-users.”

Ironically, the solution to this problem can be found in a comprehensive and strategic approach to recommendation.

Put another way, it can plug the gaps in mobile search because it will always deliver a result – even in cases where the index fails to provide an exact match. “Looking at semantics is a way to help the user and improve the quality of what you offer the user [in response to their query].” It also exposes the user to content they might not have otherwise known existed and could clinch the sale of additional content – all reasons for the industry to sharpen its focus on the technology and get down to serious business.

This entry was posted on Wednesday, April 16th, 2008 at 7:37 pm and is filed under Mobile Research, Mobile Search, On-Device Portals, Recommendation. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.