Steve Fox
Susan Dumais
Abstract
Of growing interest in the area of improving the search experience is the collection of implicit user behavior measures (implicit measures) as indications of user interest and user satisfaction. Rather than having to submit explicit user feedback, which can be costly in time and resources and alter the pattern of use within the search experience, some research has explored the collection of implicit measures as an efficient and useful alternative to collecting explicit measure of interest from users.This research article describes a recent study with two main objectives. The first was to test whether there is an association between explicit ratings of user satisfaction and implicit measures of user interest. The second was to understand what implicit measures were most strongly associated with user satisfaction. The domain of interest was Web search. We developed an instrumented browser to collect a variety of measures of user activity and also to ask for explicit judgments of the relevance of individual pages visited and entire search sessions. The data was collected in a workplace setting to improve the generalizability of the results.Results were analyzed using traditional methods (e.g., Bayesian modeling and decision trees) as well as a new usage behavior pattern analysis (“gene analysis”). We found that there was an association between implicit measures of user activity and the user's explicit satisfaction ratings. The best models for individual pages combined clickthrough, time spent on the search result page, and how a user exited a result or ended a search session (exit type/end action). Behavioral patterns (through the gene analysis) can also be used to predict user satisfaction for search sessions.
- Chickering, D. M. 2002. The WinMine Tookit. Microsoft Research Tech. Rep. MSR-TR-2002-102. Microsoft Research, Redmond, WA. Go online to http://research.microsoft.com/~dmax/WinMine/tooldoc.htm.Google Scholar
- Chickering, D. M., Heckerman, D., and Meek, C. 1997. A Bayesian approach to learning bayesian networks with local structure. In Proceedings of the Thirteenth Conference on Uncertainty in Artificial Intelligence. 80--89. Google Scholar
- Claypool, M., Brown, D., Le, P., and Waseda, M. 2001. Inferring user interest. IEEE Internet Comput. 5, 6 (Nov.-Dec.), 32--39. Google Scholar
- Cooper, G. and Herskovits, E. 1992. A Bayesian method for the induction of probabilistic networks from data. Mach. Learn. 9, 309--347. Google Scholar
- Goecks, J. and Shavlik, J. 1999. Learning users' interests by unobtrusively observing their normal behavior. In Proceedings of the IJCAI Workshop on Machine Learning for Information Filtering. 129--132. Google Scholar
- Heckerman, D., Geiger, D., and Chickering, D. M. 1995. Learning Bayesian networks: The combination of knowledge and statistical data. Mach. Learn. 20. 197--243. Google Scholar
- Horvitz, E., Breese, J., Heckerman, D., Hovel, D., and Rommelse, K. 1998. The Lumiere Project: Bayesian user modeling for inferring the goals and needs of software users. In Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence (July). 256--265. Google Scholar
- Joachims, T. 2002. Optimizing search engines using clickthrough data. In Proceedings of KDD 2004. 133--142. Google Scholar
- Kelly, D. and Teevan, J. 2003. Implicit feedback for inferring user preference: A bibliography. SIGIR For. 37, 2, 18--28. Google Scholar
- Konstan, J., Miller, B., Maltz, D., Herlocker, J., Gordon, L., and Riedl, J. 1997. GroupLens: Applying collaborative filtering to usenet news. Commun. ACM 40, 3, 77--87. Google Scholar
- Morita, M. and Shinoda, Y. 1994. Information filtering based on user behavior analysis and best match text retrieval. In Proceedings of the Seventeenth Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval (July). 272--281. Google Scholar
- Nichols, D. M. 1997. Implicit ratings and filtering. In Proceedings of the Fifth DELOS Workshop on Filtering and Collaborative Filtering (Nov.). 221--228.Google Scholar
- Oard, D. and Kim, J. 1998. Implicit feedback for recommender systems. In Proceedings of the AAAI Workshop on Recommender Systems (July). 81--83.Google Scholar
- Oard, D. W. and Kim, J. 2001. Modeling information content using observable behavior. In Proceedings of the 64th Annual Meeting of the American Society for Information Science and Technology. 38--45.Google Scholar
- Silverstein, C., Henzinger, M., Marais, H., and Moricz, M. 1998. Analysis of a very large AltaVista query log. SRC Tech. Note 1998-014, Compaq Systems Research Center, Palo Alto, CA. Website: http://www.research.compaq.com/SRC/publications.Google Scholar
- Spink, A., Wolfram, D., Jansen, B. J., and Saracevic, T. 2001. Searching the Web: The public and their queries. J. Amer. Soci. Informat. Sci. 52, 3, 226--234. Google Scholar
- Voorhees, E. 2001. Evaluation by highly relevant documents. In Proceedings of the Twenty-Fourth Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval. 74--82. Google Scholar
Index Terms
- Evaluating implicit measures to improve web search
Recommendations
Measuring and Predicting Search Engine Users’ Satisfaction
Search satisfaction is defined as the fulfillment of a user’s information need. Characterizing and predicting the satisfaction of search engine users is vital for improving ranking models, increasing user retention rates, and growing market share. This ...
Implicit Measures as a Useful Tool for Evaluating User Experience
Universal Access in Human-Computer Interaction. Design Methods and User ExperienceAbstractWith the increasing presence in our lives of interactive and technological items, the concept of “experience” becomes crucial in both the design and the evaluation phases. When aiming to measure such experience, researchers typically adopt self-...
Comparative analysis of relevance feedback methods based on two user studies
Rigorous analysis of user interest in web documents is essential for the development of recommender systems. This paper investigates the relationship between the implicit parameters and user explicit rating during their search and reading tasks. The ...
Comments