Daan Odijk

Our SIGIR short paper has been accepted. In this joint work, with MSc student Joost van Doorn, Diederik Roijers and Maarten de Rijke, we propose a multi-objective optimization approach for balancing relevance criteria.

Offline evaluation of information retrieval systems typically focuses on a single effectiveness measure that models the utility for a typical user. Such a measure usually combines a behavior-based rank discount with a notion of document utility that captures the single relevance criterion of topicality. However, for individual users relevance criteria such as credibility, reputability or readability can strongly impact the utility. Furthermore, for different information needs the utility can be a different mixture of these criteria. Because of the focus on single metrics, offline optimization of IR systems do not account for different preferences in balancing relevance criteria.

In this paper, we propose to mitigate this by viewing multiple relevance criteria as objectives and learning a set of rankers that provide different trade-offs w.r.t. these objectives. We model document utility within a gain-based evaluation framework as a weighted combination of relevance criteria. Using the learned set, we are able to make an informed decision based on the values of the rankers and a preference w.r.t. the relevance criteria. We demonstrate on a dataset annotated for readability and a web search dataset annotated for sub-topic relevance, how trade-offs between, e.g., topicality and readability, can be made explicit. We show that there are different available trade-offs between relevance criteria.

Joost van Doorn, Daan Odijk, Diederik Roijers and Maarten de Rijke. Balancing Relevance Criteria through Multi-Objective Optimization. In SIGIR’16, 2016. [BibTeX] [PDF]

Our CIKM2015 paper received the best student paper award! This full paper is the result of my internship at Microsoft Research in Redmond. In this joint work with Ryen White, Ahmed Hassan Awadallah and Susan Dumais, we investigate why some web searchers succeed where others struggle. 

Web searchers sometimes struggle to find relevant information. Struggling leads to frustrating and dissatisfying search experiences, even if searchers ultimately meet their search objectives. Better understanding of search tasks where people struggle is important in improving search systems. We address this important issue using a mixed methods study using large-scale logs, crowd-sourced labeling, and predictive modeling. We analyze anonymized search logs from the Microsoft Bing Web search engine to characterize aspects of struggling searches and better explain the relationship between struggling and search success. To broaden our understanding of the struggling process beyond the behavioral signals in log data, we develop and utilize a crowd-sourced labeling methodology. We collect third-party judgments about why searchers appear to struggle and, if appropriate, where in the search task it became clear to the judges that searches would succeed (i.e., the pivotal query). We use our findings to propose ways in which systems can help searchers reduce struggling. Key components of such support are algorithms that accurately predict the nature of future actions and their anticipated impact on search outcomes. Our findings have implications for the design of search systems that help searchers struggle less and succeed more.

Daan Odijk, Ryen W. White, Ahmed Hassan Awadallah and Susan T. Dumais. Struggling and Success in Web Search.. In CIKM ’15, 2015. [Bibtex] [PDF] [Slides]

Our TPDL2015 full paper was accepted. In this joint work in an Amsterdam Data Science seed project with Cristina Gârbacea (UvA), Thomas Schoegje (VU), Laura Hollink (VU/CWI), Victor de Boer (VU), Kees Ribbens (NIOD) and Jacco van Ossenbruggen (VU/CWI), we present an exploratory search application that highlights different perspectives across collections.

The ever growing number of textual historical collections calls for methods that can meaningfully connect and explore these. Different collections offer different perspectives, expressing the views at the time of writing or even a subjective view of the author. We propose to connect heterogeneous digital collections through the temporal references found in the documents as well as their textual content. We evaluate our approach and find that it works very well on digital-native collections. Digitized collections pose interesting challenges and with improved preprocessing our approach performs sufficiently well. We introduce a novel search interface to explore and analyze the connected collections that highlights different perspectives. In our approach, perspectives are expressed as complex queries. The results of these methods are presented in an open-source interface collections that requires little domain knowledge. Our approach supports humanity scholars in exploring collections in a novel way and allows for digital collections to be more accessible by adding new connections and new means to access the collections.

Daan Odijk, Cristina Gârbacea, Thomas Schoegje, Laura Hollink, Victor de Boer, Kees Ribbens and Jacco van Ossenbruggen. Supporting Exploration of Historical Perspectives across Collections. In TPDL’15, 2015. [Bibtex] [PDF] 

Our SIGIR2015 full-paper got accepted. In this work, we present a query modeling approach to find related content in the setting of the NOS SmartTV app.

While watching television, people increasingly consume additional content related to what they are watching. We consider the task of finding video content related to a live television broadcast for which we leverage the textual stream of subtitles associated with the broadcast. We model this task as a Markov decision process and propose a method that uses reinforcement learning to directly optimize the retrieval effectiveness of queries generated from the stream of subtitles. Our dynamic query modeling approach significantly outperforms state-of-the-art baselines for stationary query modeling and for text-based retrieval in a television setting. In particular we find that carefully weighting terms and decaying these weights based on recency significantly improves effectiveness. Moreover, our method is highly efficient and can be used in a live television setting, i.e., in near real time.

D. Odijk, E. Meij, I. Sijaranamual and M. de Rijke. Dynamic Query Modeling for Related Content Finding. In SIGIR ’15, 2015. [Bibtex] [PDF]

Broadcaster NOS launched a SmartTV app that uses our semantic linking technology. Read more on their blog (in Dutch). 

Together with Laura Hollink (VU), Jacco van Ossebruggen (VU/CWI) and Victor de Boer (VU), we got awarded a Data Science Research Center seed grant for our project "Quantifying Historical Perspectives on WWII". Here's a bit of our plan:

The Second World War is a defining event in our recent history. A huge amount of digital material has become available to study it, ranging from newspapers published during and after the war, books about the war and, more recently, Web pages about people, places and events that played a role. Each source has a different perspective on what happened, depending on the medium, time and location of publication. In this project we aim to quantify these different perspectives. For this purpose, we employ a data science pipeline for selection, structuring, linking and visualization of WOII related material from NIOD, the National Library of the Netherlands, and Wikipedia. With the data and visualization tools we produce, we provide insight into the volume, selection and depth of WOII related topics across different media, times and locations.

For the LinkedUp Challenge Veni competition at the Open Knowledge Conference (OKCon), we (Maria-Hendrike Peetz, David Graus, me,  Ork de Rooij and Maarten de Rijke) created yourHistory; a Facebook app that uses entity linking for personalized historic timeline generation. Our app got shortlisted (top 8 out of 22 submissions) and is in the running for the first prize of 2000 euro! Read a small abstract here:

In history we often study dates and events that have little to do with our own life. We make history tangible by showing historic events that are personal and based on your own interests (your Facebook profile). Often, those events are small-scale and escape history books. By linking personal historic events with global events, we to link your life with global history: writing your own personal history book.

"Automatic Thematic Content Analysis: Finding Frames in News" by Daan Odijk, Björn Burscher, Rens Vliegenthart and Maarten de Rijke has been presented at SocInfo2013. It was awarded the best paper award.

Framing in news is the way in which journalists depict an issue in terms of a `central organizing idea.' Frames can be a perspective on an issue. We explore the automatic classification of four generic news frames: conflict, human interest, economic consequences, and morality. Complex characteristics of messages such as frames have been studied using thematic content analysis. Indicator questions are formulated, which are then manually coded by humans after reading a text and combined into a characterization of the message. We operationalize this as a classification task and, inspired by the way-of-working of media analysts, we propose a two-stage approach, where we first rate a news article using indicator questions for a frame and then use the outcomes to predict whether a frame is present. We approach human accuracy on almost all indicator questions and frames.

“Exploring entity associations over time” by Ridho Reinanda, Daan Odijk and Maarten de Rijke will be presented at TAIA 2013. We address the problem of entity-oriented search in the humanities and social sciences domain. We are particularly interested in retrieving entities related to a query entity and finding associations between these entities over time. Evidence from our target end users suggests that it is more informative to view these associations as dynamic phenomena that evolve over time than as static phenomena. We present work-in-progress on methods to extract these associations and their temporal extent, and discuss a way of presenting them in an exploratory search interface. This interface is intended to help users to discover interesting associations between entities over time. 

“ThemeStreams: Visualizing The Stream Of Themes Discussed In Politics” by Ork de Rooij, Daan Odijk and Maarten de Rijke will be presented at SIGIR 2013. Both the demo paper and the demo itself are available online now. 

The political landscape is fluid. Discussions are always ongoing and new ``hot topics'' continue to appear in the headlines. But what made people start talking about that topic? And who started it? Because of the speed at which discussions sometimes take place this can be difficult to track down. We describe ThemeStreams: a demonstrator that maps political discussions to themes and influencers and illustrate how this mapping is used in an interactive visualization that shows us which themes are being discussed, and that helps us answer the question ``Who put this issue on the map?" in streams of political data.