#yahoo labs

David Carmel, Yahoo Labs, Haifa Eugene Agichtein, Emory University Donna Harman, NIST Dan Pelleg and Yuval Pinter, Yahoo Labs, Haifa

OVERVIEW

We invite you to participate in the 2016 version of LiveQA track, focusing on “live” question answering for real-user questions. Real user questions, sampled from the stream of the most recent questions submitted on the Yahoo Answers site that have not yet been answered by humans, will be sent to participant systems. The systems will provide an answer in real time.

This is the second year of the track. Last year 22 systems participated that were developed by 14 institutions from around the globe, including Australia, China, the Middle East, Europe, and North America. The human questions were collected from a few predefined categories of Yahoo Answers: Health, Sport, Computer & Internet, Pets, etc. We plan to continue using questions from Yahoo Answers, with question categories similar to last year’s.

TASKS

Track coordinators will provide a test system that polls each participant by sending one thousand newly submitted real user questions submitted to the Yahoo Answers (YA) site. The track participants will develop a web service that receives as input a YA question object, and responds with an answer of up to 1000 characters in length, within 1 minute, and include the source(s) of the answer. This is the main required task. This year we are also introducing new optional pilot subtasks on question understanding, for which the systems will categorize the posted question, as well as identify the “essential” parts of the question.

INFRASTRUCTURE

This year we will again provide initial implementation of the server based on NanoHttpd, including a definition of the API to be implemented by participants.

Resources and Data: * YA data is publicly available and many datasets have already been downloaded by the research community and can be reused for this task. Among them is the a large collection of 4,483,032 question and answer pairs provided by Yahoo Labs’ Webscope data sharing program. The specific dataset is L6. Participants are welcome to request this dataset and use it for training. * QRELS from previous system responses from LiveQA 2015 will be made available to active participants, including (potentially) a baseline system implementation. * An API for Yahoo Answers, and a list of resolved questions, representing the categories that we will use during on-line testing phase. Participants can download the suggested questions, with all their metadata (answers, user feedback, etc.), using the provided APIs, and use them for training. *Support for registered participants to experiment and technically validate their answering services against a test client. We will publish the full details of the validation server and the schedule of “dry runs” in the winter of 2016.

EVALUATION

*Runs: During the main evaluation phase, all participant systems should be ready to be invoked by the testing service. At a pre-arranged time, the testing system will start calling the registered systems with the new posted unresolved questions, and will collect the participants’ answers to construct the question-answers pool. Participant systems are allowed to ignore some of the questions (returning an empty element instead of the answer string).

* Metrics: Each question with the corresponding pool of answers will be examined and evaluated by TREC editors. The answers will be judged on a 5-level Likert scale (1— non-relevant and non-useful, 5— perfect answer) based on their relevance and responsiveness to the question. Main metrics are the same as last year, and will include: average answer score, precision (fraction of answered questions with a score above a threshold), and coverage (fraction of all questions answered). We will also experiment with new (pilot) metrics on subset of questions: - Readability: grammaticality, politeness. - Conciseness: penalize by length compared to human answers - Question annotation: type (accuracy), essential question subset detection.

2016 TIMELINE

- February:  Announcement of the list of question categories and filtering criteria to be used in the challenge. YA API to be ready for participants. - February: Skeleton server and answering service API definition released. - March: Testing system is ready. Participants can experiment with the answering service against the testing system. - May 31st:  LiveQA challenge: Participant answering services go live and ready to answer questions for a period of 24 hours.

REGISTRATION

TREC registration deadline is in February 2016. Register through the TREC website: http://trec.nist.gov/pubs/call2016.html. Official registration is required to participate.

In this work (accepted for publication at WWW 2016) we focus on estimating the post-click engagement on native ads by predicting the dwell time on the corresponding ad landing pages. 

Given an ad, characterized by a set of features (e.g. readability, media, mobile optimized, etc.): 

What is the probability that a user will spend (more than) a given amount of time on its corresponding landing page? 

What is the expected value of such dwell time and the overall shape of its distribution? 

What are the features of the ad landing-page that generally lead to high dwell time? 

To address such questions we need a predictive model that is able to infer the relationships between a given set of features and the dwell time, i.e. the length of time that the user spends on the page.

We investigate the application of Survival Random Forests for the task of predicting the distribution of the dwell time on ads. Survival Random Forests is an ensemble tree method for the analysis of right-censored lifetime data which provides us with accurate estimates for:

where

\(\vec{x}(a)\) denotes the collection of features extracted from the landing page of the ad (a). 

DT is a random variable representing the dwell time

\(\varphi\) is an application-specific threshold 

This information is then integrated into the ad ranking function with the goal of promoting the rank of ads that are likely to be clicked and consumed by users (dwell time greater than a given threshold). 

The ad ranking function combines two contributions: clickability (conditioned to the current browsing context c) and post-click engagement score:

The online evaluation over live traffic shows that considering post-click engagement has a consistent positive effect on CTR, decreases the number of bounces and increases the average dwell time, hence leading to a better user experience.

By Suju Rajan

Data is the lifeblood of research in machine learning. However, access to truly large-scale datasets is a privilege that has been traditionally reserved for machine learning researchers and data scientists working at large companies – and out of reach for most academic researchers.

Research scientists at Yahoo Labs have long enjoyed working on large-scale machine learning problems inspired by consumer-facing products. This has enabled us to advance the thinking in areas such as search ranking, computational advertising, information retrieval, and core machine learning. A key aspect of interest to the external research community has been the application of new algorithms and methodologies to production traffic and to large-scale datasets gathered from real products.

Today, we are proud to announce the public release of the largest-ever machine learning dataset to the research community. The dataset stands at a massive ~110B events (13.5TB uncompressed) of anonymized user-news item interaction data, collected by recording the user-news item interactions of about 20M users from February 2015 to May 2015.

The Yahoo News Feed dataset is a collection based on a sample of anonymized user interactions on the news feeds of several Yahoo properties, including the Yahoo homepage, Yahoo News, Yahoo Sports, Yahoo Finance, Yahoo Movies, and Yahoo Real Estate.

The news feed on Yahoo’s homepage

Our goals are to promote independent research in the fields of large-scale machine learning and recommender systems, and to help level the playing field between industrial and academic research. The dataset is available as part of the Yahoo Labs Webscope data-sharing program, which is a reference library of scientifically-useful datasets comprising anonymized user data for non-commercial use.

In addition to the interaction data, we are providing categorized demographic information (age range, gender, and generalized geographic data) for a subset of the anonymized users. On the item side, we are releasing the title, summary, and key-phrases of the pertinent news article. The interaction data is timestamped with the relevant local time and also contains partial information about the device on which the user accessed the news feeds, which allows for interesting work in contextual recommendation and temporal data mining.

The Personalization Science team at Yahoo Labs has had a ton of fun working on a full-scale version of the Yahoo News Feed dataset, which has sparked some compelling ideas (e.g. Birds, Apps, and Users: Scalable Factorization Machines and Science Powering Product and Personalization: Going Beyond Clicks) in the areas of behavior modeling, recommender systems, large-scale and distributed machine learning, ranking, online algorithms, content modeling, and time-series mining.

We hope that this data release will similarly inspire our fellow researchers, data scientists, and machine learning enthusiasts in academia, and help validate their models on an extensive, “real-world” dataset. We strongly believe that this dataset can become the benchmark for large-scale machine learning and recommender systems, and we look forward to hearing from the community about their applications of our data.

Happy (large-scale) machine learning in 2016!

By Jofish Kaye, John Alexis Guerra Gómez, David A. Shamma, and Frank Bentley

When you have thousands of photos, it can be hard to find the photo you’re looking for. Want to search for that Christmas cat you saw at last year’s party? And what if that party wasn’t on Christmas day, but sometime the week before? To help improve the search ranking and relevance of national, personal, and religious holiday photos, we first have to see when the photos were taken; when, for example, is the "Christmas season?"

Understanding what people are looking for when they search for their own photos is an important part of improving the photo-sharing site Flickr. Earlier this year, we began a study (which will be published at CHI 2016 under the same name as this post) by trying to understand how people searched for their personal photos. We showed a group of 74 participants roughly 20 of their own photos on Flickr, and asked them what they’d put into the Flickr search box to find those photos. We did this a total of 1492 times.

It turns out 12% of the time people used a temporal term in searches for their own photos, meaning a word connected to time in some way. These might include a year (2015), a month (January), a season (winter), or a holiday or special event (Thanksgiving, Eid al-Fitr, Easter, Passover, Burning Man). Often, however, the date and time on the photograph didn’t match the search term: the year would be wrong, or people would search for a photograph of snow the weekend after Thanksgiving with the word “winter,” despite the fact that winter doesn’t officially begin until December 21st in the U.S. So we wanted to understand that situation: how often does fall feel like winter?

To answer this, we mapped 78.8 million Flickr photos tagged with a season name to the date the photo was actually taken.

As you’d expect, most of the photographs tagged with a season are taken during that season: 66% of photos tagged “winter” were taken between December 22 and March 20. About 9% of search words are off by two seasons: photos tagged “summer” that were taken between December 21st and March 20th, for example. We expect this may reflect antipodean seasons: while most Flickr users are in the Northern Hemisphere, it doesn’t seem unreasonable that 5% of “summer” photographs might have been taken in the Southern Hemisphere. More interesting, we think, are the off-by-one cases, like fall photographs labeled as "winter," where we believe that the photo represents the experience of winter, regardless of the objective reality of the calendar. For example, if it snows the day after Thanksgiving, it definitely feels like winter.

On the topic of Thanksgiving, let’s look at photographs tagged "thanksgiving."

The six days between November 22nd and 27th—the darkest blue area—cover 65% of the photos. Expanding that range to November 15–30th covers 83%. Expanding to all of November covers 85%, and including October (and thus Canadian Thanksgiving, in gray in early October) brings the total to 90%. But that means that 10% of all photos tagged "thanksgiving" are outside of this range. Every date in that image represents a total of a minimum of 40 photographs taken on that day between 2003 and 2014 inclusive, uploaded to Flickr and tagged "thanksgiving" with the only white spaces being days that don't exist, like February 30th or April 31st. Manual verification of some of the public photos tagged "thanksgiving" on arbitrarily chosen dates finds these photographs tagged "thanksgiving" included pumpkins or turkeys, autumnal leaves or cornucopias—all images culturally associated with the holiday.

Not all temporal search terms are quite so complicated; some holidays are celebrated and photographed on a single day each year, like Canada Day (July 1st) or Boxing Day (December 26th). While these holidays can be easily translated to date queries, other holidays have more complicated temporal patterns. Have a look at these lunar holidays.

There are some events that occur on a lunar calendar like Chinese New Year, Easter, Eid (both al-Fitr and al-Adha), and Hanukkah. These events move around in a regular, algorithmically determinable, but sometimes complicated, way. Most of these holidays tend to oscillate as a leap calculation is added periodically to synchronize the lunar timing to the solar calendar. However Eids, on the Hijri calendar, have no such leap correction, and we see photos tagged "Eid" edge forward year after year.

Some holidays and events, like birthdays, happen on every day of the week. But they’re often celebrated, and thus photographed, on Friday, Saturday, and Sunday:

So to get back to our original question: when are photos tagged "Christmas" actually taken?

As you can see, more photos tagged "Christmas" are taken on December 25th than on any other day (19%). Christmas Eve is a close second, at 12%. If you look at other languages, this difference practically goes away: 9.2% of photos tagged “Noel” are taken on Christmas Eve, and 9.6% are taken on Christmas; “navidad” photos are 11.3% on Christmas Eve and 12.0% on Christmas. But Christmas photos are taken throughout December. We can now set a threshold for a definition of Christmas: say if at least 1% of the photos tagged “Christmas” were taken on that day, we’d rank it more relevant. That means that every day from December 1st to January 1st hits that definition, with December 2nd barely scraping in. That makes…32 days of Christmas!

The architecture of Anthelion

By Petar Ristoski, Peter Mika, Roi Blanco, and Robert Meusel

Would you like to collect large datasets from the Web? If so, then we have good news! Recently we publicly released Anthelion, a focused crawler for semantic annotations in Web pages that steers in the direction of HTML pages–which are annotated with markup languages like RDFa, Microformats, and Microdata–to GitHub.

Anthelion can be targeted to crawl for specific pages; for example, those including markup describing movies with at least two different attributes such as the title of and actors in a movie. The system includes a ready-to-run extension for the Apache Nutch Crawler (nutch-anth), which can be run on a single machine as well as a Hadoop cluster. In addition, the GitHub project includes a testing environment for crawler simulations that makes it possible to measure the efficiency of the crawler in a controlled environment, as demonstrated in our research paper, “Focused Crawling for Structured Data.”

With regard to methodology, the Anthelion system combines the benefits of online learning and a bandit-based selection strategy to adopt to the current crawling environment. Based on a given target function, each newly-discovered URL is classified, where the current crawled page is analyzed with respect to a target function and passed to the learner to further improve its quality. The final selection of what page to crawl next, is done by a bandit-based selection. The bandit chooses between exploration and exploitation (i.e., choosing between the most-promising page or a random page) based on a configuration parameter.

Experiments have shown that in comparison to a pure breadth-first search selection strategy, the number of retrieved relevant pages can be increased by a factor of three. In comparison to a pure online classification-based selection, the improvements are about 26%.

The complete code, which is released under Apache License 2.0, as well as a more comprehensive description can be found at the Yahoo GitHub repository: https://github.com/yahoo/anthelion

Have fun with the code and please let us know if you have any feedback!

By Dan Pelleg, Oleg Rokhlenko, Idan Szpektor, Eugene Agichtein, and Ido Guy

Social media gives voice to the people by allowing them to contribute and annotate content. Specifically, on Community Question Answering (CQA) sites, users provide answers to questions posted by others. The downside of this democratization is the proliferation of low-quality content. This includes:

careless or lazy respondents (“me too,” “you suck,” “just look it up”)

questions/answers from spammers or trolls

questions/answers from users confused by a site’s user interface (say, mistaking the input box for a search box)

To address this issue, the prevailing solution is to rely on the “wisdom of the crowd” to promote good content (e.g., via votes or “like” buttons), or to demote bad content. Unfortunately, such crowd feedback may be:

sparse (not enough people “like” each post to be able to compare)

subjective (I “like” my inarticulate friends; foes, however eloquent, always get a thumbs-down)

slow to accumulate

As an alternative to pure reliance on crowd feedback, we in the Answering Complex Queries group at Yahoo Labs in Haifa developed an automatic quality scoring system detailed in our research paper, “When the Crowd is Not Enough: Improving User Experience with Social Media through Automatic Quality Analysis.” This work, to be published in the proceedings of the upcoming conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2016), describes a system that computes an algorithmic quality score (or AQS) for each answer, making use of syntactic, semantic, and social signals, and a supervised machine learning approach. Moreover, we deployed it on Yahoo Answers to filter and to re-rank answers: when you visit a question page on Yahoo Answers, the highest-quality answers (as determined by our algorithm) are shown on top. Hence, the new default “relevance” filter as seen below.

We then set out to explore the effects of using an AQS system on millions of users. We first focused on basic quality evaluation and compared the best answer as chosen by our system, to the best answer as selected by the question asker. We also compared our system to the winner of the “crowd” sentiment (Yahoo Answers has a voting mechanism based on thumbs up/down). We made some interesting findings:

There was high agreement between the AQS best answer and the best answer chosen by the asker

In cases there was no such agreement, we found a significant difference in favor of our own AQS (as judged by an in-house team of professional annotators who looked at each of the answers and determined their respective quality)

When comparing against the crowd’s votes, we found a substantially lower level of agreement. In fact, here the AQS was significantly better for answers with 20 votes or fewer

It is important to note that most answers get very few votes (if any); Basically, those with up to 20 votes account for 98% of all questions and page views. This implies that: a. until the votes start piling up, it is better to rank using the algorithm, and b. most of the answers users see are in that initial stage before the crowd votes gather meaningful mass.

So far, we–like previous works that look at AQS systems in CQA sites–verified the quality of our AQS, but only in vitro. That is, we showed that the answer, in isolation, makes good sense. But there is a different and possibly more important question: does our AQS have a positive effect on one’s user experience? After all, this change of presentation order, however justified by the content of each and every answer, could, when implemented as a whole, still be making the overall browsing experience worse. In other words, a user might prefer, for his or her own reasons, to be exposed to low-quality content; for example, someone may not care very much about the quality of an answer and comes to the site for the heated arguments as well as the the provocative and trolling answers.

To test this question, we examined various signals of engagement, including answer click-through rate (CTR), average dwell time (time spent on the question page), and maximum scroll depth on a question page (how deeply the user explored the content). Adding metrics other than clicks is important, because it emphasizes true engagement–this point was discussed here before in the post “Going Beyond Clicks”.

We found that the AQS intervention (namely filtering and re-ranking) improved CTR by nearly 10%. We also discovered that both dwell time and scroll depth increased for pages with more high-quality answers as defined by our AQS (see, for example, the results for scroll depth below). Overall, these all provide solid support that AQS does, in fact, improve the overall site experience.

This research provides the first empirical proof that automatic quality scoring not only improves the quality of answer presentation, but also contributes to increased user engagement on CQA sites.

By Erheng Zhong and Suju Rajan

Birds of a feather flock together, or so the saying goes. While some individualists may bristle at the notion of being predictable, mining patterns on reading interests, or app usage, or video watches, or blog follows almost always show us that people who interact with similar items continue to read/use/watch similar items in the future.

Recommendation systems (e.g., collaborative filtering, matrix factorization, factorization machines) tend to exploit the obvious insight that users typically prefer recommendations that come from folks who behave like them. Flavors of recommendation systems have been successfully used in several companies: Netflix, Facebook, YouTube, and Amazon to name a few. However, there are a number of research challenges that need to still be addressed. First, how do recommendation systems work when the lifetime of a content item is limited, as in a breaking news article? Second, how do we make personalized recommendations for each and every user? In Yahoo terms, for the problem of news recommendations this boils down to being able to model hundreds of millions of users against millions of items. Third, how do we develop a methodology that can be applied across several recommendation problems, be it news items, videos, apps, blogs, users, stock tickers, your favorite product, etc.?

Inspired by the above challenges, the Personalization Science team at Yahoo Labs has developed a scalable recommendation system based on the concept of Factorization Machines. To further address the problem of scalability we have implemented a distributed version of the Factorization Machine which leverages the in-house parameter server implementation. Our novel distributed Factorization Machine (DFM) solution has been leveraged in a live production system and easily scales to handle live traffic.

Figure 1: Example of FM cast as a Matrix Factorization problem

Factorization Machines

Factorization Machines is a user response prediction system that models the relevance between different entities. To illustrate the concept, suppose we have collected a set of users’ interactions on items as shown in Figure 1. Each user-item interaction is then encoded into several possible matrix components. For example, user index, item index, user demographic features, item features (such as entities in the item), contextual features (such as time of day, etc.), along with a target rating indicating the user’s response to the item. FM then builds a vector of latent factors for each entity in each component by minimizing a loss function. Casting the raw vectors into a lower dimensional space allows us to directly compute similarities between the latent factors of each component. The aggregated similarities with some bias adjustments allows us to generate personalized recommendations.

There are many applications to which FM can be applied, such as contextual recommendation, topic modeling, and so on. For each use-case, the generalization capability of FM means we only have to design the input data accordingly. However, to apply FM on Yahoo’s products, we need to contend not just with big data, but also with a large model. For example, for news item recommendation on the Yahoo homepage, there are hundreds of millions of users, millions of news articles, and tens of billions of interaction records between users and articles. So we have to partition data to multiple machines. In addition, if we use a vector with 10 dimensions to represent users and articles, the model size is also on the scale of hundreds of GBs, such that it is more efficient to distribute the model storage as well.

Distributed Factorization Machines

In order to scale FM to Yahoo use-cases, we leveraged the Hadoop Map/Reduce (MR) framework and Yahoo's home-grown Parameter Server (PS) technology. The innovative PS has empowered Yahoo's large-scale machine learning algorithms (including logistic regression, word embedding) in production. We utilized MR to implement the data parallelism and let the PS handle the model parallelism. Specifically, in each mapper, we load data from HDFS and partition the data based on an appropriate key. Each reducer, then runs a mini-batch Stochastic Gradient Descent (SGD) optimization method on the data it received. In each model iteration, a pull operation is executed to get the model parameters from the parameter server and a push operation is used to update parameters on servers. Shattering the data to multiple machines and partitioning the models across parameter servers enabled us to scale to not only large data, but also allows us to work with large models.

Evaluation

To test out the performance of our DFM, we used our implementation to build profiles of user interests on news articles. The problem is one of extracting relevant item features, such as phrases in news articles, and assigning them to users based on user-item interaction records. We can then compute the similarity between the users and the items and use the score to recommend relevant items to a user. In this case, we reformatted the input instances to FM by replacing item indices as correspondent feature indices as shown in Figure 2.

Figure 2: Example of FM for User Profile Building

We evaluated our approach using A/B tests on live traffic on the Yahoo home page news feed. In our experimental setup, a random sample of users was partitioned into a control and a treatment group. The users in the control group had their profiles built via the default algorithm in production and the users in the treatment group had their profiles built via DFM. Holding all other features constant, the only difference between the control and treatment group was the computation of the relevance score between the corresponding profile and the content item. The base DFM model was trained over three months of user-item interaction data, after which we did a daily update of the base model using the interaction data for the previous week. It is worthwhile to note that the DFM model was trained in a matter of hours, although it handled on average around 0.5B users, 88 million items, and 6.4B interactions. The results of the experiment are shown in Figure 3. where DFM overall performs better than the baseline on a metric optimized for dwell time on an item. An added benefit was that we were able to expand the profile of our users. As an example, we were able to expand a user’s interest in Chris Smalling to include Manchester United F.C., Premier League, Louis van Gaal, David de Gea, Wayne Rooney, etc.

Figure 3: Online A/B Tests of User Profile Building

By Amit Goyal and Hongbo Deng

One of the most important features in a search engine is query auto-completion (QAC). QAC is the first service through which users interact with a search engine to input their search intent. In 2014, global users of Yahoo search saved more than 50% of keystrokes when submitting English queries by selecting QAC suggestions.

QAC provides and updates a query suggestion list based on each new character typed by a user in the search box. The suggestion list is ranked by considering different factors, such as most popular completion (based on historical frequency counts from query logs), time (giving more weight to breaking news or recent popular queries), location, context (based on a user’s previous queries), personalization (based on a user’s profile), and click modeling (based on a user’s past click behaviors).

The aforementioned approaches use only certain relevance features and do not fully take advantage of users’ preferences such as user-QAC interactions. Suppose a user dwells on a suggestion list for a long time without selecting the top-ranked query, it indicates that the user intent might not be satisfied by the provided query suggestions. That wealth of implicit negative feedback has not yet been fully exploited for designing QAC models.

In a research paper to be published in the proceedings of next week’s 38th Annual ACM SIGIR Conference called “adaQAC: Adaptive Query Auto-Completion via Implicit Negative Feedback,” we study implicit negative feedback during user-QAC interactions. Our findings suggest more accurate search results when redesigning QAC to include a more general “(static) relevance-(adaptive) implicit negative feedback” framework.

Motivating Example

Consider a user who wants to query Apple Inc.’s “facetime” with a popularity-based QAC. When the user types “fac,” “facebook” is ranked at the top in the suggestion list because it is most popular in historical query logs. The user dwells for a long time to examine the suggested query “facebook,” but does not select it because it is not “facetime.” With the next keystroke “e,” popularity-based QAC still makes “facebook” top in the list because it is still the most popular query that matches the prefix “face.” Figure 1 depicts our interactions with the Yahoo search engine QAC that depends on relevance scores only. Here the user implicitly expresses negative feedback to the suggestion list by dwelling on it for a long time without selecting any query. Hence, based on such implicit negative feedback, the user may not favor these unselected queries.

Figure 1: Given prefixes "fac" and “face,” popular “facebook”-related queries are returned by Yahoo to users after being ranked by certain relevance scores.

Proposed Approach

In our paper, we propose a novel adaptive model called adaQAC (adaptive QAC) that adapts query auto-completion to take into account users’ implicit negative feedback toward unselected query suggestions. Figure 2 explains the system design and data flow of adaQAC. In general, adaQAC has two steps. In Step 1, the suggestion list of the top-N queries for a given prefix is pre-indexed by static QAC. For instance, for the prefix “face,” suppose the top two queries “facebook” and “facetime” are pre-indexed by static QAC based on historical frequency count. In Step 2, adaQAC re-ranks these top-N queries based on the negative feedback from the user’s previous interactions with the system in the same search session. With a fixed N value (for example, N=10), adaQAC is fast and can be used in real-time.

Figure 2: The system design and data flow of adaQAC

We define a search session as the time from the very first keystroke until the last keystroke in formulating a single query. To address the problem of algorithm design, adaQAC makes use of both the bias information of a query and negative feedback information of a query during a search session. Since negative feedback depends on both the query and session, it can be adaptive in different sessions where different negative feedback can be received even for the same query. Bias information of a query can be computed by any existing static QAC, like ranking queries by historical frequency count. Since bias information of a query can be any existing static QAC, adaQAC is highly flexible.

Negative feedback is composed of a list of signals (such as the dwell time of each keystroke, ranking of the query in the suggestion list, etc.) and the impact weightage associated with these signals. To maximize the probability of the total observations of what users’ final queries are after expressing negative feedback via signals (such as dwell time and position of the query suggestions), the regularized log-likelihood of a softmax function becomes the objective function. Model and signal details can be found in our SIGIR 2015 paper.

Of note, adaQAC performs personalized learning; adaQAC infers different impact weightages of signals for different users because different users have different typing speeds. adaQAC infers impact weightage of signals for each individual user independently and is therefore highly parallel and extremely scalable on large query logs data. We performed a large-scale evaluation of adaQAC and static QAC in five-month Yahoo query logs in the Hadoop framework; adaQAC significantly outperforms static QAC for metrics such as mean reciprocal rank (MRR) and success rate (SR) when evaluating the top suggested search results.