#yahoolabs

By Suju Rajan

Data is the lifeblood of research in machine learning. However, access to truly large-scale datasets is a privilege that has been traditionally reserved for machine learning researchers and data scientists working at large companies – and out of reach for most academic researchers.

Research scientists at Yahoo Labs have long enjoyed working on large-scale machine learning problems inspired by consumer-facing products. This has enabled us to advance the thinking in areas such as search ranking, computational advertising, information retrieval, and core machine learning. A key aspect of interest to the external research community has been the application of new algorithms and methodologies to production traffic and to large-scale datasets gathered from real products.

Today, we are proud to announce the public release of the largest-ever machine learning dataset to the research community. The dataset stands at a massive ~110B events (13.5TB uncompressed) of anonymized user-news item interaction data, collected by recording the user-news item interactions of about 20M users from February 2015 to May 2015.

The Yahoo News Feed dataset is a collection based on a sample of anonymized user interactions on the news feeds of several Yahoo properties, including the Yahoo homepage, Yahoo News, Yahoo Sports, Yahoo Finance, Yahoo Movies, and Yahoo Real Estate.

The news feed on Yahoo’s homepage

Our goals are to promote independent research in the fields of large-scale machine learning and recommender systems, and to help level the playing field between industrial and academic research. The dataset is available as part of the Yahoo Labs Webscope data-sharing program, which is a reference library of scientifically-useful datasets comprising anonymized user data for non-commercial use.

In addition to the interaction data, we are providing categorized demographic information (age range, gender, and generalized geographic data) for a subset of the anonymized users. On the item side, we are releasing the title, summary, and key-phrases of the pertinent news article. The interaction data is timestamped with the relevant local time and also contains partial information about the device on which the user accessed the news feeds, which allows for interesting work in contextual recommendation and temporal data mining.

The Personalization Science team at Yahoo Labs has had a ton of fun working on a full-scale version of the Yahoo News Feed dataset, which has sparked some compelling ideas (e.g. Birds, Apps, and Users: Scalable Factorization Machines and Science Powering Product and Personalization: Going Beyond Clicks) in the areas of behavior modeling, recommender systems, large-scale and distributed machine learning, ranking, online algorithms, content modeling, and time-series mining.

We hope that this data release will similarly inspire our fellow researchers, data scientists, and machine learning enthusiasts in academia, and help validate their models on an extensive, “real-world” dataset. We strongly believe that this dataset can become the benchmark for large-scale machine learning and recommender systems, and we look forward to hearing from the community about their applications of our data.

Happy (large-scale) machine learning in 2016!

By Amit Goyal and Hongbo Deng

One of the most important features in a search engine is query auto-completion (QAC). QAC is the first service through which users interact with a search engine to input their search intent. In 2014, global users of Yahoo search saved more than 50% of keystrokes when submitting English queries by selecting QAC suggestions.

QAC provides and updates a query suggestion list based on each new character typed by a user in the search box. The suggestion list is ranked by considering different factors, such as most popular completion (based on historical frequency counts from query logs), time (giving more weight to breaking news or recent popular queries), location, context (based on a user’s previous queries), personalization (based on a user’s profile), and click modeling (based on a user’s past click behaviors).

The aforementioned approaches use only certain relevance features and do not fully take advantage of users’ preferences such as user-QAC interactions. Suppose a user dwells on a suggestion list for a long time without selecting the top-ranked query, it indicates that the user intent might not be satisfied by the provided query suggestions. That wealth of implicit negative feedback has not yet been fully exploited for designing QAC models.

In a research paper to be published in the proceedings of next week’s 38th Annual ACM SIGIR Conference called “adaQAC: Adaptive Query Auto-Completion via Implicit Negative Feedback,” we study implicit negative feedback during user-QAC interactions. Our findings suggest more accurate search results when redesigning QAC to include a more general “(static) relevance-(adaptive) implicit negative feedback” framework.

Motivating Example

Consider a user who wants to query Apple Inc.’s “facetime” with a popularity-based QAC. When the user types “fac,” “facebook” is ranked at the top in the suggestion list because it is most popular in historical query logs. The user dwells for a long time to examine the suggested query “facebook,” but does not select it because it is not “facetime.” With the next keystroke “e,” popularity-based QAC still makes “facebook” top in the list because it is still the most popular query that matches the prefix “face.” Figure 1 depicts our interactions with the Yahoo search engine QAC that depends on relevance scores only. Here the user implicitly expresses negative feedback to the suggestion list by dwelling on it for a long time without selecting any query. Hence, based on such implicit negative feedback, the user may not favor these unselected queries.

Figure 1: Given prefixes "fac" and “face,” popular “facebook”-related queries are returned by Yahoo to users after being ranked by certain relevance scores.

Proposed Approach

In our paper, we propose a novel adaptive model called adaQAC (adaptive QAC) that adapts query auto-completion to take into account users’ implicit negative feedback toward unselected query suggestions. Figure 2 explains the system design and data flow of adaQAC. In general, adaQAC has two steps. In Step 1, the suggestion list of the top-N queries for a given prefix is pre-indexed by static QAC. For instance, for the prefix “face,” suppose the top two queries “facebook” and “facetime” are pre-indexed by static QAC based on historical frequency count. In Step 2, adaQAC re-ranks these top-N queries based on the negative feedback from the user’s previous interactions with the system in the same search session. With a fixed N value (for example, N=10), adaQAC is fast and can be used in real-time.

Figure 2: The system design and data flow of adaQAC

We define a search session as the time from the very first keystroke until the last keystroke in formulating a single query. To address the problem of algorithm design, adaQAC makes use of both the bias information of a query and negative feedback information of a query during a search session. Since negative feedback depends on both the query and session, it can be adaptive in different sessions where different negative feedback can be received even for the same query. Bias information of a query can be computed by any existing static QAC, like ranking queries by historical frequency count. Since bias information of a query can be any existing static QAC, adaQAC is highly flexible.

Negative feedback is composed of a list of signals (such as the dwell time of each keystroke, ranking of the query in the suggestion list, etc.) and the impact weightage associated with these signals. To maximize the probability of the total observations of what users’ final queries are after expressing negative feedback via signals (such as dwell time and position of the query suggestions), the regularized log-likelihood of a softmax function becomes the objective function. Model and signal details can be found in our SIGIR 2015 paper.

Of note, adaQAC performs personalized learning; adaQAC infers different impact weightages of signals for different users because different users have different typing speeds. adaQAC infers impact weightage of signals for each individual user independently and is therefore highly parallel and extremely scalable on large query logs data. We performed a large-scale evaluation of adaQAC and static QAC in five-month Yahoo query logs in the Hadoop framework; adaQAC significantly outperforms static QAC for metrics such as mean reciprocal rank (MRR) and success rate (SR) when evaluating the top suggested search results.

It is in the spirit of collaboration and desire to discover answers to complex problems that we are excited to announce the recipients of the 2015 Yahoo Faculty Research and Engagement Program (FREP) award. This academic outreach initiative is designed to produce the highest quality scientific collaborations and outcomes by engaging with faculty and students conducting research in areas of mutual interest. The FREP awards hundreds of thousands of dollars in unrestricted gifts to support new, exciting Internet research studies and experiments between academics across the globe and their Yahoo Research Scientist counterparts.

Over the course of the next year and beyond, FREP award recipients and Yahoo Labs scientists will work closely to further research in their mutual areas of interest. We were extremely impressed with all of the program submissions and would like to thank each professor who applied. Congratulations to the following recipients of the Yahoo 2015 Faculty Research and Engagement Program:

Chengxiang Zhai

University of Illinois at Urbana-Champaign

Christian Bizer

University of Mannheim

Dafna Shahaf

Hebrew University of Jerusalem

Debmalya Panigrahi

Duke University

Dimitrios Gunopulos

National and Kapodistrian University of Athenss

Dimitry Gorinevsky

Stanford University

Eugene Agichtein

Emory University

Fil Menczer

Indiana University-Bloomington

Giuseppe Carenini

University of British Columbia

Jaime Valls Miro

University of Technology Sydney

Jennifer Preece

University of Maryland, College Park

Jian Pei

Simon Fraser University

Jiebo Luo

University of Rochester

John Canny

University of California, Berkeley

John Shawe-Taylor

University College London

Liang Huang

Oregon State University

Munmun De Choudhury

Georgia Institute of Technology

Oren Kurland

Technion-Israel Institute of Technology

Steven Swanson

University of California, San Diego

Theophilus Benson

Duke University

Tsachy Weissman

Stanford University

Yan Liu

University of Southern California

Yiqun Liu

Tsinghua University

Yvette Wohn

New Jersey Institute of Technology

At Yahoo Labs, we’re committed to forging strong alliances with top faculty by collaborating on cutting-edge research to advance Web Science. These collaborations will solve shared problems with measurable outcomes such as joint papers, advances in algorithm design, systems research, digital media studies, and marketplace design. FREP supports all areas of research at Yahoo Labs.

By Nick Cassimatis

Wouldn’t it be great if it took no knowledge of natural language processing or machine learning, and only a few dozen lines of code, to create apps that understand millions of natural language queries and commands? Now you can do just that thanks to scientific advances (and a lot of hard work) that have enabled us to create the SkyPhrase SDK.

Natural language understanding has recently been playing a greater role in many product categories from search to personal assistants to smartwatches. With intuitive commands and queries as brief as a text message, people are now able to accomplish tasks that were previously impossible or too complicated for traditional interfaces and device types. To date, these benefits have mostly been confined to products made by very large companies since it takes many highly-skilled people and huge amounts of training data to build these interfaces. The only cost-effective alternatives in many cases have been interfaces that understand a handful of very basic commands, but are otherwise very brittle.  

To address this issue, the SkyPhrase (Deep Natural Language Understanding) team at Yahoo Labs has been working on a new scientific framework for natural language understanding based on results from recent linguistics and cognitive science research. Until now, most work in computational linguistics has used grammatical and semantic theories that linguists moved beyond several decades ago. More recent research into human language syntax and semantics contains insights that are very difficult to express using these older theories. Also, much research in cognitive science has been converging on the conclusion that a modest set of concepts underlie language and thought about all topics. Little research in natural language processing has taken these developments into account until now. By rethinking the natural language understanding problem using these insights, we have been able to develop new algorithms that can understand language with great precision, while requiring several orders of magnitude less training data and human effort than ever before.

Last week we were fortunate to have had one of the foremost statistical forecasters in the world, Dr. Rob Hyndman, join us for a Big Thinkers talk on the science of forecasting and prediction. Dr. Hyndman delivered a captivating seminar called, "Exploring the boundaries of predictability: what can we forecast, and when should we give up?"

In his talk, Dr. Hyndman discussed his research efforts to realize his long-term goal of understanding what models work best at predicting various types of time-series. Professor Hyndman demonstrated the aspects that make certain time-series more predictable than others by giving real-life examples based on his own research and work consulting for the Australian government. He explored different features that can be extracted from time-series and how they can be used to judge if something is predictable. Professor Hyndman also talked about his work with Yahoo Labs on Anomaly Detection and how extracted time-series features can be used for finding anomalies. While Dr. Hydnaman's goal of finding the right time-series model for a given dataset is still under active pursuit, his work on anomaly detection in collaboration with Yahoo Labs has already been open-sourced.

The event was broadcast live on our labs.yahoo.com homepage and viewers had the opportunity to ask questions and comment on our Twitter stream @YahooLabs as well as our Facebook page.

By Don McGillen

At Yahoo, we are excited to continue bolstering computer science in academia with our largest cumulative server donation ever. Yesterday, Yahoo Labs’ Academic Partnerships team and the Yahoo Data Center Operations group had the pleasure of delivering 480 servers to the University of Texas at Arlington (UTA). Combined with the 250 servers Yahoo donated to UTA in late 2012, the gift amounts to the most servers Yahoo has donated to any individual institution in our YSTAR (Yahoo Servers to Academic Researchers) program since its inception in 2010. The capabilities provided by the large clusters comprising these refurbished machines have enabled UTA faculty and grad students to perform more robust research with more computing power than ever before.

120 of the previously donated servers are deployed in UTA’s Research Modular Data Center used for doing research related to maximizing the use of airside economization, direct and indirect evaporative cooling, direct-to-chip liquid cooling, sustainable water and energy utilization in the cooling of mission-critical facility environments, and more. The remaining previously donated servers are deployed in the on-campus Yahoo Research Data Center Room and make possible and support a variety of research and educational initiatives.

“Construction of our Research Modular Data Center would not have been possible without the generous donation of servers and PDUs from Yahoo.” - UTA's Dereje Agonafer, Jenkins Garrett Professor of mechanical engineering

Yesterday’s donation exemplifies the growing relationship between Yahoo and UTA, and will significantly upgrade and expand the university’s capabilities to enhance outstanding programs. Those programs will include research involving computationally-intensive modeling in fluid dynamics, data mining, and big data analysis work. The machines will also make possible educational course work in big data analysis and cloud computing.

Through YSTAR, Yahoo has delivered over 5,000 servers to more than 40 academic institutions. We are proud to support research efforts such as those at UTA and look forward to future innovations aided by this donation.

Mark your calendars! Dr. Rob J Hyndman, professor of statistics at the Monash Business School and director of the Monash University Business & Economic Forecasting Unit, is coming to Yahoo–and your screen–on June 26 at 3 p.m. PT.

His talk is entitled: "Exploring the boundaries of predictability: what can we forecast, and when should we give up?"

Watch LIVE on the Yahoo Labs website homepage and ask questions or comment on the Yahoo Labs Facebook and Twitter (#BigThinkers) pages.

ABSTRACT          

Why is it that we can accurately forecast a solar eclipse in 1000 years time, but we have no idea whether Yahoo's stock price will rise or fall tomorrow? Or why can we forecast electricity consumption next week with remarkable precision, but we cannot forecast exchange rate fluctuations in the next hour? In this talk, I will discuss the conditions we need for predictability, how to measure the uncertainty of predictions, and the consequences of thinking we can predict something more accurately than we can. I will draw on my experiences in forecasting Australia's health budget for the next few years, in developing forecasting models for peak electricity demand in 20 years time, and in identifying unpredictable activity on Yahoo's mail servers.

BIOGRAPHICAL NOTE

Rob J Hyndman is Professor of Statistics at the Monash Business School, and Director of the Monash University Business & Economic Forecasting Unit. He is also Editor-in-Chief of the International Journal of Forecasting and a Director of the International Institute of Forecasters. Rob is the author of over 100 research papers in statistical science, and three best-selling books on statistical forecasting. In 2007, he received the Moran medal from the Australian Academy of Science for his contributions to statistical research, especially in the area of statistical forecasting. For 30 years, Rob has maintained an active consulting practice, assisting hundreds of companies and organizations. His recent consulting work has involved forecasting electricity demand, tourism demand, and the Australian government health budget.

YAHOO LABS BIG THINKERS SPEAKER SERIES

 Thumbnail images of the 50 videos in the TVSum50 dataset.

By Yale Song

This year, US adults will spend an average of 1 hour, 16 minutes each day with video on digital devices.1 Yahoo serves over 1.5 billion content videos streams across its various sites,2 and on YouTube over 300 hours of video are uploaded every minute.3 Now more than ever, it is crucial to help viewers use their time wisely and navigate all of their video options thoughtfully. Today, we are excited to help aid in that effort with the release of our new video summarization dataset, TVSum50.

In order to provide a best-in-class video user experience so that our audience can easily browse and find interesting content, the Computer Vision group at Yahoo Labs has been working tirelessly on automatic video summarization, where the goal is to create the gist of a video by extracting highlights and important moments. We believe that well-designed video summaries have a great potential to improve many aspects of a user’s video experience. By allowing users to glance through many videos in a short period of time, we can help them make quicker decisions on what content to watch next so as not to waste their time. To that end, we've developed new algorithms for automatic video summarization and published two research papers appearing in the proceedings of next week’s IEEE Conference on Computer Vision and Pattern Recognition (CVPR) in Boston.

Our papers describe two effective and scalable ways to automatically identify important moments in a video, regardless of video genres, by leveraging the vast amount of images and videos across the Web. Our paper entitled “TVSum: Summarizing Web Videos Using Titles” proposes a system that uses title-based image search results to summarize Web videos. The approach is motivated by the observation that a video title is often carefully chosen to be maximally descriptive of its main topic, and hence images related to the title can serve as a proxy for important visual concepts of the main topic.

Our second paper, entitled “Video Co-summarization: Video Summarization by Co-occurrence,” proposes a system that exploits visual co-occurrence across multiple videos. Motivated by the observation that important visual concepts tend to appear repeatedly across videos of the same topic, the system summarizes a video by finding shots that co-occur most frequently across videos collected from the Web using topical keywords. In each paper, we describe a novel and interesting computational algorithm designed to deal with challenges unique to each scenario, and show their effectiveness by evaluating them on various genres of videos, including our TVSum50 dataset.

The TVSum50 dataset, available to the academic research community via our Webscope data-sharing program, contains 50 videos and their content importance scores annotated via crowdsourcing. The videos with their corresponding scores allow researchers who work in video summarization to rapidly evaluate their own algorithms and to iterate the development process multiple times, without having to conduct extensive user studies each time an evaluation is needed.

 Video 1: Full version of “Will a Cat Eat Dog Food?” by ElPerfecto.com. License: CC-BY 3.0.

 Video 2: Summary version of “Will a Cat Eat Dog Food?” video after applying a version of our TVSum video summarization algorithm.

The 50 videos represent various genres including news, interviews, how-to instructions, documentaries, and user-generated content (e.g., vlog, egocentric). The variety of genres helps ensure the generalizability of summarization techniques in various settings.

The content importance scores included in the dataset are annotated using our carefully designed annotation task user interface, which mimics the conventional video watching experience. The scores were measured at every two-second interval, on a scale of one to five, in order to indicate which part of a video is more important than others, and thus “summary worthy.” We've also incorporated several novel concepts to the task interface so that the resulting annotation is of high quality (meaning, it has a high level of inter-rater reliability).

There are only a few similar benchmark datasets for video summarization available in the community, and most of them are either smaller in scale, biased toward specific video genres (e.g., egocentric), or have a low inter-rater reliability. We hope the release of our TVSum50 dataset will give researchers a new, dynamic tool to evaluate their video summarization algorithms rapidly and with a significant variety of genres to choose from.

1 eMarketer, April 2015: US Adults Spend 5.5 Hours with Video Content Each Day

2 comScore VideoMetrix, April 2015, content video streams only for April 2015