#recommender systems

Running controlled experiments is essential to improve the quality of a online service, where offline performances metrics often do not match those recorded online.  

In traditional A/B testing, we choose two groups of users: one to be exposed to different versions of the service, A (control) and B (treatment). Then we run the test for sufficiently long time, collect the metrics of interest and run statistical tests to confirm that differences in performance are not due to chance. 

The interleaving approach differs from this paradigm. The basic idea behind all variants of the interleaving approach is to perform simultaneous online comparisons of two or more versions. This involves merging the outcome of the different versions into a single interleaved result and then presenting the interleaved result to the user. The logging system observes users behaviour and attribute positive events (e.g. clicks) on a given result to the corresponding source version (click attribution). The main challenge is to make the interleaving process and attribution fair, so that positive events on a interleaved result can be interpreted as unbiased user feedback for a comparison between all considered versions. 

Netflix uses interleaving to test a broad set of ideas (e.g. personalized ranking algorithms) quickly.

To ensure fairness they use team draft interleaving. Considering two ranking algorithms A and B, first they start with a random coin toss that determines whether ranking algorithm A or B contributes the first video. Each algorithm then takes turns contributing the highest ranked video that is not yet in the interleaved list. This process is repeated for each incoming recommendation request.

Their analysis shows that interleaving correlates well with  A/B metrics.

‪Smart personal assistant that is capable of anticipating a user's information needs based on a her check-in records on a location-based social network.