Screen6

There is an alternative to this — Apache MRUnit. It’s a superior approach to running a local cluster, however it still suffers from being extremely low level when compared to Midje-Cascalog or even Cascading’s built-in testing utilities. Usability of MRUnit is also undermined by the fact that most Hadoop jobs are rarely split into composable parts, often having lots of code repetition across similar tasks.

Overall it feels that Hadoop community often treats testing as a second class citizen and this sentiment truly hurts adoption of best testing practices.

Cascalog

A common use case for writing MapReduce jobs is to parse billions of log entries to produce sensible view into patterns of data access, load distribution, etc. It’s also a task we have to solve at Screen6. In fact, it’s such a common problem, that masses of MapReduce frameworks were created in the past years. The most prominent of them is Hadoop, first developed at Yahoo in 2005 and since gradually adopted as the go-to framework for data analysis.

However, it is worth noting that despite being the de-facto data science tool, Hadoop was never considered “easy" when it came to developing. It had and still has a high learning curve, which led to creation of frameworks that try to provide a higher level of abstraction when working with huge arrays of data. One of the most popular of such frameworks is Cascading, which provides a more declarative way of describing your data processing pipeline. Cascalog stands even higher on the abstraction ladder, implementing Datalog, a truly declarative language on top of existing Cascading library. Unlike many other libraries and frameworks that increase the abstraction level while dropping the ability to go one level deeper, Cascalog gives you the ability to actually resort to writing Cascading when required. In our experience that’s rarely needed and in fact our code doesn’t have any “abstraction breaking".

MapReduce with Cascalog

Problem statement

Before we can start with testing, we’ll layout a problem that we can solve with Cascalog. A common task for us at Screen6 is extracting useful information out of log files. It’s neither a unique technical challenge, nor is it inherently exciting, but it’s such a common thing to do that it’s always nice to have a way to minimize the amount of code you need to write to accomplish it.

Since our data pipeline is a quite contrived and just explaining it would take a space required for a few blog posts, we’ll devise a simpler example of such usage. Let’s say you’ve created a map service that has a URI scheme similar to OpenStreetMap.

There are a few valuable pieces of statistics that you can get from analyzing the log files accumulated by such service. For example, you could easily determine which areas of the map get most requests, thus enabling you to tune your caching. If you are using subdomains for faster loading of your tiles from JavaScript you might check if the request distribution is equally spread among such subdomains.

Let’s say your tile server has Nginx serving the tiles and it is running with a slightly customize access log format. Here is a made-up example of a line in this log:

2013-08-01T:07:51:23+0100 123.65.150.10 GET a.tiles.example.com /16/1213/721.png 200 1013 4512 "http://.example.com/mapview" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.472.25 Safari/534.3"

First test

So, we have the following information in our log files: timestamp of request, client’s IP address, HTTP method used, host name, URI, status code, response time in microseconds, size of response in bytes, referer and user agent. Let’s write our test for this exact line. For now we just want to use plain Clojure. We will be also using Midje library, which integrates with Cascalog easily. We can just jump into Clojure REPL and play around with writing our test:

(def line "2013-08-01T07:51:23+0100 123.65.150.10 GET a.tiles.example.com /16/1213/721.png 200 1013 4512 \"http://map.example.com\" \"Mozilla/5.0\"") (fact (parse-line line) => [1375339883 "123.65.150.10" "GET" "a.tiles.example.com" "/16/1213/721.png" 200 1013 4512 "http://map.example.com" "Mozilla/5.0"])

This is plain Midje and plain Clojure, we don’t actually do anything with Cascalog. However, this example will later evolve into Midje-Cascalog based test.

Here’s a short excerpt of the code that would be responsible for parsing the line in a way that satisfies the test:

(ns ... (:require ... [clojure.data.csv :as csv] [clj-time.format :as time.format] [clj-time.core :as time.core] [clj-time.coerce :as time.coerce] ...) ...) (defn set-timestamp [entry] (let [time-format (time.format/formatter "yyyy-MM-dd'T'HH:mm:ssZ")] (->> (entry :timestamp) (time.format/parse time-format) to-unixtime (assoc entry :timestamp)))) (defn parse-line [line] (let [fields [:timestamp :ip :http-method :host :uri :response-code :response-time :response-size :referer :user-agent] entry (first (csv/read-csv line :separator \space))] (-> (zipmap fields entry) set-timestamp ... (mapv fields))))

Midje-Cascalog

This seems simple enough, we’re simply treating each line as a map, run a few transformations on this map and then return a vector of fields in the correct order. Now we can actually write the test against Cascalog code:

(ns ...-test (:require ... [midje.cascalog :refer [produces]])) (fact (parse-logs [[line]]) => (produces [[1375339883 "123.65.150.10" "GET" "a.tiles.example.com" 16 1213 721 200 1013 4512 "http://map.example.com" "Mozilla/5.0"]]))

As we can see, not much has changed. In fact, we had to change very little to accommodate for Cascalog: we pass vector of vectors to the query (parse-logs), the right part of the fact is wrapped in produces call and the response is also a vector of vectors. We can also see that the actual expected output changed a bit, instead of getting URI /16/123/721.png we are actually getting three integers [16 123 721]. Let’s take a look at the corresponding Cascalog query:

(defn parse-logs [source] (<- [?timestamp ?ip ?http-method ?host ?zoom ?x ?y ?response-code ?response-time ?response-size ?referer ?user-agent] (source ?line) (parse-line ?line :> ?timestamp ?ip ?http-method ?host ?uri ?response-code ?response-time ?response-size ?referer ?user-agent) (parse-uri ?uri :> ?zoom ?x ?y)))

Simple query composition

Of course, in the real world you’d have some checks interspersed, but for the purpose of this blog post we’re omitting those. This query can be our building block for future queries. For example, if we want to know during which hours our servers gets most requests, we could write the following query:

(defn most-popular-hours [source] (<- [?hour-of-day ?hits] ((select-fields source ["?timestamp"]) ?timestamp) (get-hour-of-day ?timestamp :> ?hour-of-day) (cascalog.ops/count :> ?hits)))

And now our test “fact" becomes quite simply this:

(fact (most-popular-hours (parse-logs [[line]])) => (produces [[6 1]]))

As can be seen from these examples, writing tests with Midje-Cascalog is quite an enjoyable experience.

Conclusion

The test cases we shown so far have been quite short. In fact, it makes little to no sense to run your tests using only a handful of log entries. More often you’ll extract a few thousand lines of relevant data, make sure there are a few broken entries and test against that. However, writing Cascalog tests doesn’t become a harder task even then, the only part that you need to take care of is loading of resources, your tests however will retain their structure.

Midje-Cascalog is a fantastic tool and is certainly a great selling point when considering Cascalog as an option for handling your MapReduce tasks. And since it’s still Clojure, it’s extremely easy to compose your queries and your facts. In our experience, using Cascalog allows to spend more time on tackling the problem at hand and trying different approaches when compared to writing pure MapReduce jobs using Hadoop. And since Cascalog is still using Hadoop to actually execute the jobs, we’re getting the best of both worlds — interactivity and ease of development with Clojure and stability and scalability of Hadoop platform.

Further reading

Advertisers and TV networks feel challenged by these changes. If people are moving away from linear broadcasting how can they still get their premium quality TV ads across? If people are shifting their attention to their mobile devices when an ad slot starts, how do advertisers continue to reach them?

The first step will be to accept these changes and not try to fight them. The main reason why people are still watching traditional TV is that it is almost the only way to get access to premium live content such as football* matches and talent shows. Why not distribute this content over the Internet (and I don’t mean just the live TV apps that cable companies now often offer to their clients, but through any publisher or app owner that the TV channel wants to work with)? 

Yes, it changes a lot, for example the way we measure GRPs, the way we book our campaigns, and so on. But it will satisfy consumers and it will open up a lot of new opportunities to advertisers and TV channels. 

People will still continue to use the TV (the big screen) for watching premium content, but it will be an integrated experience, connected with everything else that they do online using their phones and tablets.

While most are wondering when the Internet will come to television, I believe the real question is: when will traditional television come to the Internet?

Advertisers and TV channels will benefit because TV programmes broadcasted over the Internet allow advertisers to track their TV campaigns in the same way as they are currently tracking their web, mobile, social and video campaigns, allowing them to optimize campaigns on the fly. It will allow for personalised TV ads, conversion attribution across your TV and adword campaigns, retargeting of users on TV when they’ve abandoned your online shop, and so on…

The more data that’s requested means that the frequency of incoming data will increase rapidly as well.

The reason advertisers alike are in search of more detailed data is because they want to be able to optimize accordingly, preferably during the campaign(s) duration. This is where real-time comes into play, because the sooner you can start optimizing (adjusting), the bigger the gain. Or reversed; the later you gain access to your data, the more money you will potentially lose by spending it the wrong way.

REAL TIME ADVERTISING Besides the obvious advantage of having real-time access to your data, the current trend in online advertising is to buy inventory in real-time. Over 50% of impressions are already being sold through fully automated auctions (Real Time Bidding; RTB).

Usual bidding parameters are set in the ‘auction house’ (or exchange) during initialization of the campaign, with some adjustments made throughout its duration. However, what happens if the system spends most or all of your money - in real-time - in the blink of an eye? What if you are basing your decisions on things happening now when you can only see the figures after 30 minutes, 1 hour or tomorrow?

That’s why we are developing the most advanced real-time reporting platform for RTB campaigns: RTBidControl. It gives clients instantaneous access to what’s happening right now. The platform tells clients what they are spending and with what velocity, but also how much they are earning, and what the best performing elements are.

We acknowledge that nobody has all the time for real time. Therefore we are making the system as smart as possible. Besides the standard metrics we are including Multi Screen figures such as True Reach, GRP and Cross Device conversion attribution. You can check every level, dimension and metric in real-time if required, just like any reporting system, or you can sit back and wait for the smart alert system to notify you in case of rapid changes to your KPI’s. If you are interested, just shoot us an email at [email protected].

However, technology hasn’t quite kept up with this huge shift from single device usage towards a multi-device or multi-screen ecosystem. Most platforms measure cookies and report those as users. As cookies have many drawbacks (e.g. they operate on a single device; see http://goo.gl/hqF2Z) advertisers will overspend on their campaign because they base their strategy on incorrect input.

Detail: every ‘screen’ or device you encounter during a campaign will be accounted for as one unique user by cookies. Quite silly if you consider that one actual user is using multiple devices, isn’t it?

To put it bluntly: advertisers are losing money because technology isn’t prepared for multi-screen advertising campaigns.

This creates a huge gap - correct data versus incorrect data - that we are working on bridging. The first challenge is to ignore cookies and devices altogether and to provide a solution that focuses on users instead. Because, without question, this is what it should be about in advertising.

Only when advertisers can once again ‘see’ who they are interacting with, across all devices, can they start to build and execute an actual multi-screen campaign strategy.

The second challenge is to be correct. Assumptions won’t cut it in our business; only hard figures will increase effectiveness while decreasing spend. We strive to be transparent in everything we do, so even if we aren’t correct, we inform our clients by how much we are off.

The platform that we are building will provide both a real-time reporting front-end and a fully API manageable multi-screen advertising platform.

By offering both sides of the equation we will be able to offer a tool that truly bridges the gap between devices.

The advantages for users are numerous. First and foremost, users don’t have to rely on outdated - cookie based - technologies anymore, but can work with tools that are actually built for the new multi-screen era. New figures will start flowing into the reporting interface(s), such as multi-screen conversion, actual frequencies (unique exposure) and True Reach, which is Screen⁶’s de-duplicated multi-screen reach figure.

Because Screen⁶ can also determine the actual potential reach, we can start reporting and optimizing on online GRP’s. No more guessing required. As advertisers finally come to know how much they are spending for digital ‘eyeballs’ across devices, they can start optimizing accordingly. E.g. by limiting the frequency, optimizing by $ earned, retargeting across devices and including TV within the digital media mix.

Opt-in / opt-out limitations

The most common implementation of a ‘cookie-law’ is to either require an opt-in (users should indicate whether they allow a website to set cookies) or an opt-out (users can unsubscribe from receiving cookies). The biggest threat posed by an opt-in mechanism is that, by default, cookies cannot be set on a computer, which results in very high denial percentages (see cookie refusal in Category 1). Higher rates of denial then again equal a distorted truth when regarding reported figures. Giving users an opt-out won’t influence figures to a great extent. However, because the removal of opt-out cookies is persistent, the denial rate will be somewhere between opt-in and cookie-removal percentages (which result in a factor 5.7 decreased effectiveness). 

Consumer safety regulations

Aside from stricter rules relating to opt-in or opt-out, governments are also pursuing more stringent rules regarding usage of cookies. Cookies are becoming more ubiquitous; marketers want to combine certain information gathered by cookies in databases to increase their effectiveness. However, as some laws forbid the aggregation of consumer related information without active consent, it becomes less viable to increase the value of data collected through cookies. 

Merging multiple cookies with the intention of tracking users is therefore not always possible, and will give less interesting results. 

About Screen6

Screen6 is the multi-screen advertising platform. Our mission: providing the digital media industry with the best multi-screen advertising platform. 

The online and offline advertising markets are changing rapidly. As more and more people are using multiple devices to consume their media, it becomes ever more important for advertisers to reach their audience across these devices as well. As a matter of fact, online advertising is converging with TV advertising as the latter is increasingly available through digital forms.

While innovations in digital media evolve quickly, it becomes more difficult for advertisers to distinguish their reach across multiple devices. Advertisers are losing money because they don't have the tools to combine devices into one singular approach. Just take a moment to think about the number of connected devices you use daily: from a commercial standpoint, wouldn’t it make sense to combine these?

At Screen6 we recognize the need for a platform that makes advertising possible across devices, without any technical hassle. To achieve this we have formed a team of skilled engineers and bright minds who are working hard to build our proprietary and patent-pending Screen6 platform.

Screen6 will be the platform that allows all involved in digital advertising (media agencies, trading desks, advertisers, publishers, data suppliers, etc.) to spend their media budget across multiple screens. It will provide seamless multi-screen advertising and reporting capabilities through its True Reachtm set of tools.

Screen6 is located in the capital of the Netherlands, Amsterdam, where we enjoy a very nice view of the canals. If you are ever in the neighbourhood and would like to drop in for a chat; please let us know! Please also follow us on Twitter to stay up to date: twitter.com/screen6hq

Why this white paper

The reasoning behind this white paper is clear; we are fed up with the ubiquitous usage of cookies throughout online advertising. While we understand that there is a need for some kind of tool, we also strongly believe that cookies are for now the best of the worst. Hopefully, this white paper gives insight to all who are using cookies daily, to understand the caveats they are facing. 

Site optimization issues

One of the most basic usages of cookies in tracking users is to gain insights as to user behaviour on websites. This information is, amongst other things, used for optimising the sites' content, placement, offering, pricing, etc. If the cookies ‘tell you’ that many people left your site at page ABC, you may try to work on a way of countering this. Much money is spent on A/B testing to optimize site performance, including SEO (Search Engine Optimization) and landing page optimization. 

However, what cookies don’t tell you is whether some paths are taken by the same user, but on different devices. Also, they won’t tell you who has revisited your site after removing their cookies. This will cause your data to be merely partially accurate. In this case it’s even more obvious; there is more than meets the eye. If you think a certain path doesn’t work, it’s always better to know. "Prevention is better than cure"!

Conversion / attribution tracking limitations

Conversion tracking is the option of taking one impression or click and linking it to a single conversion - this is extremely important for marketers as it may prove to be the bond between an online campaign and the actual (online) sales. One of the more bitter pills to swallow when using cookies is the fact that they are, by default, not suitable for tracking conversions. 

To explain why, we should understand the different kind of conversions one may track.

Direct post click conversions (PCC); where someone clicks an ad and in the same ‘session’ converts

Direct post impression conversions; where someone sees a banner and, within the same session, converts

Indirect post click / impression conversions; where someone clicks or sees a banner but later on (typically within 30 days), converts

Attribution and Contribution definition

Conversion attribution; rules that determine ‘who’ will get the conversion attributed. E.g., when there are 5 impressions usually the latest will get the conversion

Conversion contribution; rules that determine whether a conversion will be divided amongst one or more previous exposures

All of these conversions (and methods) are heavily reliant on the use of cookies. This means that only exposures and conversions made on the same device/browser combination will be tracked and attributed accordingly. On top of that, cookies may get deleted between user actions and conversions.

However, in a cookie-centric world, any exposure made on a device other than the device used for converting will not be attributed towards the final conversion. Where advertisers are spending huge amounts of money to reach their audience through different channels (TV, online, etc.), they remain in the dark and have no way of truly knowing whether one, does in fact, strengthen the other. 

User value limitations

Campaigns could have basically one (or both) of the following goals: branding and/or performance. For both goals it’s important to be able to track, within a certain level of detail and accuracy, how much was spent on either a unique ‘eyeball’ or to reach a certain goal. As explained above, cookies don’t qualify as the right tool to track all exposures a single user has made during the lifetime of a campaign. This causes cost estimations to be heavily off the mark! 

Interaction tracking

Many current advertising campaigns rely on some form of interaction tracking. Video banners for instance have play/pause buttons - when someone clicks the ‘pause’, this is called an event. Tracking these interactions becomes increasingly important as it shows whether users are involved in the campaign, thus showing increased alertness!

The same rules apply here as in the above subject (Conversion / attribution tracking limitations), where interactions made on TV for instance cannot be used to optimize the campaign on tablets. 

Next week we’ll tell you more about the implications legal regulations have on cookies

Short intro to cookies

Cookies are nothing more than tiny files created (and only readable) by a website, but stored on your local computer. These files cannot be larger than 4 kilobytes and therefore cannot contain much information. A usual cookie contains some form of ID to identify the cookie for later reference, i.e. values such as name, timestamp, order ID, etc., as well as an expiry date. Cookies, on their own, aren’t ‘dangerous’, but aggregating cookies could constitute a privacy issue (will be elaborated upon later).

Cookies reside within the browser; therefore if you simultaneously run two browsers alongside each other (e.g. Chrome and Firefox), they will be unable to share cookies. One website will give you two unique cookies. Cookies also ‘live’ in just one device (PC, iPhone, iPad, …), and cannot ‘transfer’ from one to another. 

Why cookies are important to website owners and advertisers is because it gives them a means of keeping track of who’s been where and what they’ve seen. If a website creates a cookie on your device today and you come back tomorrow with the same device (and browser), the website will relate those two visits. This on a large scale gives insight into visits, visitors and their behaviour. 

Category 1. User measurement

Cookies are most typically used to measure user frequency and occurrences of certain actions on websites - one may refer to this as a ‘Google Analytics’ style of tracking. We’ll explain later why cookies, in fact, aren’t suitable for usage as a tracking tool.

Overview

In short, these are limitations to cookies related to user measurement:

Multiple devices/browsers limitations; each device/browser receives its own separate cookies.

Cookie removals; many people remove their cookies, new unique cookies will be set, which results in an overestimation of factor 5.7

Cookie refusal; besides the above mentioned, some users don’t accept cookies at all, resulting in, yet again, dubious figures.

This means that:

Single users using multiple devices will be, mistakenly, "over-counted".

Frequency capping; limiting the amount of impressions shown to individuals is an important part of advertising - using cookies would overexpose users.

Multiple devices / browser limitations

Not too many years ago, one consumer used just one device (PC or Laptop) to connect to the Internet. Today they have access to multiple devices, e.g. iPhones, iPads, and soon TV. As cookies and their implementation have not evolved over the years, this results in each device (and browser or App) still getting its own separate cookie for each website. 

One user may, for example, use an iPhone to go to website abc.com in the morning, and get a unique abc.com cookie, whereas using a laptop at night to visit the same website would result in yet another unique abc.com cookie! At the end of the day, abc.com will calculate how many unique cookies it encountered that day by simply adding them up. The sum of all these unique cookies will be reported as the unique visitors for that day. This particular user would in fact be counted as two separate users. 

You understand the implications this has: unique visitors equals the sum of all unique devices / browsers. However this couldn’t be more off the mark! If you would zoom in on your household, for example, and add up all the different devices you and your family use daily, this number will, in all likelihood, be higher than the total number of people living in your home. 

Increase in number of devices used within households 1995 - 2015 (est.) showing weakness in Cookie usage. © Screen6 2013

Cookies cannot “side-step” this limitation, as it is built right into their core: each cookie resides in just the one browser and can therefore NOT be used as a tool to report/record the amount of users!

Cookie removals

A second limitation to cookies is that users frequently remove cookies from their system - add to this the facility offered, by default, by most browsers and virus scanners, which remove most cookies right after the browser is closed! It is estimated that almost 33% of first party cookies are deleted within the month, along with a whopping 43% of third party cookies (source: Comscore research ‘Impact of Cookie Deletion on Website Audience Measurement in Australia’).

Besides this, a small percentage of ‘serial cookie deleters’ account for a huge increase in unique cookies; thus almost 50% of cookies are only seen once. Most ad-server systems therefore overestimate their reach by a factor 5.7 while underestimating the frequency to the same degree.  

Cookie refusal

Cookies may be removed with ease, but some users and/or browsers don’t allow storage of cookies; either actively or through a government enforced law. Most notably, Apple’s Safari browser, by default, denies third party cookies. Being unable to place cookies on a computer results, yet again, in less trustworthy unique visitor information.  

Frequency (capping) issues

A ubiquitous feature in online advertising is the possibility to cap (or limit) the amount, or frequency of banners each person is exposed to. After a certain frequency the effectiveness of each following impression decreases. Hence, advertisers would want to strictly limit these frequencies, as overexposure would cost them valuable advertising dollars. 

However, the use of cookies would extremely limit the capabilities of frequency capping. Let’s look at an example. For a certain campaign the frequency cap is set at 5, meaning each individual should have a maximum exposure of 5 impressions. One person uses an iPad, laptop and digital TV at home. For each of these devices unique cookies are created, which will quickly reach the 5 impressions cap. However, this one person has been exposed to 15 impressions by the end of this campaign, instead of the set 5. This means that the advertiser has overexposed one single user, when not being able to use the 10 (overexposed) impressions to reach more users.

Next week we'll tell you more about the implications cookies have on user tracking. 

Continue reading part 2 of 3