Simon Bedford

During a discussion with my wife about the Batman vs. Superman film from earlier this year, the question came up as to whether the box office drop from the opening to the 2nd weekend was normal, or whether it dropped more than it should have due to bad word of mouth.

I decided to get hold of some data from the BoxOffice Mojo website to try and answer this question, and it then also evolved into an exercise in data visualization.

The data was all obtained from BoxOfficeMojo using a couple of web scraping scripts. For additional details, as well as known issues, see more at https://github.com/simonb83/DataScienceIntensive/tree/master/projects/box-office

What does success mean

In reality we see that films can be successful in different ways depending on the metric we use.

If we look at box office takings for the opening weekend then, as expected, Sci-Fi, Action and Animated are the genres that typically dominate.

However if we also take budget into account, then it turns out that Horror films are particularly successful.

When we look into why this might be, we see that on average Horror films have a budget one quarter of the size of that for Action, SciFi and Animated films, so it is much easier for them to make their money back.

When we look further into budgets we see that they have increased steadily since the 1980s in both nominal and adjusted terms.

At the same time, the average opening weekend takings seem to have been on a slight downward trend since 2000.

This could be because the total number of films released has almost doubled since 2000, pushing down the box office average.

In terms of total takings, the vast majority of releases make less than $25 million in their opening weekend.

Whilst ‘blockbusters’ continue  to break box office records on a regular basis.

On average, films drop by about 40% in Box Office takings between their opening and second weekends.

That said, there are films that actually increase their box office takings in their second weekend, although these are typically from more obscure genres.

Batman vs Superman

Now we look at the Batman vs Superman film in more detail and compare with two specific films:

Avengers: Civil War

Batman: Dark Knight Rises

We also use a general group consisting of 89 Sci-Fi and Action/Adventure films, with budget > 100 million USD, released since 2010.Batman vs. Superman performed very well in its opening weekend.

It was also within the top 10 films with a highest grossing opening weekend based on our comparison group:

However, by the second weekend it did a lot worse than its peers, dropping nearly 70% vs. an average drop of 50% for similar films.

Finally, we look at a couple of very simple scenarios for how well BvS could have performed in its second weekend, had it not dropped more than average vs. its peer group.

We find that it could have made an additional $12-30 million in its second weekend had it performed similar to comparative films.

During a recent personal project I was working with Deep Learning models, and in particular trying to fine tune a CNN (AlexNet) in Caffe. At first I was getting some pretty strange, and frustrating, learning curves.

It turned out that my error was that I was that I was running a script to augment my images prior to training, but forgetting to randomize the resulting dataset.

Luckily, once I corrected this, everything started nicely converging.

This is a brief summary of my first experience with a Kaggle competition. I entered for the learning experience, and because I wanted to play around with a real problem in machine learning.

The Problem: Crime In San Francisco

I chose the Crime Classification in San Francisco as it seemed like a fairly well-defined problem, and it is also in the Playground section of Kaggle so is probably more of a gentle introduction.

You are given a training data set of 878,049 crimes from the past 12 years, each with the following information:

Date Crime Occurred

Day of Week

Police District

Street Address

X and Y coordinates

Type of Crime

Description

Resolution

The challenge is to develop an algorithm which can then correctly classify other crimes using only the date and location features mentioned above.

Given a test data set of another 884,262 crimes, for each instance the algorithm needs to assign probabilities to 39 different crime categories.

The output is evaluated using the Multi-Class Logarithmic Loss function (more in another post), where each predicted probability is compared to the true probability. The idea is to make the loss function as small as possible.

Submission 1

Being very eager to jump in and try something out, I very naively threw something together using the Python Sklearn Random Forest algorithm (chosen as I had read that it is one of the algorithms that most frequently wins Kaggle competitions), with the following features:

Time of day (based on number of second in the day)

X & Y coordinates

Results:

Score of 26.85, ranked 1,150 on leaderboard.

Submission 2

My second submission was the result of randomly playing around. I cut down the features to only:

Police precinct

Time of Day

However I think the biggest improvement was using the predict_proba instead of predict method. Whereas predict gives effectively a binary classification for the most likely class (i.e. 1 or 0), predict_proba assigns a probability to every single class.

It turns out that Logarithmic Loss penalizes confident but wrong answers very heavily, so it is better to “spread your bets” as it were rather than predicting a single category.

Results:

Big improvement to score of 4.9, moved up 172 places.

Submission 3

Around this point, I started to test locally prior to submitting by splitting the train data set approximately 80%/20% into training and test data in order to better measure whether my changes were improving my score or not.

For the third submission, I used only two categorical features:

Hour of day

Police precinct

However the biggest improvement came from specifying a max depth of 10 for the Random Forest.

Results:

Improved score to 2.59, moved up 426 places.

Submission 4

Submission 4 was when I realized that pretty much all of my success up until this point had been beginner’s luck, and I needed a more systematic approach. So, after reading quite a bit about feature engineering, I used the following variables:

Precinct

Day of Week

Time of Day

X and Y (scaled to 0,1)

For Precinct, Day of Week and Time of Day I finally began treating them as categorical variables (using pandas get_dummies).

Results:

Improved score to 2.49, moved up 267 places.

Submission 5

For submission 5 I got some inspiration from comments on the competition forum (thanks papadopc and SatendraKumar).

In particular I did additional feature engineering on the X and Y coordinates to:

Rotate coordinate frames through 30, 45, 60 and 90 degrees

Convert to polar coordinates and calculate R for each point

I was also inspired by a comment from papdopc to create a crime count per address, that is to say, using the training data:

Get a list of unique street names

For each street name count the instances of each crime category

Use these counts as new categorical variables

(Note: I have already seen some issues with this approach so will revisit later on).

Results:

Improved score to 2.44, and moved up to current position of 277 on leaderboard.

Submission 6

It would only be honest to note that I have made an additional submission with an approach pretty similar to Submission 5, but filtering for outliers in the training data, however this did not improve on my current best score.

Summary and Next steps

So far it has been a really fun and interesting experience and it is pretty cool that there are so many tools out there that enable you to make headway so quickly with complex problems.

In some respects this can also be a disadvantage because it also makes it very easy to implement algorithms when you have very little idea of what you are doing, or how they work. However, on balance, I think it is also very useful not to have to get bogged down in the detail of creating efficient implementations of algorithms and enables far faster learning of some difficult concepts.

I am already working on my newest model including:

Some new categorical features: Season of Year, Daylight Saving Time Indicator, Whether the address is an Intersection or not

Smarter featurization of the address data vs. crime counts (based on papdopc solution)

Some other approaches I want to try include:

Using a Neural Network

Combining predictions from multiple algorithms

I am currently working on a particularly complex project involving many moving parts, at least three different companies all with their own internal requirements, and which is a mixture of IT development and process migration and improvement.

In short, there is a lot that can go wrong and pretty much every day we have a new surprise to deal with.

So the other evening, while I was re-reading a chapter of Nate Silver’s excellent The Signal And The Noise, it occurred to me that there are many aspects of project management that should really be treated more explicitly as a formal prediction or forecast.

Take timeline estimation for example. Whereas informally we often try and make a judgment about how a new or changed requirement could impact the go-live date, I think our planning would benefit significantly by being more formal in our treatment of time (or cost) estimates.

In practice this means two things:

Thinking and talking in terms of probabilities and ranges

Constantly updating our estimates based upon new information

Living with Uncertainty

For the first point, current practices are very focused on using fixed estimates, often leading to conflict and recrimination when the actual figure turns out to be different from the initial estimate.

Indeed when you look at a traditional Gant chart or other visualization of a project plan, all of the activities are typically set out in a very linear and causal fashion, all building up to a fixed go-live date.

Reality, however, is very different. Even for smaller individual tasks, when someone says that it will take 5 days to complete activity X, in practice it could take 5 days, however it could also take 4 or maybe 6 or more, depending on the activity. People try and round to whole numbers but there is often an implicit probability distribution built into their estimates.

Each of these individual estimates + error then compound at the work-stream and project level, so a fixed implementation timeline of two months, would be far more usefully expressed as something like:

10% probability of 7 weeks (if everything goes extraordinarily well)

70% probability of 8 weeks

10% probability of 9 weeks

10% probability of 10 weeks or more

I know that many people, particularly project owners / sponsors, would not be very comfortable with these kinds of estimates as in general people are not comfortable working with probabilities and uncertainty. Often people mistake expressions of uncertainty for lack of competence and will try and push for a “definite answer”.

Unfortunately the real world is full of uncertainty, and ignoring it or pretending it isn’t there doesn’t make it disappear. In fact, pretending that we can forecast the future with absolute certainty leads to overconfidence and leaves us less prepared to adapt to changing circumstances.

Institutional Statistics

What does this mean in practice? The probability estimates are not just numbers based upon the gut feel of whomever is responsible for that activity, but should be based upon real historical data.

This means that if you were to say that the time estimates for completing activity Z are:

4 days or less, 10% probability

5 days, 80% probability

6 days or more, 10% probability

then looking at all instances of where the company or team has performed activity Z,

In 10% of cases, it should have taken 4 days or less

In 80% of cases, it should have taken 5 days

In 10% of cases, it should have taken more than 5 days

One way to address this challenge is at the Project Management Office level. Rather than asking individual areas or teams to forecast their timelines (which we know are often wildly inaccurate due to the Planning Fallacy), the PMO should begin to construct and set of statistics at both the activity and project level.

Then, when constructing a project plan, they would be able to look first at actual historical data on the distribution of how long similar projects took to implement, giving them greater insight into the potential range of total time required for their particular project.

However perhaps more usefully, they would also be able to look at distributions for how long specific activities have taken too. Whereas it can often be hard to find a comparable project, most projects are made up of very similar activities, such as recruitment, training, setting up computers, systems changes, contract signing etc.

So for example, rather than building your project plan based on the HR estimate of hiring X positions in 4 weeks and then being surprised when it takes longer, the inherent uncertainty would already be built in based upon actual data on historical recruiting activities.

Bayesian Updating

The other key element of maintaining more realistic project estimates is through constant updating as new information appears.

I have seen so many projects managed by assuming that all requirements will be finalized by X date, only to see conflict arise when the scope needs to be modified in some way.

Maybe it is possible to implement some projects this way, however in my experience, it is far better to think of project scope and requirements as fluid:

External factors can lead to changing requirements in the middle of the project (i.e., competitor products, regulatory needs etc.)

Testing with real-world examples can lead to the realization that the initial requirements were not quite correct

Plain old human error can mean that somebody made a mistake or forgot to include something initially

Methodologies like Agile are certainly one way of addressing and incorporating this inherent uncertainty in project management, however whatever methodology you use, I believe it is key to, almost on a daily basis, take new evidence and use it to update the project time and cost estimates.

Returning to our recruiting example from above. Suppose we had initially estimated the hiring of 4 positions to take 4 weeks with 80% probability.

Then our HR contact comes back to us and says that having reviewed the specific profiles and salary ranges, they realized that these positions are more complicated to fill. We would then look again at our historical data and see how this could impact our estimates, and we may revise our timeline to be 6 weeks with probability 80%, but with a 20% probability of 8 weeks.

Even if we don’t have the historical data available, we should still go through a formal updating process, looking at our original timelines, and figuring out how this new information could affect the timings and probabilities.

Business Deadlines

Perhaps the biggest challenge to this type of project management comes in the form of imposed and hard-business deadlines.

Even though the Project Manager might estimate the project will require 8 weeks with 80% probability or up to 12 weeks with 20% probability, the reality could be that for whatever reason the business needs to launch in exactly 8 weeks.

I would argue that a probabilistic approach is still useful for two reasons:

By sensitizing the project sponsor to uncertainty early on, later conversations regarding time or cost risks should be at least slightly easier than if you had used fixed and 100% confident estimates from the start

Consider the following question posed on http://fivethirtyeight.com/:

There is a very long, straight highway with some number of cars (N) placed somewhere along it, randomly. The highway is only one lane, so the cars can’t pass each other. Each car is going in the same direction, and each driver has a distinct positive speed at which she prefers to travel. Each preferred speed is chosen at random. Each driver travels at her preferred speed unless she gets stuck behind a slower car, in which case she remains stuck behind the slower car. On average, how many groups of cars will eventually form? (A group is one or more cars travelling at the same speed.)

 For example, if the car in the very front happens to be slowest, there will be exactly one group — everybody will eventually pile up behind the slowpoke. If the cars happen to end up in order, fastest to slowest, there will be N groups — no car ever gets stuck behind a slower car.

Overall I think it took me too long to solve, and I am sure that this isn’t the fastest way of getting to the answer, but here is my solution.

Note: It turns out that I made a silly mistake and ended up solving a slightly different problem. The mistake was due to abstracting the question too soon and only thinking about relative orders of speed in a given sequence and not taking into account that in the real world, cars will catch up with each other, and so under-counting the number of single groupings. Lesson learned.

I will leave my original solution at the bottom, because it was still an interesting problem and I ended up learning about something I didn’t know much about: Eulerian Numbers.

Correct Solution

Given a particular arrangement of the cars, we can consider a recursive algorithm for counting the number of pairings as follows:

Find the location of the car with the lowest velocity

That car and everything behind it represent 1 grouping, as all of the cars behind it have a higher speed

Add 1 to the number of groupings of the set of cars in front of it, etc.

This is clearly a recursive problem, where every time we add a car, the new expected number of groupings is dependent on the previous number of groupings.

To simplify the problem let us suppose, without loss of generality, that we for some n<N, we line up all of the cars in order of decreasing speed and then place them randomly on the road, starting with the fastest car and moving downward. Let’s call the number of resulting groupings G(n).

Now we add in the next slowest car, that is a car whose speed is slower than all of the existing cars on the road. There are n+1 possible positions where we can place this car, and in any location, and we can see that:

With probability 1/(n+1) the car goes at the back and creates 1 new group, so the total number of groups is G(n) + 1

With probability 1/(n+1) the car goes at the front and results in a total of just 1 group

In general, with probability 1/(n+1) the new car creates 1 group consisting of all the cars behind it + the expected groupings of all the cars in front of it

For N=1, we know that E(G) = 1, so we’re done and:

which is the harmonic series.

Original Solution

The problem that my original solution answers is more like the following question:

The astronomer Simon Newcomb played the following game of solitaire. Cards numbered 1,2,....,n are shuffled, drawn and put into a pile as long as the card drawn has a number lower than its precessor. What is the average number of piles?

First of all, a couple of comments on the problem:

This is clearly a counting problem involving permutations: we are interested in different ways of arranging the cards, and the order matters because the relative sizes of the cards dictate the number of piles

In fact we can abstract away from cards and simply consider permutations of the integers 1 to N.

As mentioned in the question, the extreme cases are quite simple:

To end up with one pile, the only possibility is having all of the integers in decreasing order of magnitude, that is to say N > N-1 > N-2 >..>2 > 1 (a strictly decreasing sequence).

To end up with N piles, the only possibility is having all of the integers in increasing order of magnitude, that is to say 1 < 2 < 3 <… < N-1 < N (a strictly increasing sequence)

If we play around with fairly small values of N, we can see some other interesting patterns start to emerge. Let us consider, for instance, how many ways there are there of having 2 piles.

For N=3, the possibilities are:

(3,1,2), (2,1,3), (2,3,1) and (1,3,2)

For N=4, the possibilities are:

(1,4,3,2), (2,1,4,3), (2,4,3,1), (3,1,4,2), (3,2,1,4), (3,2,4,1), (3,4,2,1), (4,1,3,2), (4,2,1,3), (4,2,3,1), (4,3,1,2)

With a little bit more work, we can see that what matters is the relationship between consecutive pairs of integers, specifically whether they are < or >.

For N=4 there are three consecutive pairings, and in order to have two piles, we need two of the pairings to be > and one pairing to be <. By similar reasoning we are able to see that for general N, we need N-1 pairings to be > and one pairing to be <.

 In fact, what dictates the number of end piles is the number of ascents (pairings where the 1st is < the 2nd) or descents (pairings where the 1st is < the 2nd) in an ordered sequence.

Furthermore, the number of permutations of a sequence with the specific number of ascents or descents will tell us how many possible ways there are to end up with a certain number of piles.

Eulerian Numbers

It turns out that what we are looking for are numbers called Eulerian Numbers.

From Wikipedia:

the Eulerian number A(n, m), is the number of permutations of the numbers 1 to n in which exactly m elements are greater than the previous element (permutations with m "ascents")

It is possible to generate the Eulerian Numbers algorithmically by counting the number of permutations of N that result in a given number of groups, or alternatively using well-defined iterative or generating functions.

Again from Wikipedia, here is a table with the Eulerian Numbers for small N:

Before we get to the final solution, there are two interesting properties of Eulerian numbers that are worth noting:

For a given row in the table above, the Eulerian Numbers for a particular N sum to N! (because we end up counting all possible permutations of N). Alternatively, we can write this as:

There is also symmetry within the Eulerian Numbers for given n, i.e., A(n,0) = A(n,n-1), A(n,1) = A(n,n-2) etc. We can see this by repeating our arguments above for a reversed sequence of integers.

The Solution

What we are looking for is the average, or expected, number of groups (piles) for a given N.

By definition of expectation:

The probability of a given grouping is the number of ways of achieving that grouping / the total possible arrangements of N, which is given by:

Where A(N,G-1) is the Eulerian Number.

If we sum over all possible groupings, we are looking for:

We can write this sum out as follows:

We can then fill-in the triangle in order to make a rectangle:

However, from the properties of the Eulerian Numbers we know that

Each row sums to N!

The sum of the red terms is equal to the sum of the blue terms due to the symmetry in the Eulerian Numbers.

Thus we can see that:

And so we get our final result:

One of my personal projects for the beginning of this year was to get more comfortable with Python in preparation for (maybe) doing a Data Science course of some kind. However I knew that just trying to work through theory wouldn’t be very effective, and instead wanted to play around with an actual problem as a learning device.

So, a few weeks ago when I was stuck waiting at a particularly bad set of traffic lights near my house and thinking about how bad the timing seemed to be, I hit upon an idea for doing something in Python: modeling the behavior of traffic at a traffic light.

Quick disclaimer: the only thing I really wanted to achieve was to improve in Python, so I make no claim as to the accuracy of any models, assumptions or results when it comes to actual traffic.

Model 1: Single Car arriving per interval

The first version of my model was a very-simple one, looking at single cars arriving at a light during 1-minute intervals and calculating the average car waiting time.

I created a simple TrafficLight class with two basic properties:

greenTime being the number of seconds the light is green

redTime the number of seconds the light is red

and for simplicity I assumed that greenTime + redTime = 60

I modeled car arrival time as a Uniform random variable on [0,60] (i.e., each individual car is equally likely to arrive at any time during the minute interval.)

I ran simulations for greenTime starting at 0.6 seconds, and in increments of 0.2 seconds (up to 59.4), and for each greenTime calculated the average for 20 independent car arrival times.

The results are not that surprising. When the greenTime % is low, then the average wait time is around 30 seconds. We would expect this because the total redTime is approximately 60 seconds, and the expected car arrival time is at 30 seconds, so on average each car would be likely to wait 30 seconds before the light turns green again.

Similarly, we can see that when the green/red proportion is 50-50, the average waiting time is somewhere between 5 and 10 seconds. In this case, we would expect half the cars to arrive between 0-30 with a wait time of 0, and half to arrive between 30-60 with an average wait time of 15 seconds, leading to a global average of 7.5s.

Finally, when the light is green nearly all of the time, the average wait time is around 0 seconds, again as we would expect.

Model 2: Multiple cars arriving per interval

Here I move on to a more complex case where multiple cars can arrive in a given 60-second interval.

I model cars arriving as a Poisson Random Variable, and again look at average car waiting time based on three variables:

Car arrival rate: Between 2 and 38 per minute

Green time proportion: % of 60-second interval traffic light is green

Car pass rate: the number of cars that can pass the light per second when it is green

First we plot car arrival rate vs. average waiting time, for different values of the greenTime percentage.

Again the results are not so surprising. As the car arrival rate increases, the average wait time increases, with the gradient determined by the % of the interval for which the light is green.

The picture is pretty similar if we vary the value of car pass rate instead of greenTime %, although with different gradient values.

Model 3: Multiple cars arriving per interval + blocked junction

For the next iteration, I took Model 2 and added in an extra factor: junction blocking.

From personal experience (where I live), I have seen that in heavy traffic (i.e. high car arrival rate), traffic junctions can often become blocked as cars either remain stationary across the junction when their light turns red, or continue to pass after the light turns red and hence end up blocking the oncoming traffic.

I model this using a blocking probability, p, so when cars arrive at the light then with probability p, the junction is blocked and so even if the light is green, then all cars must wait for the full 60 seconds (greenTime + redTime) before they have the possibility of advancing (or again being blocked with probability p).

In this example I assume that if the junction is blocked, then the block lasts for the full greenTime.

[Note: in both Models 2 and 3, I am assuming that all the cars arrive at the start of the 60 second interval.]

Here we plot Average car waiting time vs. Car arrival rate for different values of p, the blocking probability, all for the case where the light is green for 30 seconds.

As we would expect, average waiting time increases substantially as the probability of the intersection being clocked increases, but also variability in waiting time (around the best fit line) also increases quite dramatically.

Model 4 Part I: Minute-by-minute simulation

In the next iteration, I looked at a minute-by-minute simulation over the course of a one-hour period.

The rough procedure for each one-minute interval is:

Simulate new number of cars, X, that arrive and join the queue by drawing from a Poisson distribution

Calculate the number, Y, of cars that pass the traffic light based upon traffic conditions

Calculate the average wait time for the cars passing in that period

Update the ‘new’ queue of cars to reflect the situation after that period.

I also modified some of my assumptions regarding the traffic flow:

The rate at which cars can pass a green light decreases as the traffic gets heavier – based on personal observation, what I see is that the heavier the traffic, the more people try and do things like cut across lanes at the last second to turn etc., and so overall everyone advances more slowly

The probability that other cars block the junction on a green light also increases with traffic density: again in heavier traffic, it is more likely that cars will end up blocking when their corresponding light turns red

If cars do block the junction, then it is not for the whole of the green-light period, but for a varying proportion of time

In this model we use a new model, Car, with a waitTime property to keep track of how long each cars waits before it passes.

I defined three types of traffic conditions, Light, Medium & Heavy as follows:

Light Traffic:

Car arrival rate: 8 cars per minute

Car pass rate: 1 car per second when the traffic light is green

Blocking probability: 0.05

Blocking time: 5% of the green light period

Medium Traffic:

Car arrival rate: 12 cars per minute

Car pass rate: 0.9 car per second when the traffic light is green (90% of normal rate)

Blocking probability: 0.2

Blocking time: 20% of the green light period

Heavy Traffic:

Car arrival rate: 20 cars per minute

Car pass rate: 0.72 car per second when the traffic light is green (72% of normal rate)

Blocking probability: 0.45

Blocking time: 50% of the green light period

I divided the 60 minute period into five different sub-periods:

5 mins of Light traffic

15 mins of Heavy traffic

20 mins of Medium traffic

15 mins of Heavy traffic

5 mins of Light traffic

For each 1-minute period the program adds a new number of cars to the waiting queue, based upon the arrival rate, and uses a runTraffic function to:

Model the probability that the intersection is blocked

Calculate the number of cars that can pass

Update the wait time for each car that passes

Calculate the average wait time for the cars that passed in that period

Remove the passed cars from the queue

Update the wait time for each of the cars still in the queue

Note regarding waiting times:

For each minute interval, for the cars that do not pass the light, we add 60 seconds to their total wait time.

For cars that do pass the light in a given interval, I originally added 0 seconds to their wait time, effectively assuming that once the light turns green all cars that are going to pass do so immediately.

However this assumption is not realistic, and so I changed this by dividing the light greenTime by the number of cars that pass, taking into account any blocking time, and assuming that these cars pass uniformly in that interval.

Finally, I run the simulation 1000 times and look at average results per minute for

Number of cars still waiting in the queue

Number of cars passing per minute

Average car waiting time

Regarding the size of the traffic queue, unsurprisingly we see the number of waiting cars around 0 when the traffic is Light or Medium and pretty much all cars are able to pass in the given time, and the size of the queue steadily increases to a maximum of around 12 or 13 cars in heavier traffic.

In terms of passing cars, we see fairly steady (but different) pass rates in each of the traffic conditions, plus a little bit of noise being generated by the blocking probabilities.

If we look at the average waiting time per period, defined to be the average wait time for the cars that make it past the traffic light in the minute, we see large variability. In Heavy traffic conditions, in some of the simulations, the average waiting time reached nearly 600 seconds, whereas the average and median statistics across all of the simulations are much lower at around 50 seconds.

We can get a better picture of what is going on by looking at the distribution of waiting times across all cars, and it turns out that about 75% of cars wait for less than 50 seconds, 20% wait for between 50 and 100 seconds and less than 5% wait for more than 100 seconds.

In a subsequent post I will explore two further improvements:

Simplifying the structure of Model 4 by introducing new classes

Creating a second-by-second simulation

1. The Concept

AI Sous-Chef

Ever find yourself needing to manage several key steps at the same time? Fear no more, your digital sous chef can help you multi-task like never before.

Chef Jarvis automatically integrates with all of your smart appliances giving you several extra pairs of hands in the kitchen.

Recipe Manager

Do you find that you can’t remember if you used 600g or 900g of flour for your cake last time or how long it took to bake in your oven? Do you like to experiment with recipes, quantities and timing?

Chef Jarvis lets you annotate, modify and store every aspect of your recipes on-the-fly. Plus with in-built version control, you will have full access to every version of the recipes behind your signature dishes.

Tailored Suggestions

Not sure what to cook tonight? Maybe you have a vague idea in mind but are looking for some inspiration?

Chef Jarvis suggests dishes and finds recipes based upon key ingredients, cuisines, cooking techniques, preparation time or even wine or beer pairings.

Over time, Chef Jarvis learns through its interactions with you, making its recommendations more personalized to your preferences and style of cooking.

Workflow Control

Overcooked meat, burned vegetables and frantic last-minute scrambles are a thing of the past. Chef Jarvis keeps you on-track with all of your recipes whether you are cooking a single dish for supper or a multi-course haute cuisine meal for your friends.

By carefully tracking your progress through all of your active recipes, Chef Jarvis can warn you in advance of key upcoming steps and set alarms for time critical activities.

Furthermore, Chef Jarvis automatically takes care of conversions between imperial and metric systems, scales quantities, translates ingredients or suggests alternatives so that you can concentrate on the cooking.

Smart Shopping Lists

Once you have selected your recipes, Chef Jarvis automatically adds the necessary ingredients to your smart shopping list on the device of your choice.

Do you often work with recipes from across the world?

Chef Jarvis has you covered by automatically translating exotic ingredients into locally-available alternatives as well as finding nearby shops for buying the more unusual items.

Conversational Interactions

Juggling multiple dishes and utensils with messy or wet fingers doesn’t leave you much flexibility for interacting with a computer or other device.

Chef Jarvis listens for and responds to your commands and makes you feel like you have a real assistant at all times in the kitchen.

2. Background

Cooking has been a favourite hobby of mine for many years. However much of the time I find myself working alone in the kitchen trying to multi-task between various complex steps in concurrent recipes, all the while re-washing and drying my hands to return to my computer or phone in order to look up the next step, calculate quantities for ingredients, convert between cups and grams etc.

Separately, I have also been trying to study the principles behind machine learning techniques and in particular neural networks (fantastic introduction by Michael Nielsen here), and have long been interested in trying to link my interests in cooking and technology.

One day I was reading about Artificial Intelligence applications focused on home control systems, and I started wondering about more specialized applications for very specific tasks. It dawned on me that I could be far faster and more efficient working in the kitchen if I could automate everything to do with following recipes whilst trying to do something else with your hands, and so I started to develop the idea from there.

The name is (obviously) inspired by Iron Man, as in my head I imagine the helper to be as useful, friendly and easy-to-interact-with as Jarvis is for Tony Stark.

3. Some Thoughts on Implementation

These are just some very high level thoughts on how I might go about taking the first steps to implement something like this.

Recipe Parsing

A key requirement would be for the AI to be able to identify what a recipe looks like as well as break it down into sections (i.e. ingredients vs. method), as well as subsections (individual steps in method etc.)

On the face of it, this doesn’t look too hard as recipes themselves often have headings like Ingredients and Method, however I’m sure in this case the devil is in the detail. For instance you could start to think of ingredients as having some sort of predefined format {number - unit - ingredient}, however often some sub-quantity of ingredients are used multiple times in the method so you would need to be able to distinguish between many different cases.

As a start I would probably put together some type of web scraper to grab a large number of recipes from sites like BBC Cooking, SeriousEats, Chowhound etc.

Then I might use some kind of supervised learning to begin to break recipes down into their individual elements.

Recipe Storage

Once a recipe has been parsed then it can be stored locally for the user in some type of database structure.

I would look to integrate version control like git so that, for each recipe, the user is able to create and store copies, modifying specific elements. So for instance if the base recipe calls for 1kg of flour for serving 10 people, and you want to save a version scaled down for serving 4 people, then you would be able to store a modified copy.

Similarly if you find that you needed a higher oven temperature, longer cooking time or any other type of change, then you can store your own personalized version of that recipe.

Basic Helper Methods

I consider these to be some of the simpler requirements to implement, that is for things like using voice commands to convert between measurement systems, translate foreign ingredients or even suggest alternative ingredients.

Currently you could probably do much of this with Siri, however in this case I think the problem is far narrower and better-defined (i.e., you don’t need to be able to process all natural language, just very specific commands for executing very specific functions).

So perhaps for conversions you could (at least at the start) use a specific command phrase to indicate that the AI will need to listen out for 1) a number, 2) input unit and 3) output unit.

For being able to suggest alternative ingredients, you could start by creating a large dictionary with common alternatives for harder-to-find ingredients (many cooking sites already have these types of lists).

Integrating with Appliances

While I think that just the recipe-management functionality could be a big help while cooking, it would also be fantastic for this type of AI to help you with controlling kitchen appliances too.

The very futuristic version would be just like Jarvis, with some sort of robotic arm that could move around and actually interact with items in the kitchen.

Obviously this is a far-fetched scenario, however more immediately I am seeing a move by appliance manufacturers to building devices that you can control via apps. For example I was recently given as a present a small Anova Sous Vide machine, which can be controlled from an app including turning it on and off, changing the temperature, setting a timer etc.

Potentially as more appliances emerge with associated apps, then a kitchen AI could use those interfaces to give you the ability to interact with appliances using only voice commands.

Command and Control

A key part of the value of this sort of AI would be the ability to interact with it hands-free.

As I mentioned before, this problem could probably be reduced (somewhat) in complexity by defining specific commands that can be used to activate certain functions. Another gift I received last Christmas was the BB8 robot by Sphero which has a small list of in-built voice activation commands that work surprisingly well.

So similarly you could program it to a list of 5-10 voice commands for things like:

Conversions

Set alarm

Get next step

Over the past couple of years I found myself getting frustrated that I wasn’t reading enough, and so in 2015 when I saw friends signing up for Goodreads’ Reading challenge, I thought it might be a nice way to set a goal and try and stick to it.

I wanted to challenge myself so went for quite an aggressive goal of 52 books. This is how I fared.

Overall statistics:

53 books in total

17,512 pages

343 page average book length

41 Non-Fiction (77%) vs. 12 Fiction

Month with most books finished - December (8)

Month with least books finished - January, September, October (2)

The List

Methodology:

I used a small Python script to download my books from 2015 via the Goodreads API.

I manually tagged the books into the following categories (which are not necessarily mutually exclusive), and also reevaluated the score that I originally gave when I saved the book in the Goodreads app.

For each book I have specified in brackets my original score followed by an updated score based upon my impressions 1-12 months after finishing.

In the case where I don’t remember much about the book (surprisingly often for the fiction books), I have left the score as the original. Also within the updated score I have included half points which are not currently allowed on Goodreads.

The numbers following the category titles are the average scores of my original reviews, followed by updated reviews.

Any thoughts or comments I think to be worth noting about a particular book are in italics.

Some observations:

I was already sure I had a read a lot more non-fiction than fiction, but how much more actually surprised me a little (41 non-fiction vs 12 fiction)

I hadn’t realized I had read so many biographies

In general I remember a lot more about the non-fiction rather than fiction books

Overall I think I was pretty consistent in my ratings between before and after - although I changed my rating for 25 books, this was mostly to be able to take advantage of half points (21 books)

I only change my mind by 1 point on 4 books (all increases)

Maths, Computing, Technology: (3.7, 3.8)

Naked Statistics (4, 3.5)

The Information: A History, a Theory, a Flood (4, 4) - very interesting overview but quite dense in parts

The Theory That Would Not Die: How Bayes' Rule Cracked the Enigma Code, Hunted Down Russian  Submarines, and Emerged Triumphant from Two Centuries of Controversy (3, 4) - very detailed overview of Bayes; I still remember a lot of the detailed examples

The Simpsons and Their Mathematical Secrets (3, 3)

A Mathematician's Apology (3, 3.5) - felt more like a book that I had to read rather than wanted to read

The Signal and the Noise: Why So Many Predictions Fail - But Some Don't (5, 5) - awesome, really enjoyable. Want to re-read.

How Not to Be Wrong: The Power of Mathematical Thinking (4, 4) - one of the best popular maths books I have read in a while; want to re-read.

Six Degrees: The Science of a Connected Age (3, 4) - the maths is really fascinating; there are parts I want to look over in more detail

Probability: A Very Short Introduction (3, 3)

Countdown to Zero Day: Stuxnet and the Launch of the World's First Digital Weapon (4, 4) - really interesting and eye opening

The Shadow Factory: The Ultra-Secret NSA from 9/11 to the Eavesdropping on America (3, 3) - while it was full of interesting tidbits, I thought it could have been more

No Place to Hide: Edward Snowden, the NSA, and the U.S. Surveillance State (5, 4.5) - an absolute must read book

The Dark Net: Inside the Digital Underworld (4, 4) - very interesting, sightly scary

Biographies: (4.2, 4.1)

The Immortal Life of Henrietta Lacks (5, 5) - amazing story, highly recommeneded, so important to modern science and medicine

Alcatraz-1259 (4, 4) - true story of someone who was incarcerated in Alcatraz (I have a copy signed by the author)

Thomas Cromwell: The Untold Story of Henry VIII's Most Faithful Servant (4, 4.5) - I knew nothing about him beforehand; fascinating book

Not Much of an Engineer (3, 3) - Some very interesting stories and engineering, but I wasn’t a big fan of the writing

Longitude: The True Story of a Lone Genius Who Solved the Greatest Scientific Problem of His Time (5, 4.5) - Really good

Einstein: His Life and Universe (4, 3.5) - I was very glad to learn more about such an important and interesting figure, but was a bit long and hard going in places

Agent Zigzag: A True Story of Nazi Espionage, Love, and Betrayal (5, 4.5) - Very cool story, and easy to read

A Spy Among Friends: Kim Philby and the Great Betrayal (4, 4) - Very good

A Numerate Life: A  Mathematician Explores the Vagaries of Life, His Own and Probably Yours (4, 3.5) - Mixed, good in parts

Historical: (3.4, 3.6)

In the Heart of the Sea: The Tragedy of the Whaleship Essex (4, 4) - amazing and slightly harrowing story (and then I discovered they were making a film!)

Operation Mincemeat: How a Dead Man and a Bizarre Plan Fooled the Nazis and Assured an  Allied Victory (4, 4) - Really well-written, gripping

Killing the  Bismarck: Destroying the Pride of Hitler's Fleet (3, 3.5)

The Secret Life of Bletchley Park: The WWII Codebreaking Centre and the Men and Women Who Worked  There (3, 4) - Really good overview of Blethley and the people who worked there

Tambora: The Eruption That Changed the World (3, 2.5) - I bought this book because of the story; didn’t live up to expectations

Other Science, Medical etc: (4.25, 4.5)

Bad Pharma: How Drug Companies Mislead Doctors and Harm Patients (4, 4.5) - Very important book, should be mandatory reading in schools

What If? : Serious Scientific Answers to  Absurd Hypothetical Questions (5, 5) - I love XKCD, I loved this book

Thinking, Fast and Slow (3, 3.5) - the subject matter I found fascinating, but I have read better expositions

The Emperor of All Maladies: A Biography  of Cancer (5, 5) - amazing and epic journey into ‘the war on Cancer’

Management: (4.3, 4.3)

The Halo Effect:  How Managers let Themselves be Deceived (3, 3.5) - Some really interesting ideas

Joy, Inc.: How We Built a Workplace  People Love (5, 5) - I loved this book and recommend it to nearly everyone I know. Maybe the best business of management book of all time

The Checklist Manifesto: How to Get  Things Right (5, 4.5) - Everyone needs checklists!

Other: (3.6, 3.6)

The End of Power:  From Boardrooms to Battlefields and Churches to States, Why Being In Charge  Isn't What It Used to Be (3, 2.5) - Sounded interesting but too dry and uninspired writing

The Antidote: Happiness for People Who  Can't Stand Positive Thinking (4, 4)

Average Is Over: Powering America Beyond  the Age of the Great Stagnation (3, 3.5)

Mornings in Mexico (3, 4)

The Road to Little Dribbling: More Notes  From a Small Island (4, 4) - Not his best work, but typically gentle and amusing stories

How to be a Husband (5, 4.5) - Very amusing

The House of Wigs (3, 3.5)

Fiction: (3.8, 4.0)

The Bone Clocks (4, 4)

The Temp (4, 4)

A Delicate Truth (4, 4) - solid le Carré

The Farm (4, 4)

Jane Eyre (4, 4.5) - really enjoyed

The State We're In (2, 2)

The Memory of Love (5, 5)

The Martian (4, 5) - loved the film, loved the book more

The Lathe of Heaven (4, 4)

Make Me (Jack Reacher #20) (3, 3.5) - once again, rescued me from a long flight

Binti (3, 3)

I have been looking over some elementary probability recently, and spent some time thinking about a couple of questions that I think have pretty neat solutions.

1. Birthday Problem

The problem is very simply stated, but the answer is counterintuitive to most people.

What is the smallest number of people that you need in a room (or some type of gathering) for there to be a better than even chance that two people share the same birthday?

What seems to throw people off here is that they immediately try and answer a different question: What is the probability that someone else shares my birthday (or some other specific birthday)?.

In fact, we can quite quickly estimate a rough value for what this number of people should be (n). If we do not concern ourselves with a specific day, then what we are looking for is any pair of people who share the same birthday.

For some small values of n, the number of possible pairings are: n = 18, 153 pairings (153/365 = 0.419) n = 19, 171 pairings (171/365 = 0.468) n = 20, 190 pairing (190/365 = 0.521)

So intuitively, we think that the answer should be somewhere around 20.

The actual calculation follows standard combinatorial reasoning:

Suppose n = 2. It turns out that it is simpler to think about the probability of the people not sharing a birthday, and then take the complement.

We can see that there are 365×365 total possible birthday combinations (365 for each person), and 365×364 combinations for them not sharing a birthday (365 possibilities for the first person, and 364 for the second as one day is now “out of the question”).

So for 2 people, the probability that they don’t share a birthday is

365×364365×365≈0.997.

For n = 3, similarly we can see that P=365×364×363365×365×365≈0.992

In fact, for general n, by similar reasoning we can see that the probability that none of them share a birthday is:

P=365×364×...×(365-n+1)365×365×...×365=365!(365-n)! 365n

(in the numerator, each person removes a possibility for the number of allowed birthdays).

Using Stirling’s approximation, we can show that this is

~e-n 365365-n365-n+0.5

Once this probability drops below 0.5, then we have a better than 50-50 chance of two people sharing a birthday.

For values of n around 20, we can see that: n = 20, P = 0.589 n = 21, P = 0.556 n = 22, P = 0.524 n = 23, P = 0.493

So we see that with only 23 people, the probability that two people share the same birthday is 0.51.

2. Boxes and balls

Suppose we have n balls that are tossed independently into n boxes. What is the probability that exactly one box is empty?

Once again let’s start by thinking about some small values of n.

If n=2, the total number of possibilities is two boxes for ball 1 x two boxes for ball 2 = 4 in total.

For one box to be empty, we only want to consider those arrangements where both balls are in one box, of which there are 2 (both in box 1 or both in box 2), so

P = 24=12

If n=3, suppose box 1 is empty, then we can either have two balls in box 2 and one in box 3 or one ball in box 2 and two in box 3.

So for our numerator we need three possibilities for the empty box x three ways of choosing 2 out of 3 balls x two possibilities for the box with 2 balls.

And in the denominator we have three possible boxes for each ball.

P=3×3×23×3×3=23

For general n, we can use similar reasoning.

The key is seeing that with exactly one empty box, all of the remaining n-1 boxes must have exactly one ball except for one box which will have two balls.

What might this look like:

There are n choices for the empty box. The two balls can be chosen ways. There are n-1 choices for the box with 2 balls. The remaining n-2 balls have (n-2)! arrangements among the n-2 boxes.

So the numerator is n×(n-1)×(n-2)!×(nC2), and the denominator is nn, and

n!nn×(nC2)=n!×n×(n-1)2nn

What does this look like for small values of n? n = 2, P = 0.5 n = 3, P = 0.667 n = 4, P = 0.563 n = 5, P = 0.384 n = 6, P = 0.231 n = 7, P = 0.129 n = 8, P = 0.067 n = 9, P = 0.0337 n = 10, P = 0.0163

Choosing Yourself is hard!

If you don’t know what I mean by Choosing Yourself, then you clearly don’t know James Altucher, and you should. His writing is refreshingly honest, often funny and at the very least thought provoking.

For reference, his book ‘Choose Yourself’ is on Amazon and the e-version is very cheap.

What Does Choose Yourself Mean?

There are two big problems.

First, in most societies, we are conditioned to measure success according to a very specific and limited set of conditions:

Success = a steady, well-paying job

Success = becoming wealthy

Success = owning a house

Success = owning a couple of cars

Second, the world doesn’t work how we think or are led to believe it works, and even worse, in most cases the rules of the game are stacked against us.

The general prevailing wisdom is: Go to college to get a degree to get a job to work hard to get a promotion to buy a house and a car and then retire.

But, as many people over the past 10 or so years have painfully discovered:

College (or even advanced) degrees don’t necessarily guarantee a good job.

Jobs are not for life; companies are not loyal (I have seen many people ‘let go’ after nearly a lifetime of loyal service).

Owning a house does not guarantee happiness, and can even lead to more pain (i.e. when you can’t pay your mortgage).

Choosing yourself comes down to confronting each of these problems through personal choice and responsibility.

Choice 1. Define success on your own terms.

Throw out everything that depends on external influences (other peoples’ opinions) and choose what you want for success.

Be grateful every day for what you already have. 

Choice 2. Choose to create your own success.

Don’t sit around waiting for someone else to fulfill your dreams and make you successful. Figure out what you want to do and then figure out the very first step you need to take to make it happen.

For example, if you want to write a book, take out a sheet of paper and write down a title, or a main characters name, or their description, or an idea for the story, or even the ending.

As James says:

build your own platform, have faith and confidence in yourself instead of the jury-rigged system and define success by your own terms.

Probably the key point though is dealing with failure and rejection by picking yourself up and trying again. Treat your ideas as small experiments to learn what does and doesn’t work for you.

Don’t invest yourself emotionally too much in one single idea or outcome.

One of James’ most powerful quotes comes from a later book (The Choose Yourself Guide to Wealth):

The new way of doing things, the red pill, is to jump into the abyss. To try and fail and cry and wail until finally you say, “I can’t do this one more time. I just can’t. Please don’t make me.

But you do it anyway. And that one last time, you chose yourself

I know a lot of it sounds corny, although I think that’s mostly on purpose in order to appeal to the dreamer in all of us.

And even if you consider yourself to be a rationalist and can’t stand all the self-help mumbo jumbo, I don’t think you can deny:

It doesn’t make sense to measure your success based upon other people’s ideas and values

You have to take responsibility for your own success (you can’t become a published author if you never write a book, you can’t own your own business if you don’t try and start one and you can’t win the lottery if you don’t buy a ticket).

The trouble is that it feels so damned hard!

In the moment, as you read James’ writing or listen to his podcast, you feel inspired. Everything becomes clear in your mind and it is very easy to choose to change. 

Maintaining that commitment is the hard part. Throughout the day you get worn down as you are bombarded by malicious thoughts: 

“I can’t stop doing this project: I need the money to pay the bills”

“Don’t be ridiculous, no-one would buy my book.”

“I don’t even know where to start.”

“My business idea will never work; there are too many challenges.”

“I’m breaking the rules if I’m not working 9-5, five days a week.”

“Work is supposed to be difficult and feel like a chore.”

And then by the end of the day all of your positivity and optimism have been crushed and you are left with the thought that

“those ideas sound nice, but they’re not for everyday regular people like me. That’s not how the world works. I should just put my head down and keep going with what I’m doing.”

So once again you have to summon up all of the positivity and remind yourself that there are different and better ways to work, that there is no universal or moral definition of success or of when or how you should support yourself and, most importantly, that if you never try there is no hope of change.

Right now it feels like kicking a habit, and that’s probably because that’s what you have to do: break a mental habit of a lifetime.

(Full disclosure I have never had to kick a physical addiction, and I am certainly not trying to belittle what addicts must go through!)