#chefjarvis

1. The Concept

AI Sous-Chef

Ever find yourself needing to manage several key steps at the same time? Fear no more, your digital sous chef can help you multi-task like never before.

Chef Jarvis automatically integrates with all of your smart appliances giving you several extra pairs of hands in the kitchen.

Recipe Manager

Do you find that you can’t remember if you used 600g or 900g of flour for your cake last time or how long it took to bake in your oven? Do you like to experiment with recipes, quantities and timing?

Chef Jarvis lets you annotate, modify and store every aspect of your recipes on-the-fly. Plus with in-built version control, you will have full access to every version of the recipes behind your signature dishes.

Tailored Suggestions

Not sure what to cook tonight? Maybe you have a vague idea in mind but are looking for some inspiration?

Chef Jarvis suggests dishes and finds recipes based upon key ingredients, cuisines, cooking techniques, preparation time or even wine or beer pairings.

Over time, Chef Jarvis learns through its interactions with you, making its recommendations more personalized to your preferences and style of cooking.

Workflow Control

Overcooked meat, burned vegetables and frantic last-minute scrambles are a thing of the past. Chef Jarvis keeps you on-track with all of your recipes whether you are cooking a single dish for supper or a multi-course haute cuisine meal for your friends.

By carefully tracking your progress through all of your active recipes, Chef Jarvis can warn you in advance of key upcoming steps and set alarms for time critical activities.

Furthermore, Chef Jarvis automatically takes care of conversions between imperial and metric systems, scales quantities, translates ingredients or suggests alternatives so that you can concentrate on the cooking.

Smart Shopping Lists

Once you have selected your recipes, Chef Jarvis automatically adds the necessary ingredients to your smart shopping list on the device of your choice.

Do you often work with recipes from across the world?

Chef Jarvis has you covered by automatically translating exotic ingredients into locally-available alternatives as well as finding nearby shops for buying the more unusual items.

Conversational Interactions

Juggling multiple dishes and utensils with messy or wet fingers doesn’t leave you much flexibility for interacting with a computer or other device.

Chef Jarvis listens for and responds to your commands and makes you feel like you have a real assistant at all times in the kitchen.

2. Background

Cooking has been a favourite hobby of mine for many years. However much of the time I find myself working alone in the kitchen trying to multi-task between various complex steps in concurrent recipes, all the while re-washing and drying my hands to return to my computer or phone in order to look up the next step, calculate quantities for ingredients, convert between cups and grams etc.

Separately, I have also been trying to study the principles behind machine learning techniques and in particular neural networks (fantastic introduction by Michael Nielsen here), and have long been interested in trying to link my interests in cooking and technology.

One day I was reading about Artificial Intelligence applications focused on home control systems, and I started wondering about more specialized applications for very specific tasks. It dawned on me that I could be far faster and more efficient working in the kitchen if I could automate everything to do with following recipes whilst trying to do something else with your hands, and so I started to develop the idea from there.

The name is (obviously) inspired by Iron Man, as in my head I imagine the helper to be as useful, friendly and easy-to-interact-with as Jarvis is for Tony Stark.

3. Some Thoughts on Implementation

These are just some very high level thoughts on how I might go about taking the first steps to implement something like this.

Recipe Parsing

A key requirement would be for the AI to be able to identify what a recipe looks like as well as break it down into sections (i.e. ingredients vs. method), as well as subsections (individual steps in method etc.)

On the face of it, this doesn’t look too hard as recipes themselves often have headings like Ingredients and Method, however I’m sure in this case the devil is in the detail. For instance you could start to think of ingredients as having some sort of predefined format {number - unit - ingredient}, however often some sub-quantity of ingredients are used multiple times in the method so you would need to be able to distinguish between many different cases.

As a start I would probably put together some type of web scraper to grab a large number of recipes from sites like BBC Cooking, SeriousEats, Chowhound etc.

Then I might use some kind of supervised learning to begin to break recipes down into their individual elements.

Recipe Storage

Once a recipe has been parsed then it can be stored locally for the user in some type of database structure.

I would look to integrate version control like git so that, for each recipe, the user is able to create and store copies, modifying specific elements. So for instance if the base recipe calls for 1kg of flour for serving 10 people, and you want to save a version scaled down for serving 4 people, then you would be able to store a modified copy.

Similarly if you find that you needed a higher oven temperature, longer cooking time or any other type of change, then you can store your own personalized version of that recipe.

Basic Helper Methods

I consider these to be some of the simpler requirements to implement, that is for things like using voice commands to convert between measurement systems, translate foreign ingredients or even suggest alternative ingredients.

Currently you could probably do much of this with Siri, however in this case I think the problem is far narrower and better-defined (i.e., you don’t need to be able to process all natural language, just very specific commands for executing very specific functions).

So perhaps for conversions you could (at least at the start) use a specific command phrase to indicate that the AI will need to listen out for 1) a number, 2) input unit and 3) output unit.

For being able to suggest alternative ingredients, you could start by creating a large dictionary with common alternatives for harder-to-find ingredients (many cooking sites already have these types of lists).

Integrating with Appliances

While I think that just the recipe-management functionality could be a big help while cooking, it would also be fantastic for this type of AI to help you with controlling kitchen appliances too.

The very futuristic version would be just like Jarvis, with some sort of robotic arm that could move around and actually interact with items in the kitchen.

Obviously this is a far-fetched scenario, however more immediately I am seeing a move by appliance manufacturers to building devices that you can control via apps. For example I was recently given as a present a small Anova Sous Vide machine, which can be controlled from an app including turning it on and off, changing the temperature, setting a timer etc.

Potentially as more appliances emerge with associated apps, then a kitchen AI could use those interfaces to give you the ability to interact with appliances using only voice commands.

Command and Control

A key part of the value of this sort of AI would be the ability to interact with it hands-free.

As I mentioned before, this problem could probably be reduced (somewhat) in complexity by defining specific commands that can be used to activate certain functions. Another gift I received last Christmas was the BB8 robot by Sphero which has a small list of in-built voice activation commands that work surprisingly well.

So similarly you could program it to a list of 5-10 voice commands for things like:

Conversions

Set alarm

Get next step