#data scrape

So the Natural Language Processing looks to be pretty much there although I still had to manually add a few words to remove - if I have time I’ll go back and fine-tune this to ensure that it works a little better.

After a bit of an oversight I remembered there are legal implications with data scraping so before I continued with my machine learning (ready to delete everything I had just done) I contacted allrecipes.com. They came back with some legal jargon and I replied explaining my purposes and they were all on board. I got a 100% can use the data from them and everything is right with the world. THANK YOU ALLRECIPES.

Looking back... a data scrape, if possible, really is the best way to gather data quickly. Doing this you can get just the information you need, format it in the way you want, run through thousands of web pages a day without the blocks of an API. An API doesn’t necessarily have the newest data, there are blocks on the number of calls you can make limiting said data, and there are often costs associated. Anyways, back to the task at hand...

So after 30 minutes of trying to figure out how to download the libraries and actually get them to work (I finally understand pip and .whp files) I now have sklearn, numpy and scipy. I have also got some methods that will load and split the dataset into training and test data. I haven’t quite figured out what the split ratio should be but I can tweak this when I have more to go off - currently its set at 60% train (50,682), 40% test (33,789). 

Below shows the first 5 in each - this will be randomly generated when it is started.

19/02/16

Finding the minimum ID (the lazy way)

To find the minimum ID for a recipe on allrecipes.com to ensure a quick download of data once created I ran the code from ID 1 writing to file once it found a valid ID. Ideally I would like to grab the maximum ID as well but with some gaps between ID’s this isn’t really possible so I may just encorporate a limit on the number of recipes downloaded instead. This prolonged use of the code flagged a silly mistake I had made - once running too long the site occasionally times out with the connection so I added a failsafe to try for 10 seconds and time itself out and try another ID rather than crashing. 

Here is the moment it found a valid ID:

My struggles with the csv writer

I looked for a few copy and paste tutorials for some csv code writing that I could have a look at and ensure worked. I usually do this to grasp how they work and then completely change them and continue testing it as I go. This way I can figure out where I’m going wrong and refer to a working copy when it all goes horribly horribly wrong.

Unfortunately when I went to copy and paste code that other people have used - using the “csv” library and “csv.writer” function it just did not work.

I then figured that maybe the csv import wasn’t working and went onto download anaconda - this package has SciPy, NumPy and Pandas (libraries I was eventually going to download anyway for the remainder of my project). It also handily has csv as one of its libraries and a nifty new command prompt. But alas I was getting the same errors:

AttributeError: ‘module’ object has no attribute ‘writer’...

I would understand this if I wasn’t following tutorials, stack overflow suggestions and the python docs themselves but I was sure that this was how you wrote this function.

csvout = csv.writer(csvfile, delimiter=‘ ‘...)

csv.writer doesn’t exist according to the Command Prompt and also the Anaconda command prompt and any other methods I attempted. I then looked as to whether it was my version of Python - I’ve already been caught out by that before. So I decided to import the library and have a look at what built-in functions there are:

Writer is there... So what’s the error!? 

Turns out a terribly named file really can cause all sorts of issues and I shall never use csv.py again as long as I shall live. Two hours I won’t get back but I will always use a proper naming convention from now on and its sorted and now I can progress :)

Changing the way it saves to a csv file

Having it save to a text file is great - you can view it exactly how you want to but this really isn’t a good way in which to use it to actually analyse the data. Currently I’m torn between turning it into a csv file or actually putting it into a database. A csv file maybe easier right now but in the long run a database is something I have more experience with and is generally a good way to analyse data. 

Creating a csv file now and attempting it this way will be the route I go down however as turning a csv file into a structured dataset in a database isn’t difficult so I’m really killing two birds with one stone.

How to store the data within the csv file

This really isn’t a simple thing, with there being a list of ingredients per recipe what is best really? I’m going to take a sample of 100 recipes to test out a few different methods. 

<ID>, <Recipe Name>, <Number of Ingredients>, <List of Ingredients Separated>, <Stars>, <Number of ratings>

Two separate csv files

<ID>, <Recipe Name>, <Number of Ingredients>, <Stars>, <Number of ratings>

17/02/16

So looking at a few different tutorials on how best to start using python (being a complete noob) I started by installing and getting Python to work on my machine. I then ran a few Hello World attempts and realised I had downloaded the wrong version and then went ahead and got Python 2.7. This now works! Woo!

API’s - the awesomeness and then the fall

Firstly I looked at APIs (application programming interface), the best and easiest way to grab data is via an API. An API is basically a program (usually a website) that you gain a key for and then you can input your search constraints and it will bring back data in a format that you want - usually JSON. This would be awesome so I ran through a couple of different tutorials via youtube and got the hang of it. 

Then I went on to search for some APIs that would meet the criteria:

One large data scrape of around 200,000 recipes (at least) 

Possible additional data scrapes in the future

A site that has good reviews and an active community

Turns out... Not as easy as I thought, most API’s have large running costs and to be honest I don’t have the money to invest into the project as a student. 

API’s were in turn scrapped as an idea, but I learned how to use them so no time was wasted (positive thinking).

Data Scraping via inner HTML

The backup plan... So API’s weren’t going to work so instead I decided to look at how data scraping worked at a lower level. This looks at the inner HTML within the webpage and then grabs the data from the webpage by targeting the specific areas you want.

Surprisingly this really didn’t take long. I got the lxml/html libraries and the requests library and used the webpage like a tree of information and grabbed the specific contents that were necessary. 

ID of recipe (so that I could check it was correct on the website)

Name of recipe

Ingredients (most important part of the recipe)

Star Rating (the average rating that people felt the recipe deserved)

Number of people who rated the recipe (the number of people who rated it signifying how reliable the star rating is)

This was pretty simple - I limited it to only grabbing one random recipe, randomising the id of the recipe. I then started to notice that the same recipe continued to pop up with different ID’s. This must be a bug within the recipe site or possibly a default recipe for those ID’s that don’t have any information. The recipe that continued to pop up was the Johnsonville® Three Cheese Italian Style Chicken Sausage Skillet Pizza. I decided that due to not wanting any recipes without any reviews on them this wasn’t an issue so I removed all recipes that had yet to be reviewed.

The simple data scrape now works exactly as I wished it to. Output pictured below:

FlowingData

This is the biggest struggle to most projects: STARTING!