Electric Freefall

Docker can be used in lots of ways as a solution for several different problems, so different that there don't seem to be a consensus on how Docker should be used as part of a development workflow. Here are some flows I've developed while using Docker for webdevelopment (originally written as an answer to this question on Reddit.

Disclaimer: I'm not a docker expert. I'm just figuring out docker as I go along- I've still not completely got my head around volumes and how to manage non-stateless containers. If you have a better process than this please let me know!

#1

First, if you just want to run npm / bower without installing it on the host, you can find a docker image with npm / bower installed (lets say, this one found on docker hub: digitallyseamless/nodejs-bower-grunt, then run this instead of how you would normally run npm:

docker run -v $(pwd)/:/data digitallyseamless/nodejs-bower-grunt npm install

Let's break it down:

-v $(pwd)/:/data

Will mount your current (host) working directory to /mnt inside the container so that npm can actually install files that will remain on the host after the container finishes. Normally the image's readme would describe where to mount your files to, but I think the creators expect this to be used as a base for building an entire app, not a one-off run of npm so they don't explicitly mention /data in the readme. I looked at the image's dockerfile to find that it's using /data.

digitallyseamless/nodejs-bower-grunt

The name of the image to pull & run.

npm install

The command to run in the container. Replace this with whatever you want to run (eg. bower install).

Doing the above allows you to run npm / bower without installing it on the host. Useful for if you need to build on coworkers machines, or on machines with different versions of npm. It's more lightweight (low memory usage) than a fully-blown virtual-machine, starts up quickly, and doesn't need to be kept running (the container will stop as soon as npm completes its task, like npm normally would).

#2

Another use of Docker is to build an image that contains your app for easy deployment. For this, use a Dockerfile starting from the same image above (so we already have node, npm and bower installed in the container). This example assumes you're writing a node program called server.js that listens on port 80. Create a file named Dockerfile in your project's root directory:

FROM digitallyseamless/nodejs-bower-grunt ADD package.json /data/ RUN npm install ADD bower.json /data/ RUN bower install ADD server.js /data/ #ADD any other files required here that are not installed by npm or bower EXPOSE 80 CMD node server.js

Now run `docker build .` and a new image will be built, with all of your npm & bower dependencies installed. You can do `docker run -p 80:80 <hash>` to run it and make it accessible on the host's port 80. Open a browser and point to http://localhost to see it.

This gives you an image that you can test, then push to a docker repository and deploy from there, and know it's identical to what you tested. The downside is that it's slow during development- you have to rebuild the image and re-run every time you make a change. There are tweaks to make sending the build context faster and use layers effectively to avoid a *complete* rebuild every time, but it's still slower than the traditional flow if you've only made a one-line change in your server.js program or added a single dependency which slows down development.

#3

I've launched a new site that allows you to easily convert data from any format to any other format by demonstrating an example conversion.

http://extractbyexample.com

You provide a sample (~5 examples) of your input data (eg. 5 sales rows copied in from excel), and the output data you would like those 5 examples to be converted into (eg. the date and sales quantity in JSON format). The site then uses machine learning to spot what you did (extracted certain fields and converted to JSON) then applies the same steps to your entire input file.

No coding needed. No wrangling with regular expressions or figuring out the python parsedate library. Just provide an example of the conversion which the site can learn from, and it will learn and apply the same conversion to your entire file.

Try it free at http://extractbyexample.com - No signup required.

Changing an Amazon ELB from an SSL on port 443 to plain TCP on port 443, I received this error:

Fetching existing polices for port: 443. Removed listener on port: 443. Create listener on port: 443. Failed to set policy for port: 443. ELBSecurityPolicy-2014-01 can be associated only with a listener with one of HTTPS, SSL as frontend protocol

Angular bootstrap accordions are great for menus and hiding things away, but they really do just hide the options: The elements still exist in the DOM. This means that if your menu is dynamically generated (eg. using ng-repeat) from a model in memory, these hidden DOM elements will still be checked and updated, whenever the model changes. If the accordion is closed, this can be a waste of processing time and slow down your UI, especially if you have a lot of <ng-repeats> inside the accordion.

To alleviate this problem, you can use the ng-if tag to only add the DOM elements to the accordion body when the accordion is open (lazy DOM adding). Use the is-open attribute to the accordion-group tag to specify a variable to keep track of when the accordion is open (eg. <accordion-group is-open="myAccordionIsOpen">) and then an ng-if on the element(s) inside the accordion (eg. ng-if="myAccordionIsOpen") to remove them from the DOM when the accordion is closed.

The final template should look something like this:

<accordion close-others="false"> <accordion-group heading="My Cool Accordion" is-open="openAccordions.coolAccordion"> <label ng-repeat="cool in cools" ng-if="openAccordions.coolAccordion"> <input type="checkbox" ng-model="cool.selected"> <span>{{ cool.name }}</span> </label> </accordion-group> </accordion>

This works, but the is-open variable doesn't get updated until after the "opening" animation has completed, so you see it slide open, finish opening, then the internals appear due to the ng-if evaluating. Since angular-ui's accordion tag doesn't support disabling animations, you need to override the template to change the collapse="!isOpen" attribute to ng-show="isOpen", so that ng-show (with no animation) is used to hide the internal content of the accordion instead of angular-bootstrap's collapse directive. To override the original template for the accordion wherever it is used, add this script tag at the end of your app.html:

<script id="template/accordion/accordion-group.html" type="text/ng-template"> <div class="accordion-group"> <div class="accordion-heading" ><a class="accordion-toggle" ng-click="isOpen = !isOpen" accordion-transclude="heading">{{heading}}</a></div> <div class="accordion-body" ng-show="isOpen"> <div class="accordion-inner" ng-transclude></div> </div> </div> </script>

You shouldn't use this everywhere, because the DOM elements still need to be added when the user clicks on the heading to open the accordion, and that can take some time, and you also lose the animation. But in many cases it saves time elsewhere to keep the DOM small and remove the elements that are not currently visible

While doing some work for Raiteas, I recently needed to connect to a Teradata database from our Ubuntu server using an ODBC connection. Here are the steps (for when I need them later).

I am using Ubuntu 12.04 (also tested on 12.10), 64 bit version. On other operating systems you will need to make adjustments.

Unzip / untar the downloaded file tdodbc__linux_imdep.14.10.00.00-1.tar (http://downloads.teradata.com/download/connectivity/odbc-driver/linux) and also untar each of the 3 components within it to get 3 rpms. Convert these to deb format using Alien, and install using dpkg:

sudo apt-get install alien sudo alien tdodbc-14.10.00.00-1.noarch.rpm sudo alien tdicu-14.10.00.00-1.noarch.rpm sudo alien TeraGSS_linux_x64-14.10.00.06-1.noarch.rpm sudo dpkg -i tdodbc_14.10.00.00-2_all.deb sudo dpkg -i tdicu_14.10.00.00-2_all.deb sudo dpkg -i teragss-linux-x64_14.10.00.06-2_all.deb

Copy and modify the ODBC definitions:

cp /opt/teradata/client/14.10/odbc_64/odbc.ini ~/.odbc.ini cp /opt/teradata/client/14.10/odbc_64/odbcinst.ini ~/.odbcinst.ini

Edit them to replace all "TDODBC_INSTALL_DIR" with "/opt/teradata/client/14.10/odbc_64" and replace "TDODBC_LIB_DIR" with "lib"

Install pyodbc and odbcinst:

sudo apt-get install python-pyodbc odbcinst

Add the teradata files to ldconfig, by creating a new file, /etc/ld.so.conf.d/Zteradata.conf with the contents:

/opt/teradata/client/14.10/odbc_64/lib /opt/teradata/client/14.10/tdicu/lib64 /opt/teradata/teragss/linux-x8664/14.10.00.06/lib

Refresh the cache to include these new lib directories:

sudo ldconfig -v

At this point, the following python test script is useful:

import pyodbc pyodbc.pooling = False conn=pyodbc.connect('DRIVER={Teradata};DBCNAME=87.254.29.172;UID=dbc;PWD=dbc;QUIETMODE=YES;', ANSI=True, autocommit=True) curs=conn.cursor() curs.execute('select current_timestamp') print( curs.fetchone() )

The ANSI=True parameter in the test script is neccessary! Without it you will get an error when you run it with Python:

pyodbc.Error: ('IM002', '[IM002] [DataDirect][ODBC lib] Data source name not found and no default driver specified (0) (SQLDriverConnectW)')

At the moment, running the test script will fail with a cryptic error message:

eaaa[DCTrdt rvr nbet e aao tig

By default, the drivers point to the default (non teradata) files. See and Redirect them:

cd /usr/lib/x86_64-linux-gnu ls -lha | grep odbc (You should see the files below which are to be replaced). sudo mv libodbc.so.1.0.0 Xlibodbc.so.1.0.0 sudo ln -s /opt/teradata/client/14.10/odbc_64/lib/libodbc.so libodbc.so.1.0.0 sudo mv libodbcinst.so.1.0.0 Xlibodbcinst.so.1.0.0 sudo ln -s /opt/teradata/client/14.10/odbc_64/lib/libodbcinst.so libodbcinst.so.1.0.0

At this point the test script will still give an (at least now English) error when run:

pyodbc.Error: ('HY000', '[HY000] [Teradata][ODBC Teradata Driver] Major Status=0x04bd Minor Status=0xe0000007-[terasso]Cannot load TDGSS library. (0) (SQLDriverConnect)')

To fix this:

sudo mkdir /usr/teragss/site sudo cp -R /opt/teradata/teragss/linux-x8664/14.10.00.06/etc/* /usr/teragss/site sudo mkdir /usr/teragss/linux-x8664 sudo mkdir /usr/teragss/linux-x8664/client cd /usr/teragss/linux-x8664/client sudo ln -s /opt/teradata/teragss/linux-x8664/14.10.00.06/lib lib cd /opt/teradata/teragss/linux-x8664/14.10.00.06/bin/ sudo ./run_tdgssconfig

You should now be able to connect using the test script above. There may still be problems if you are using unicode (passing unicode strings as parameters to curs.execute() caused problems that I overcame by converting them to ASCII. There is probably a better solution.

curs.execute( SQL, (someStringParameter.encode('ascii', 'ignore')) )

Harder than counting sheep: Predicting the price of dozing bulls.

After coming in the top 10% of the Stack Overflow competition in October 2012, I recently competed in a team in another Machine Learning competition on Kaggle, this time to predict the auction sale-price of heavy machinery.

The competition provided the details of 400,000 machines at auction, with details like the size, type, weight and features of each machine, when it was manufactured, how much use it had had, where it was auctioned (anonymised), and the final sale price of each machine.The public leader board validation set was made of 40,000 rows with the sale-price hidden. The challenge was to predict the sale-price for these rows, from the non-hidden features.

The first step was to set up a traning / test harness. Kaggle provide a "benchmark solution", so we modified that to split the training set into parts, train on the first part, then test on the second part using 10-fold Cross-Validation. Once we had the benchmark solution working, we had something to test against: We'd run once without any changes, then make a change, run again, and see if there was any improvement in the error-score.

To give you an idea of things that we tried over the month that we competed, this is a (non-exhaustive) list of the techniques that we tried:

Append the appendix data (an addtional file which was provided, to fix several problems with the originally released data), with more information about each machine. This improved our score (as expected). Although APPENDING it seemed to perform better than just replacing the original "incorrect" columns.

Dummy Coding categorical variables. Create new true / false cols like "size = large" / "size = medium" / "size="small". Perfomed worse than just feeding the original columns with the benchmark coding into a Random Forest.

Random coding of the nominal variables. The benchmark code iterated through the nominal (textual) features giving the first value encountered "0", the subsequent one "1", etc. We tried coding them in a random order to see if it worsened performance. It did, which isn't really surprising since you're destroying data in some sense by removing any effect the order of the rows had on the coding.

Coding based on frequency of the nominal variables. Code the most common value as 0, the next most common 1, etc. Was better than random-coding, but worse than the benchmark coding. I'm still not sure why.

Add "is null-like" columns for each feature. True if the value was null, 0, 1000, empty, unknown, etc. Did not improve score.

Principal Component Analysis / Multiple Correspondance Analysis to try to reduce the number of attributes to just the most important ones. Performed worse than just using the original columns (although, for a given N components, (with N relatively small), generally performed better than any subset of N columns. But there was never a great reason to keep the number of columns down, so this wasn't that much use.)

Forward selection on columns. Selected a few columns, but was not as good as feeding in ALL the columns and letting the random-forest decide.

Split "ProductClassDesc" into 2 features (split on the "-"). Improved score.

Treat datasource and auctioneerID as nominal values, not integers, and code as above. Performed worse.

Add columns: "Age at auction", "years since manufacture", "appendix differs (true/false?)". Did not significantly affect perfomance.

Remove invalid values. Set values that were obviously wrong (like those machines that were sold before they were manufactured) to 0. Did not significantly affect perfomance.

Remove invalid rows. Remove any rows with invalid values from the training set. Worsened perfomance. Presumably because these rows still appeared in the test-set, but we now had never seen "invalid" values before.

Bagging - Train on 10 different subsets of the training data (90% of the training data at each round), to produce 10 different models. Each model ran over the test/validation set and the results were averaged (mean) to produce the final submission. Not sure if this improved performance or not. Untested.

Using different models than the random forest. Nothing better found.

Tuning Random Forest parmeters: Nothing better than default found.

Using external data to augment the data we had. No useful external data found.

Creating additional features from the data. Nothing found that improved the RMSLE.

Since most of these attempts didn't improve our score, In the end, we basically ran the benchmark code on the training set + Appendix, with a couple of extra features, more trees, and some of the less important features removed.

The final set of features we used (with reported importances from the random forest) were:

('ProductSize', 0.41652172001561916) ('ageatauction', 0.19948251510717771) ('apx_PrimaryLower', 0.13406362618392142) ('fiBaseModel', 0.12148613718214749) ('Coupler_System', 0.051217663198178283) ('apx_MfgYear', 0.021145885459952378) ('apx_fiManufacturerID', 0.0097494283775320933) ('Pushblock', 0.0089972298725232661) ('Engine_Horsepower', 0.008846433447453169) ('apx_ModelID', 0.0042763625747894753) ('apx_PrimaryUpper', 0.0036585522523700344) ('Scarifier', 0.0036413421737741384) ('yearsSinceManufacture', 0.0028904643776582117) ('Blade_Width', 0.0028731453236328121) ('fiSecondaryDesc', 0.0022618679097654356) ('YearMade', 0.0019901130435210253) ('apx_fiManufacturerDesc', 0.0014476744929245078) ('ProdCapacity', 0.0012626674871026706) ('apx_fiSecondaryDesc', 0.0010425325600432653) ('fiProductClassDesc', 0.00093630558400319901) ('fiModelDesc', 0.00056719203319523953) ('apx_fiModelDesc', 0.00029437944727283606) ('SaleYear', 0.00022935103892434739) ('apx_ProdCapacity', 0.00021697456333554102) ('Drive_System', 0.00020643489848706439) ('Enclosure', 0.00010215641151035445) ('Forks', 9.5813103086392245e-05) ('Blade_Extension', 8.0352144873435045e-05) ('Tip_Control', 5.9438105305499288e-05) ('apx_fiBaseModel', 4.5516313762997594e-05) ('apx_fiProductClassDesc', 4.33268809663658e-05) ('fiModelSeries', 3.4079419042772179e-05) ('Enclosure_Type', 3.1262092602450725e-05) ('apx_fiModelSeries', 2.6341749513859476e-05) ('fiModelDescriptor', 2.2807850943366345e-05) ('Ride_Control', 2.2505958064719364e-05) ('SaleMonth', 2.1849036919695667e-05) ('Ripper', 1.8778815914797885e-05) ('Hydraulics', 1.4807454596585348e-05) ('Coupler', 1.2188403143554827e-05) ('apx_fiModelDescriptor', 9.4231488471798423e-06) ('apx_MachineID', 9.3998985325911364e-06) ('state', 6.7549014932989303e-06) ('SaleDay', 5.8738279187689774e-06) ('Blade_Type', 5.4057744430051251e-06) ('auctioneerID', 3.9776719234206763e-06) ('MachineHoursCurrentMeter', 3.9611219174820141e-06) ('Tire_Size', 3.4886586776409558e-06) ('Travel_Controls', 2.0961153892056989e-06) ('Stick_Length', 1.4384744234822175e-06) ('Stick', 1.2726766881212763e-06) ('datasource', 1.1001817337094394e-06) ('apx_ProductGroup', 1.0266089810096326e-06) ('UsageBand', 9.4128083421217395e-07) ('Thumb', 9.2267259440680872e-07) ('Transmission', 9.040701186308483e-07) ('Backhoe_Mounting', 8.1541235667766097e-07) ('ProdClass', 7.9739045406991217e-07) ('Undercarriage_Pad_Width', 6.9253297534224442e-07) ('Differential_Type', 4.7160145738810293e-07) ('Grouser_Type', 4.1470891159320132e-07) ('appendixdiffers', 3.8711509850675122e-07) ('Track_Type', 3.4502876528014946e-07) ('Pad_Type', 3.3943373773663519e-07) ('apx_PrimarySizeBasis', 1.6448148840859249e-07) ('Turbocharged', 1.1104579477061898e-07) ('Steering_Controls', 1.0653425857162143e-07) ('Pattern_Changer', 9.3398440962281264e-08) ('ProductGroupDesc', 3.7256255581379795e-08) ('apx_ProductGroupDesc', 8.6086322380585466e-09) ('Grouser_Tracks', 3.326743211015174e-09) ('Hydraulics_Flow', 2.6945620128168258e-09) ('MachineID', 0.0) ('Transmission', 0.0)

We got a RMSLE of 0.26231 (beating the benchmark of 0.26704 but not approaching the winning score of 0.22910).

Unfortunately, this scattergun approach is the best way I know of going about these competitions. I attacked the Stack Overflow competition in essentially the same way (kept adding more and more features, and trying different techniques), but just had more luck there. I think with experience it should be possible to learn which techniques work best in which scenarios, but without being taught what works well where, or being in a team learning these things from people who know what they are doing, it really takes a long time to attempt to try all these different methods.

tldr: us-east-1 has the best ping to graph.facebook.com.

I needed to move my Facebook Application (Friend Wheel) to Amazon's AWS EC2 hosting, and wanted to find out which was the best region to host it in. us-east (N. Virginia) is cheapest, but as my application makes a lot of calls to the Facebook Graph API, latency is crucial: 20calls * 50ms slower per call = 1 second of slowness. As part of my reason for moving was to increase the speed of the app and ideally make it work in real-time, a second of slowness is pretty important.

Ping from my home computer (located in Bath, UK) ~22ms:

H:\>ping graph.facebook.com Pinging star.c10r.facebook.com [31.13.72.55] with 32 bytes of data: Reply from 31.13.72.55: bytes=32 time=29ms TTL=249 Reply from 31.13.72.55: bytes=32 time=22ms TTL=249 Reply from 31.13.72.55: bytes=32 time=22ms TTL=249 Reply from 31.13.72.55: bytes=32 time=22ms TTL=249

Ping from my Dreamhost server (somewhere in California I think) ~92ms:

[ps40433]$ ping graph.facebook.com PING star.c10r.facebook.com (69.171.229.25) 56(84) bytes of data. 64 bytes from edge-star-ecmp-06-prn1.facebook.com (69.171.229.25): icmp_seq=1 ttl=239 time=92.9 ms 64 bytes from edge-star-ecmp-06-prn1.facebook.com (69.171.229.25): icmp_seq=2 ttl=239 time=92.8 ms 64 bytes from edge-star-ecmp-06-prn1.facebook.com (69.171.229.25): icmp_seq=3 ttl=239 time=92.7 ms 64 bytes from edge-star-ecmp-06-prn1.facebook.com (69.171.229.25): icmp_seq=4 ttl=239 time=92.9 ms

Ping from an instance in us-west-1 (North California) ~22ms:

ubuntu@ip-10-170-139-121:~$ ping graph.facebook.com PING star.c10r.facebook.com (69.171.237.20) 56(84) bytes of data. 64 bytes from edge-star-ecmp-12-prn1.facebook.com (69.171.237.20): icmp_req=1 ttl=244 time=21.5 ms 64 bytes from edge-star-ecmp-12-prn1.facebook.com (69.171.237.20): icmp_req=2 ttl=244 time=21.8 ms 64 bytes from edge-star-ecmp-12-prn1.facebook.com (69.171.237.20): icmp_req=3 ttl=244 time=21.8 ms 64 bytes from edge-star-ecmp-12-prn1.facebook.com (69.171.237.20): icmp_req=4 ttl=244 time=21.9 ms

Ping from an instance in us-east-1 (North Virginia) ~2ms:

ubuntu@domU-12-31-39-0C-0A-20:~$ ping graph.facebook.com PING star.c10r.facebook.com (66.220.158.27) 56(84) bytes of data. 64 bytes from edge-star-ecmp-01-ash4.facebook.com (66.220.158.27): icmp_req=1 ttl=241 time=1.31 ms 64 bytes from edge-star-ecmp-01-ash4.facebook.com (66.220.158.27): icmp_req=2 ttl=241 time=2.44 ms 64 bytes from edge-star-ecmp-01-ash4.facebook.com (66.220.158.27): icmp_req=3 ttl=241 time=1.35 ms 64 bytes from edge-star-ecmp-01-ash4.facebook.com (66.220.158.27): icmp_req=4 ttl=241 time=2.04 ms

BigML is a web-service which can be used to analyse data. Their beta has been open to the public for a few months, but so far I have completed their tutorial but not experimented any further. I uploaded the 1% sample file to BigML. It took a little while to upload 35MB on my connection. If I want to use it on bigger / unsampled data, I may need to use an AWS instance to do it so that I’m not hampered by my home internet connection.

After uploading, the next step with BigML is to create a “dataset” from the “datasource”. A dataset seems to be a view on the data. BigML correctly parsed the data file and identified “numeric”, “text”, “category” etc, for most of the columns. It selected “text” for the date-formatted columns but there was no more suitable option, and I left the postId and OwnerUserId columns as numeric, but I’m not entirely sure they should be since they don’t really signify numeric values: They are simply IDs. Mousing over the “Create dataset” button showed a pricetag with a price of “30”. Luckily this turned out not to be priced in dollars, but “Credits” which cost 5 cents each. The real price would be 30*0.05 = $1.50, and the first 700 credits are free anyway, so I continued.

After the dataset was created, BigML do a very good job of summarising your data. There are columns to show you the number of rows, the maximum and minimum in each column, and a histogram of how the values are distributed, with informative rollovers. It’s also very fast and snappy. If I’d tried to find the maximum value of a column in Excel, it would have taken a long time to complete. BigML created the dataset asynchronously in under 30 seconds and once complete, all the data is there. It even starts to show you the summary while it’s still processing some of the rows. I could instantly see that of the 33,704 rows in the sample, 33,003 were “Open”, 305 were “not a real question”, 168 were “off topic”, 161 were “not constructive” and 67 were “too localized”.

An odd result was that it seemed to completely ignore any of the columns that I specified as “text” or “category” (with the exception of the OpenStatus column). I guess this is because it doesn’t have any capabilities to process text, but I think it could probably have done something with the dates, even if it was just converting them to timestamps so they could be analysed by the model. I feel that it would be good to have an attempt at parsing the text columns into something useful (ie. numerical metrics), but there are a lot of different ways to do this, and I guess BigML are leaving it up to the user to do this kind of data transformation before uploading your data source to them. This is the area that I think automated machine learning as a service will really start to take off over the next few years, but I’m getting ahead of myself.

BigML gives a lot of non-descriptive error messages.

Once you have a dataset with the columns you want to include, and the target column set to “OpenStatus”, the next step is to train a model. I set the holdout option to 10% of the training set. I’m not really sure what this did: There doesn’t seem to be anywhere to test the resultant model on this holdout set after the model is generated. Their FAQ states that “Currently we don’t offer tools to assess the accuracy of your model. It is high on our to-do list!”. The other options were simple to choose, and I was informed it would cost 150 credits to create my model.

I created a dataset from the same datasource containing only 3 fields, hoping that maybe the problem was with the text columns or strange formatting. Oddly, this restricted dataset still used up 35MB of my allowance and cost the same number of credits to create it.

I seemed to be being charged for the failed models (hopefully this doesn’t happen once you’re outside the free tier!?), so I tested creating a model from only 1000 rows as this uses less credits (~5 credits or $0.25, rather than 150 credits ($7.50) ), then scaled back up to the full dataset once I had a dataset I could successfully build a model on. As it happened, cutting out the textual columns from the dataset fixed the non-specific error and the model generated successfully.

BigML currently only creates “Tree” type models. They state that they intend to add more model types, but they seem to be focusing on making sure that the models generated are easily understandable and can be used by people who don’t have a background in Machine Learning or statistics. The tree model generated in the tutorial is shown below.

This model attempts to predict a person’s income (either < 50K or > 50K per year) based on other information about the person, eg. Their job, relationship status, etc. To use the model, you start at the top (root) of the tree and treat it similarly to a flow chart, answering the questions and following the lines until you end up at a node marked “>50K” or “<50K”, and that is your answer. This is quite a powerful model when the target can be predicted by asking a series of multiple-choice (often Yes / No) questions, and is certainly easy to understand & use by a novice. Here is the model that BigML learned based on my 3-attribute dataset:

You can see that there are no questions to answer. The model predicts “open” for all classes. This is actually a pretty good guess: Guessing “open” all the time is actually quite a good strategy, since 95% of the questions really are “open”. In the benchmark files provided by Stack Overflow, they also predict “open” as the most probable class for every question, only the % confidence changes between different questions. I suspect this might be a common problem with other models (the data really is in favour of predicting “open” all the time), and that I’ll need some way of either making the classes equal (by sampling) and adding a bias towards the “open” class afterwards, or I’ll need a model that provides confidences for each class, which can be used to form a prediction file.

Unfortunately, BigML does not provide confidences. Their tree simply returns a predicted class (“open”), with no indication of how confident we should be that the question actually belongs to the “open” class.

My Thoughts on BigML

I am not convinced that BigML’s focus purely on a tree model is a good idea. Tree models alone cannot deal with all tasks that Machine Learning is used to solve (aside from the Stack Overflow confidences problem I have, it also can’t predict values numerical values like “income” without restricting them to multiple-choice (eg. “>50K” or “

It’s also quite easy to generate a tree-model offline using WEKA, RapidMiner or various other libraries, or even to write one yourself using nested if statements- BigML is simply a browser based way of generating it. An offline method means that you don’t need to upload all of your training data over the internet, have no limits on the number of predictions / models you can make, and you don’t require an internet connection to make predictions. BigML actually offer a downloadable version of the tree model as python code. As they offer this, I don’t see why anyone would ever use their API to make predictions: cloud computing is hardly necessary to make a prediction by evaluating a series of simple “if” statements!

On the positive side, the interface is very slick (except for the errors), and as it’s a cloud solution you don’t need a powerful computer to train a model. It coped with my 35MB file well, and the tutorial, tooltips and free tier make it very easy to get started. Training a simple model has the advantage that not many parameters need to be set, so it’s easier to use than some offline libraries that can require more configuration.

I think BigML are 90% of the way there. They have the infrastructure and a solid API and framework, now they just need to add more ML. I think that the strength of a web-based service like BigML should be that the user can send over the data, train a model, and then request predictions. The details of the model do not need to be understood or even known about by the user: they can just use it as a black-box to make predictions. By restricting themselves to tree-models, BigML are severely restricting the number of applications that their service can be used for. That said, they say that they plan to add more models (Naïve Bayes etc.) and perhaps with those models added and selected automatically, it will start to serve a useful purpose. BigML is essentially a very shiny, easy to use “tree generator”. This screenshot shows the end result, an offline python copy of the trained tree model. Once you have this, you no longer need BigML.

Tiberius looks interesting. The name reminds me of Command & Conquer...

Reading through the forums and posts about machine learning, Kaggle and the Netflix Prize, I found the application “Tiberius” which looks promising and I’ve downloaded for investigation later. It also seems that a lot of statisticians use R. I’ve been introduced to R before, although my main memory is of my friends on Statistics courses at University hating it. I’ll look into it later.

For now, I'll stick with Python... I ran all of the other python scripts that are provided by Kaggle & Stack Overflow & figured out what they do:

basic_benchmark.py, prior_benchmark.py and uniform_benchmark.py generate the x_benchmark.csv files (files of predictions for the leaderboard test set) which can be submitted to Kaggle. They can be adjusted to produce predictions for other test-sets by changing the filename.

split_train.py splits the training data into train-A.csv, which contains all of the training data prior to 18th July 2012, and train-B.csv which contains any training rows with a date after 18th July 2012 (2 weeks' worth, or around 2.3% of the rows). This takes a long time to complete on the full train.csv.

sample_train.py creates a sample file which contains ALL of the closed questions, and an equal number of open questions. This means that the proportions of open:closed questions in the sample set is very different to the proportions in the real train.csv file.

Python & Machine Learning?

Since we only get 2 submissions per day to Kaggle, it seems like a good idea to create a script to test locally and attempt to calculate the Multiclass Loss without needing to submit to Kaggle every time. I ported the calculation from the wiki (see yesterday's post) and my Excel implementation into Python. It takes both the test-set and a file of predictions, and from this performs and prints the log loss calculation.

To test it, I separated train-sample.csv into train-sample-A.csv (containing ~97.7% of the rows) and train-sample-B.csv (containing 4976 rows) using the split_train.py script. I then ran uniform_benchmark.py on train-sample-B.csv to create a uniform_benchmark.csv file containing 4976 rows which looked like this:

0.20.20.20.20.2

Running my Multiclass Loss calculator script over this resulted in a calculated Multiclass loss of 1.60943, which is very close to the benchmark that Kaggle reported for the uniform_benchmark on the public_leaderboard test set. It looks like my script works!

Portions for Pros

I tried running the prior_benchmark and basic_benchmark on the train-sample-B dataset, but received Multiclass Loss scores greater than 1, ie. Nowhere near the scores these benchmarks received on the Kaggle public_leaderboard dataset. I traced this to the problem I mentioned above- The proportion of open:closed questions in the sample file is very different to that of the complete training data. In the complete dataset, 98% of the questions are “Open” (the actual ratio is 3300392 : 70136). This makes it much easier to predict the OpenStatus column! I created a new 1% sample from the train.csv file, which kept exactly 1% of the open questions and 1% of the closed questions, and split that into train-sample-realproportions-1percent-A.csv and train-sample-realproportions-1percent-B.csv using the split_train.py script.

Training and testing on a 1% sample with the correct open:closed proportion (777 test-set rows post 18th July):

Prior Benchmark Multiclass Loss = 0.240672184324

Basic Benchmark Multiclass Loss = 0.278314089594

Uniform Benchmark Multiclass Loss = 1.60943791243

Training and testing on a 10% sample with the correct open:closed proportion (7754 test-set rows post 18th July):

Prior Benchmark Multiclass Loss = 0.240166843047

Basic Benchmark Multiclass Loss = 0.257527936857

Uniform Benchmark Multiclass Loss = 1.60943791243

These are worse than the benchmarks shown on the leaderboard by around 0.02, which I suspect is due to the smaller training set. I also note that the Basic benchmark scores are worse than the prior benchmark scores: Possibly I incorrectly modified the basic_benchmark.py script which actually trains the model and predicts the scores?

Training and testing on the entire dataset (not a sample) (77862 test-set rows post 18th July, a similar number to those in the public_leaderboard.csv sample):

Prior Benchmark Multiclass Loss = 0.242508475684

Basic Benchmark Multiclass Loss = Did not finish.

The disadvantage of using the entire dataset is that it takes a very long time to do all of the operations. Splitting the dataset into train-A.csv (for training) and train-B.csv (for testing) alone took 20 minutes. I left the basic_benchmark.py script to train a model and make predictions running overnight, and came back to a “TypeError: object of type 'float' has no len()”. I didn’t fancy debugging someone else's script that took the best part of an hour to run, so I left it at that.

Since the scores don’t seem to vary too much depending on whether we’re running on a 1%, 10% or 100% sample, I imagine I’ll use the 1% (or smaller) samples to check that my code runs and works correctly, then use the 10% sample to estimate performance, and only run it on the full training set when there’s a candidate for submission to Kaggle.

Stack Overflow has millions of questions and answers: Some good, some bad. This visualization is Stack Tunnel, my entry to the last SO contest.

Stack Overflow, the programming Q&A site, have started a new competition to use Machine Learning to identify bad questions that should be closed. I enjoyed the last competition they ran (I used their API and learnt WebGL to create Stack Tunnel). I’m interested in Machine Learning, and my PHP contracting work is going to be very light for the next few weeks, so this seems like an ideal way to improve my skills, and also test out several new ML products I’ve been meaning to try recently- particularly BigML and Prior Knowledge, 2 web-based APIs for automatically creating predictive models from data without needing to know detail about how the models work underneath.

The competition is being run through Kaggle, a Machine Learning competitions website where “data scientists” compete to produce the best predictions for the given data set. The most famous of these competitions (although run before Kaggle existed) was the Netflix Prize where competitors attempted to predict the scores (number of stars) Netflix customers would give to films (movies!) they had rented. The top prize was 1 Million dollars, and was collected after 3 years of fierce competition by team “BellKor’s Pragmatic Chaos” on September 21st 2009.

The Stack Overflow competition has a top prize of $11,000 (somewhat less than a million), and will only run until October 9th. I don’t expect to win (or even place) but as my first competition it will be good to get a feel for how these things work and learn more about what is needed to create a good model. I plan to document my progress on this blog, and hopefully it will be useful for other people interested in starting to use Kaggle / Machine Learning to avoid my pitfalls, or for the developers of Kaggle / BigML / Prior Knowledge / RapidMiner to see where I got stuck & what is unclear about their services from my point of view as an amateur data scientist.

Hello Wo.............................rld

To get started, I signed up for Kaggle, read through the rules, briefly looked at the leaderboard and forums, read a few blog posts / articles on Kaggle (the idea of the site seems brilliant- I can’t believe I’d not heard of it before!), and downloaded the trial_sample.csv file to get a feel for what the data we have to work with looks like.

Opening the trial_sample.csv file in Excel worked… sort of. It took ~30 seconds to open the file, then was quite slow to respond at times. I’m running Windows 7 on a 2.8GHz Core i7 with 8GB RAM and a solid-state drive (in short: A good spec, 3-month-old £900 laptop) and Excel was struggling with this *sample* file. Not even the full training set. I think I may need to install some programs which are made for opening large files. The standard Linux programs (cat, head, tail, vim etc.) will certainly be fine, but it looks like a lot of the data manipulation might need to be done from the command line or programmatically.

Anyway, Excel worked well enough to view the data. It’s formatted as I expected: One row for each question, with the question title, text, some meta data about the question (eg. The date it was posted) and the user asking the question (eg. Their Stack Overflow reputation at the time they asked the question). There is only one file, one “table” in the train_sample.csv file- Stack Overflow have not provided full user-information or category/tag-information in separate tables that need to be joined to the list of questions. I think this is good: having several linked files was the first thing about the Netflix Prize that I thought made it difficult to start working with the data, and required you to first import the data into some kind of relational database (MySQL) or merge the two files in order to feed all of the information into a model-builder to produce a model. If you want access to this extra data, it is possible to download the dump of Stack Overflow (a file that is 6.2GB before it is unzipped!) but it appears that this data will "not be available as inputs" which presumably means it can't be used as part of the training data anyway (there's a yet-unanswered forum post asking exactly what outside data is permitted).

Excel showing the data. You can see our target column "OpenStatus" on the right- This is what we have to try to predict from the other columns.

The College of Atlantis: Multiclass Loss

I looked at the calculation for the Multiclass Loss, the metric which is used to determine who is winning and the places on the leaderboard.

The logloss formula used to calculate the scores for the leaderboard

This wiki page describes it. I recreated the calculation in Excel, and amended the wiki page to make the details of the calculation more obvious, and explicitly note that row 7 is not used in the example calculation (a detail that threw me, as I was dividing by 7 rather than 6 for the 1/N part of the calculation, and hence not getting the correct result). The multiclass loss function is designed to give high penalties to predictions that are both wrong, and confident. Predicting “It’s definitely A” when it turns out to be “B” is punished much more harshly than saying “It might be A”. This comes from the log of the prediction for the true value (here, “B”): log(a small number, say 0.01) will be much larger than log(a value close to 1, say 0.99). Since the prediction rows must sum to 1, a prediction of “it’s definitely A” looks like row 1, and “it might be A” looks like row 2:

AB 0.990.010.60.4

If B is the correct value, the first row adds log(0.01) = 4.605 to the sum, but the second row adds only log(0.4) = 0.916 to the inner sum of the calculation, helping to keep the overall score down.

import stupidprediction

I downloaded the rest of the data files, determined what the basic_benchmark, prior_benchmark and uniform_benchmark are: They are example predictions which can be submitted. I installed the python prerequisites for the provided Stack Overflow python scripts and started to look at them. I’ve barely used Python before, but I managed to run uniform_benchmark.py to generate the uniform_benchmark.csv file. The uniform_benchmark predictor predicts equal probabilities for each of the possible OpenStatus classes. A prediction for 4 different questions looks like this:

not a real question not constructive off topic open too localized 0.20.20.20.20.20.20.20.20.20.20.20.20.20.20.20.20.20.20.20.2

Since there are 73,290 questions in the public_leaderboard test set, the uniform_benchmark.csv file contained 73,290 rows of this! Not particularly clever, but it’s a prediction nevertheless. I submitted this to Kaggle to make sure I was correct about how the leaderboard worked and this “stupid” prediction achieved a Multiclass Loss of 1.60944, identical to the “Uniform benchmark” row on the leaderboard (as expected, since it’s the same file).

When you post a link to Facebook, Facebook makes a call to the URL you post and pulls details such as the title of the page and the image for the post. 

This Facebook crawler can be fickle and not pull images or title text correctly from your website. Facebook provide the Facebook Debugger (previously Facebook Linter), but it will often return "Error Parsing URL Error parsing input URL, no data was scraped.", an exceptionally useful error message which doesn't even make grammatical sense and gives no clue as to how to fix the error. They also allow you to see exactly what the scraper sees for your URL by visiting http://developers.facebook.com/tools/debug/og/echo?q=https%3A%2F%2Fmydomain.com%2Fmypage.htm, but this will return "Document returned no data" if there is a problem. Another useful message.

How's it work?

When the Facebook debugger attempts to access your page, your Apache access.log record will look like this:

66.220.153.248 - - [11/Apr/2012:23:06:53 +0000] "GET /path/to/page.htm HTTP/1.1" 206 1893 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"

And the HTTP headers passed are:

GET /path/to/page.htm HTTP/1.1 host mydomain.com Accept */* Accept-Encoding deflate, gzip Range bytes=0-40960 User-Agent facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

An unusual point is that it requests only the first 40KB of the page (Range bytes=0-40960), so Apache will respond with a 206 HTTP status code (Partial Content), rather than a 200 (OK). This isn't an error, and the Facebook parser will expect and handle it with no problems. If your page is more than 40KB then you may hit problems. My page was under 40KB and I haven't tested, but I'd guess if your Open-Graph, Title, Meta or Image tags are after those 40KB they will be ignored. As an aside, 40KB is rather large for a HTML page, so you might want to look into shrinking that to speed up page loading.

Invalid SSL Certificates

An invalid SSL certificate may stop Facebook crawling your URL. It is possible for the SSL certificate to be found to be trusted by your browser, but for Facebook to fail to authenicate it. In my case, this happened because Apache wasn't sending the intermediate certificate. A fresh install of Firefox or wget also failed to authenticate this. Wget gave this error:

tom@ubuntu:/mnt/hgfs/workspace/SCRATCH$ wget https://mydomain.com --2012-04-11 19:52:54-- https://mydomain.com/ Resolving mydomain.com... 123.123.123.123 Connecting to mydomain.com|123.123.123.123|:443... connected. ERROR: cannot verify mydomain.com's certificate, issued by `/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=PositiveSSL CA 2': Unable to locally verify the issuer's authority. To connect to mydomain.com insecurely, use `--no-check-certificate'.

Firefox gave me an error "sec_error_unknown_issuer". What's happening here is that Firefox and Wget are verifying that someone (in this case COMODO CA Limited from Salford, Greater Manchester) has certified that I own mydomain.com, but it doesn't know who COMODO is, so doesn't know if it should trust them or not.

To fix this, your server needs to add an additional certificate, certifying that COMODO is a trusted key signer, and that they can be trusted to certify that I own mydomain.com. This additional certificate is called an "Intermediate Certificate" and contains the trust links from an organisation that the browser knows is trusted (such as Verisign) to COMODO. In my case the intermediate certificate was provided in the email with the original issuing of my mydomain.com certificate, and availiable from the Dreamhost "Secure Certificates" section (I ordered the secure certificate through Dreamhost).

On Apache, find your intermediate certificate and add the following line to your httpd.conf or virtualhost file:

SSLCertificateChainFile /path/to/comodo-intermediate-ca.crt

You can then retry with your fresh install of firefox or wget which should now work without any warnings.

Even an image on a site with an invalid SSL certificate can stop the page from loading in the  Facebook Debugger. I created a page with only a <img> tag pointing at an image on the same domain, and the image tag was what caused it to fail. Once the chain certificate was added to the domain, the page loaded in the debugged with no problems.

Canonical HTML tag

The second issue that I encountered with Facebook's scraper was that Wordpress (or Wordpress' framework Genesis) was adding a "canonical" HTML tag to the page for SEO reasons:

<link rel="canonical" href="https://mydomain.com/mypage" />

This meant that although I was trying to share https://mydomain.com/mypage?userid=12345, the Facebook crawler was hitting that page, reading the "link rel=canonical" tag, and then attempting to share the "canonical" page. This resulted in a second call by the crawler to https://mydomain.com/mypage and it pulled the title and image from that page, rather than the page which was meant to be shared.

I'm guessing that the reason Facebook does this is so that people sharing several similar links (like http://mydomain.com/somearticle?referrer=twitter, http://mydomain.com/somearticle?referrer=facebook and http://mydomain.com/somearticle?cachebreaker=1334186904 ) can all be grouped together and shown as a single story in the news feed (http://mydomain.com/somearticle).

The fix is to remove that tag from your page by editing the page templates to add the following lines. If you're using Genesis, you need to first disable Genesis' tag, which exposes the basic wordpress one, then remove that one too.

/** Remove canonical URL tag */  remove_action('wp_head', 'genesis_canonical', 5);  remove_action('wp_head', 'rel_canonical');