DataHackathon at Node5
Last week Futurelytics Team in the composition of Jan, Marcel and Radek participated on an event named DataHackathon at Node5 organised by Petr Ocásek from Click2stream who deserves our thanks for covering this event. Hackathon is basically an event in which computer programmers and others are involved in developing software, especially at Data Hackathon solving some data puzzle, processing data, running anomaly analysis on the data etc. For those who want to know more about hackathons there is a site where you can learn more.
Friday
Node5 was fully loaded by 85 registered people who were interested firstly in networking and secondly for planned presentation:
Jan Hřivňák - chief analyst at Futurelytics had a pitch about Gamining study case and BigQuery processing.
Štěpán Bechyňský - Measurement of noise and vibration on the highway D1 and D11 in the Czech Republic using smartphone, external sensors (Arudino technology etc.) and analysing them using Azure, Excel and PowerExcel add-on (MS SQL server is needed on Azure background)
Petr Šimeček from Keboola - quick introduction of BigQuery, Redshift and HP vertical and their advantages and disadvantages
Adam Herout from Click2Stream - computer seeing - video as a data source - counting cars, measuring speed, all that looks nice, but it is still in a development phase. It could be good in future when they will be able to capture more objects with more accuracy
Two guys from ataccama - company presentation of DQ solution. It looks interesting and nice but company pitches etc. should not be part of Hackathons
Jan presenting the Gamining study case and BigQuery processing.
Dinner is served. The rest of the evening was about networking and eating great food from Gazdinka - classic home made genuine food, very tasty :-)
Saturday
Team presentation part:
Each team had to select an area on which to focus during the hackathon. There were a few prepared scenarios and tasks to solve:
Forecast of development of shares based on twitter activity.
Analysing noise and vibration of D1 and D11 highways.
Analysing connection between companies based on public register.
Analysing crime data from 2013.
We chose the last option - analysing crime data from 2013. Some of the other teams brought their own data so there was no competition in the sam area.
We spent a huge amount of time extracting, cleaning and processing the data from mapakriminality.cz. We considered using public data from Czech Police but at the end the first source was much cleaner and let us avoid data quality issues.
Finally during the late evening we successfully extracted all required data, created a flat file to be an input for BigQuery, then we could start analysing data and looking for some anomalies. We ended up exhausted by visiting a genuine Indian restaurant, enjoying some masala chicken and discussion about what awaits us tomorrow.
Sunday
We spent he whole Sunday analysing the data using K-means analysis (clustering and looking for outliers) to see some data anomalies. More about K-means is here. We successfully analysed data and found hundreds of anomalies in crime data.
Some numbers about data:
we processed 325 000 crimes (raw data didn’t contain specific crimes, but was agregated)
data was divided according to specific regions, districts and local police departments interesting data:
- 100% solved - attacks on rescuers, so do not even try it - 0% of solved stealing of aircraft delivery
Tomas Trnka expressed interest in downloading the data we processed and in return offered uploading them into GoodData platform. We used that for a quick visualisation and a review of specific crime locations where some anomalies were found. :)
Winners:
1. sinfin.cz - monitoring people movement in node5 offices using iBeacons, 2. daty.cz - visualization of company connections and their hierarchy including an overview of the owners of respective companies 3. Futurelytics - crime data, anomaly detection using K-means analysis (calculated on AWS with MS SQL server). Nice third place considering we have solved an unknown puzzle in such a short time.
Special Thanks
Gazdinka - great food
Node5 - nice place, keep it growing!
Petr Ocásek - it was great to be part of the event, Petr keep it up! See you next time
Few insights for the next time
We expected some entry to be the subject of hackathon and we were ready to solve some puzzle to compete other teams and show our power :), unfortunately each team had their own puzzle to solve so there was not any competition since the tasks differed.
Networking – it is nice to see some people from Czech Republic and their ignition and commitment to beat and defeat their assignments :)
Hackathon is a great event where a lot of things could be done in a short time :)
