Cam Peterson

Aviation

National Flight Data Center (NFDC)

https://nfdc.faa.gov/xwiki/bin/view/NFDC/WebHome

FAA Data & Research

http://www.faa.gov/data_research/

Flight Delay Information

http://www.fly.faa.gov/flyfaa/usmap.jsp

FAA Aviation Safety Information Analysis and Sharing (ASIAS)

http://www.asias.faa.gov/pls/apex/f?p=100:1:

Aircraft Situation Display to Industry (ASDI)

http://www.fly.faa.gov/ASDI/asdi.html

NTSB Accident Database & Synopses

http://www.ntsb.gov/aviationquery/index.aspx

OpenFlights.org

http://openflights.org/data.html

The Center for Innovation in Engineering and Science Education Real time data sites

http://www.k12science.org/realtimedatasites.html

MIT Airline Data Project

http://web.mit.edu/airlinedata/www/Revenue&Related.html

Space

Real-Time Space Weather Data Sources

http://space.rice.edu/ISTP/#RT

Politics

Data on the U.S. Congress – A Joint Effort from Brookings and the American Enterprise Institute

http://www.brookings.edu/research/reports/2013/07/vital-statistics-congress-mann-ornstein

Sports

Open Sports Data/API

http://www.openligadb.de/

Football (Soccer) Stats

http://www.11v11.com/

Government

Public Government Data Sets

https://explore.data.gov/catalog/raw/

U.S. Department of Homeland Security Data

http://www.dhs.gov/data

Public Data for the State of Utah

http://www.utah.gov/data/

Compilations by others

Finding Data on the Internet - Inside-R

http://www.inside-r.org/howto/finding-data-internet

Nathan Yau's collection of data sets

http://flowingdata.com/2009/10/01/30-resources-to-find-the-data-you-need/

Dr. Jerry A. Smith's Favorite Data sets

http://datascientistinsights.com/2013/02/02/data-monetization-road-paved-on-top-of-data-sets/

Hilary Mason's "Research Quality" Data-sets

https://bitly.com/bundles/hmason/1 This is a bundle that gathers public data sets that might be interesting to researchers in a variety of fields in one place.

Peter Skomoroch's list of data sets on Delicious

https://delicious.com/pskomoroch/dataset

Data Wrangling blog data set list

http://www.datawrangling.com/some-datasets-available-on-the-web

Other

DonorsChoose.org - Hacking Education: A Contest for Developers and Data Crunchers

http://developer.donorschoose.org/the-data

Datasets for "The Elements of Statistical Learning"

http://www-stat.stanford.edu/~tibs/ElemStatLearn/data.html

Enron Email Dataset

http://www.cs.cmu.edu/~enron/ CALO Project (A Cognitive Assistant that Learns and Organizes). It contains data from about 150 users, mostly senior management of Enron, organized into folders. The corpus contains a total of about 0.5M messages.

Yandex

http://imat-relpred.yandex.ru/en/datasets

The Data Page

http://pages.stern.nyu.edu/~adamodar/New_Home_Page/data.html

Public Data Sets on Amazon

http://aws.amazon.com/publicdatasets/

Miami School of Business Statistical Data Sets

http://www.bus.miami.edu/research-library/free-internet-sites/Statistical%20Data%20Sets.html

Public data put to good use

http://datamob.org/datasets

ASU GeoDA Center Data

http://geodacenter.asu.edu/sdata

UC Irvine Machine Learning Repository

http://archive.ics.uci.edu/ml/index.html http://archive.ics.uci.edu/ml/datasets.html

European Cities 1M Data Sets

http://image.ntua.gr/iva/datasets/ec1m/

University of Edinburgh School of Informatics Data Sets for Data Mining

http://www.inf.ed.ac.uk/teaching/courses/dme/html/datasets0405.html

Opinion Mining, Sentiment Analysis, and Opinion Spam Detection

http://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html

Quandl - Intelligenct search for numerical data

http://www.quandl.com/

Gephi Graph Visualization Sample Data Sets

https://wiki.gephi.org/index.php?title=Datasets

CitiBike, by NYC Bike Share - Station data

http://citibikenyc.com/stations/json

http://www.quora.com/Data/Where-can-I-get-large-datasets-open-to-the-public

Air Quality Notifications

http://feeds.enviroflash.info/

The GDELT Project - Global Database of Events, Language, and Tone

This is just a short demo of MQTT via Clojure. The Clojurewerkz team has produced a great MQTT client called Machine Head. Prerequisites: lein try Start up a REPL by typing:

lein try clojurewerkz/machine_head "1.0.0-beta8"

First we’ll require Machine Head library

(require '[clojurewerkz.machine-head.client :as mh])

Then we’ll set up a connection to the MQTT broker. For this example we are going to use the Mosquitto test server (Mosquitto is an open source MQTT broker).

(def conn (mh/connect "tcp://test.mosquitto.org:1883" (mh/generate-id) {}))

Now all we need to do is publish to a channel. I’m going to publish to the “temp/random” channel. There is a demo application that subscribes to this channel and updates a gauge, based on the value. See it here. The channel is “temp/random”.

(mh/publish conn "temp/random" "23.0")

Now open the demo app in your browser and it should read 23.

(mh/publish conn "temp/random" "-13.0")

It should now read -13

NOTE: This is an open channel, so others could be submitting new values at any time.

I’ve been wanting to play around with the Hoplon (Clojure web application framework) for while now and after prototyping an MQTT based system for work, I decided to bring them together.

The Goal

Use MQTT to push status messages to a Hoplon status page

Overview

This system has three parts:

MQTT Broker

MQTT client publisher (for publishing messages to the broker)

Hoplon status page – MQTT client subscriber (also a client) (for subscribing to the topic and displaying the most recent status message)

MQTT Broker

The Broker is the hub through which all message are sent and received. Sending a message to a broker is called “publishing”. When you publish a message, you publish it to a “topic”. The topic is like a channel. To receive messages, a client “subscribes” to a topic.

In my example I’m going to publish messages to the “hoplon-mqtt-status” topic. You can do some awesome things with topics. Read more about them at http://mqtt.org/wiki/doku.php/topic_format.

For the sake of simplicity, we are going to use test.mosquitto.org as our broker

MQTT client publisher

Since Hoplon is written in Clojure and I’m using the Machine Head library for the MQTT, I derived a simple client from the the Machine Head examples to publish our status to the topic “hoplon-mqtt-status”. I’ll do this in a Clojure REPL with the help of lein try.

Hoplon status page

I’ve modified the Castra Simple example to subscribe to the topic “hoplon-mqtt-status” and display the current status.

Building it

MQTT Broker

Nothing to set up here. The folks at Mosquitto.org provide a test server at test.mosquitto.org. We’ll use the open port 1883.

MQTT client publisher

Make sure you have lein try set up.

Open your terminal

lein try clojurewerkz/machine_head "1.0.0-beta8" - This should give you a Clojure REPL with the Machine Head dependencies

(require '[clojurewerkz.machine-head.client :as mh])

(def conn (mh/connect "tcp://test.mosquitto.org:1883" (mh/generate-id) {})) - Make a connection to the broker

(mh/publish conn "hoplon-mqtt-status" "On") – Publish the message “On” to the topic

(mh/publish conn "hoplon-mqtt-status" "Off") - Publish the message “Off” to the topic

See more examples of Machine Head on Github

Okay, our publishing client is ready in the REPL. On to Hoplon.

Hoplon status page

git clone [email protected]:tailrecursion/hoplon-demos.git - Clone the Hoplon demos repo

cd hoplon-demos/castra-simple/ – Move into the Castra Simple project

Edit build.boot

Remove line 5 – (load-file "../build.util.clj")

Remove line 6 – (require '[build.util :as build])

Update set-env!

(set-env! :repositories #{"https://repo.eclipse.org/content/repositories/paho"} :dependencies '[[tailrecursion/boot.task "2.2.3"] [tailrecursion/hoplon "5.10.14"] [clojurewerkz/machine_head "1.0.0-beta8"]] :out-path "resources/public" :src-paths #{"src"})

Edit src/index.cljs.hl

Change line 4 to: (defc= message rpc/message)

Change line 12 to: (h1 (text "Status: ~{message}"))))

NOTE: watch your parenthesis!

Edit src/app/rpc.cljs

Change line 8 to (defc state {:message "Waiting for status..."})

Change line 12 to (defc state {:random nil})

REALLY! Watch your parenthesis!

Edit src/app/api.clj to look like this:

(ns app.api (:require [tailrecursion.castra :refer [defrpc]] [clojurewerkz.machine-head.client :as mh])) (def message (atom "Initial message...")) (def conn (mh/connect "tcp://test.mosquitto.org:1883" (mh/generate-id))) (defn handle-delivery [^String topic _ ^bytes payload] (swap! message (fn [& x] (String. payload "UTF-8")))) (defrpc get-state [] {:message @message}) (mh/subscribe conn ["hoplon-mqtt-status"] handle-delivery)

Then, in another terminal, run: boot development to start Hoplon development server

Open http://localhost:8000 in your browser.

Sending message

Go back to the Clojure REPL and run (mh/publish conn "hoplon-mqtt-status" "On")

Change the text from "On" to anything you’d like and publish again. Watch as the browser status updates.

Notes

So… this is all a little light on explanation. My first goal was to get the tutorial out there. I’ll be adding explanations as I find more time.

Further Exercises for the Reader

This is just a short demo of MQTT via Ruby. Nicholas Humfrey has produced a great Ruby MQTT client.

Prerequisites:

gem install 'mqtt'

Open up a IRB and require the MQTT library

require 'mqtt'

Define a ‘client’

client = MQTT::Client.connect('test.mosquitto.org')

Now all we need to do is publish to a channel. I’m going to publish to the “temp/random” channel. There is a demo application that subscribes to this channel and updates a gauge, based on the value. See it here. The channel is “temp/random”.

client.publish('temp/random', '20')

Now open the demo app in your browser and it should read 20.

client.publish('temp/random', '-13')

It should now read -13

NOTE: This is an open channel, so others could be submitting new values at any time.