#phirehose

About 2 weeks ago, Twitter promoted their little-known Streaming API to production status and, further to that, have now recommended that all high volume and repeated search queries should migrate to the new API.

In their own words, the streaming API is a better solution because of:

   - Complete corpus search: Search is focused on result set quality and 

   there are no guarantees to return all matching tweets. Complete results 

   are only available on the Streaming API. Search results are increasingly 

   filtered and reordered for relevance. 

   - Lower latency results: From tweet creation to delivery on the API, 

   latency is usually within a second. 

   - Predictable rate limits: Streaming is built upon well-defined elevated 

   access roles so that client rate-limit-avoidance heuristics are eliminated. 

   - Higher peak capacity: During a peak event, when tweets spike, the 

   Streaming API is less likely to fall behind or begin aggressive rate 

   limiting. Furthermore, the risk of a large client peak capacity emergency 

   blacklisting is reduced. 

   - More consistent results: Hosting a continuously updated REST API on a 

   large cluster inevitably leads to temporal result skew due to 

   internal propagation delay. This issue is largely eliminated by long-lived 

   connections. 

   - More efficient: Bandwidth and processing are not wasted 

   on identical results. Also, repetitive and long-tail queries are processed 

   more efficiently in the Streaming architecture. 

   - Improved Search experience: Shifting the heaviest users away from 

   Search should dramatically improve the overall Search experience. Resources 

   can be allocated to the search architecture's strength: historical, complex 

   and high value queries. 

That said, the stream API is harder to use than a traditional REST resource - Rather than firing off a GET request with a search query attached whenever you want some Twitter data, the streaming API works by connecting only once and then consuming the stream (or "drinking from the firehose") continuously (or at least until you want to change your stream configuration).

In the PHP-centric view of the web-world, this is not really compatible with the "normal" way of doing things, ie: being firmly in the world of HTTP requests, cron jobs and short-lived processes. It would be very easy to do a bad job of trying to consume the stream and ending up banned from the service (which it will do if you hammer it with badly configured connections).

Consequently, about 3 months ago I released an alpha version of the Phirehose PHP library for the (then alpha) Twitter Streaming API. It allows you to very easily connect and consume the stream without having to worry about the complexity of connection handling, persistent HTTP connections and filter predicate updates. Consuming the stream is as easy as:

class MyStream extends Phirehose

{

  public function enqueueStatus($status)

  {

    print_r($status);

  }

}

$stream = new MyStream('username', 'password');

$stream->consume();

You still need to understand the basics of running long-lived PHP CLI processes and how to process the stream once you've consumed it but the library does a lot of the messier bits for you that PHP is traditionally fairly bad at and includes some simple examples around handling and processing statuses.

Since release, I've had a few hundred downloads and some feedback that has allowed me to improve the library which is now in production powering a variety of services and I'm now considering it largely stable. If you're looking for a way to integrate with the (now recommended) Twitter Streaming API, it might make things easier for you. 

The API is listed on the Twitter Libraries page or you can find it on Google Code here: http://code.google.com/p/phirehose/

If you're interested in using the streaming API in other languages, I suggest you check out:

http://github.com/joshthecoder/tweepy (Python)

http://gistinc.github.com/TwitterClient/ (Java)

Twitter's Streaming API Documentation is pretty slim for advanced uses. Sure they give you some of the bare the concepts and examples but when it comes time to actually build yourself a scalable application you can find yourself stuck with an inefficient product. I am not going to give you code examples but more discuss certain concepts that will help you succeed.

First of all, these concepts can differ per application and may not apply directly to yours. From my experience, I have discovered three concepts when building an effective application using Twitter's Streaming API.

Use System Daemons:

First note that you don't want the Streaming API being run directly in your Web Application. Cron Jobs work great for scheduled tasks but not for handing a constant stream. In my case I ended up using System_Daemons. They work pretty well and fairly easy to set up in a UNIX environment.

Get out of being Rate Limited:

Option 1: Apply and become a Twitter Partner.

Option 2: If you are wondering how to track more than 400 keywords without becoming a twitter partner. Use your application to authorize more twitter users and use those auth tokens for each 400 or (399 to be safe) keywords being tracked.

Storing Tweets Rapidly:

This concept is directly related to speed and redundancy. First of all if you are tracking a lot of keywords, users, or whatever you may be doing. Twitter is going to be sending those to your application in great abundance and rapidly. In one of my applications we were getting in 1000 tweets a second. Now the key is to handle that amount of data is not to process every tweet right when you get them in. You want to save the tweets coming in as fast as possible and insure that they will be processed and that involves just doing one thing, saving them into a database or file.

In a previous app I was building we were using MongoDB and that worked great. Basically just converted the tweet coming in, into an array and then inserting the array directly into the MongoDB. This was fast and worked great.

Once the tweets are queued you want to processes them. To really benefit from Twitter's Streaming API, you want to process the tweets in the queue as fast as you can. If you can keep the queue count at zero, then you are golden.

Using a queue helps you prevent your application from losing tweets and creates an efficient way for processing tweets.

Conclusion:

Planning is key. Think about how you are going to integrate Twitter's Streaming API. Write clean and efficient code. Speed and redundancy is key.

Notes:

Last but definitely not least, use Phirehose for ease of use.