#superfeedr

Medium acquires SuperFeedr to make feeds realtime

Medium has acquired a startup which helps to manage and speed up the way users manage their feeds. SuperFeedr manages all kinds of feeds including RSS, Atom and JSON. A service like medium can leverage the technology of SuperFeedr to deliver the feed realtime to the subscribers. SuperFeedr currently works with a few great services like About.me, Etsy, Player Fm and IFTTT. Following the…

Recently I added PubSubHubbub support to my site. It's pretty simple to add, so I thought i'd share it here.

PubSubHubbub is a pub/sub model for the internet. e.g. publishers can update their feeds and then inform a central hub. Subscribers register with the hub and get pinged whenever the published has posted new data. For a more thorough explanation, refer the PubSubHubbub  site.

I will explain how to add support for a rss feed.

Step 1:

You first need to find a hub (or maybe host your own) I used the hub at Superfeedr. Once you register, you will receive your hub url, something like: mydomain.superfeedr.com

Step 2:

You need to augment your rss feed with the url of the hub the publisher will ping once new data appears. For good measure, also add it to the response headers. Here's what I added to my rss feed:

<!-- PubSubHubbub Discovery --> <link rel="hub" href="https://{{ meta.hub }}" xmlns="http://www.w3.org/2005/Atom" /> <link rel="self" href="{{ meta.self }}" xmlns="http://www.w3.org/2005/Atom" /> <!-- End Of PubSubHubbub Discovery -->

Here's how I added new headers (assuming a node/express type framework):

res.links({hub: cfg.hub, self: cfg.transportRequire + '://' + cfg.apiHost + '/feeds/rss2' });

Thus, the first time anyone accesses the feed, the links in the feed (or alternatively in the header), will inform them that the pubsubhubbub protocol is supported and the client can register with the hub url specified in order to get updates.

Step 3:

Subscribe with the hub and provide an appropriate callback to be called each time the publisher updates it's feed.

Step 4:

The publisher needs to ping the hub any time it's feed is updated. Here is a sample gist of a function written in javascript to achieve that.

PuSH is great because instead of downloading the entire feed over and over again, PuSH sends new content to Feedbin making it so articles show up right after they are published. Feedbin already has thousands of articles getting pushed every hour.

Google stellt zum 1. Juli den Google Reader ein. Damit verschwindet ein über viele Jahre zentraler Baustein im Feed/RSS/Atom Ökosystem. Was geschieht eigentlich ab diesem Zeitpunkt mit PubSubHubbub (PuSH) bzw. dem von Google betriebenen Hub, ein weiterer zentraler Baustein dieses Ökosystems?

Zur Erinnerung: PuSH ist ein offenes Protokoll, das die Auslieferung von RSS und Atom Feeds wesentlich beschleunigt, d.h. Blog Beiträge werden z.B. unmittelbar nach Veröffentlichung im Feed Reader angezeigt. PuSH wurde 2009 von Brad Fitzpatrick und Brett Slatkin bei Google im Rahmen eines 20% Projekts entwickelt.

Das Protokoll ist dezentral angelegt, d.h. jeder kann einen Hub betreiben, der die Kommunikation zwischen Publisher - also z.B. Blogbetreiber - und Abonnent übernimmt. Google hatte damals einen eigenen Hub zur Verfügung gestellt, der auch weiterhin noch betrieben wird. Allerdings ist dort schon seit Beginn des Projekts dieser Absatz zu lesen:

PubSubHubbub is just a protocol, not a service, but we’re running this as an open test server for anybody to use to help bootstrap the protocol. Feel free to publish to or subscribe from this hub. You can migrate in the future when you want to run your own hub server, or you can just keep using this one.

Ein Test Server, um das Protokoll zu verbreiten!

Neben dem Google Hub ist mir nur noch der Hub von Superfeedr als öffentlicher, frei zugänglicher Hub bekannt. Dieser kann bis zu einer Grenze von 10.000 Benachrichtigungen frei genutzt werden; mehr Benachrichtigungen kosten (nicht viel) Geld. Den Superfeedr Hub nutzt meines Wissens auch Tumblr; andere Blog Plattformen haben eigene Hubs, z.B. WordPress.com. Für selbst gehostete Blogs bleiben aber eigentlich nur Google und Superfeedr, die auch beide vom besten WordPress.org Plugin zum Thema von Hause aus unterstützt werden.

Zur Klarstellung: Bislang hat Google nicht verkündet, den PuSH Hub aufzugeben. Aber angesichts der Entwicklungen bei Google zum Thema Feeds/RSS/Atom (s.a. Schließung der Feedburner API), sollte man zumindest die Alternative(n) im Auge behalten.

Bloggers love being promoted inside apps like FanPulse!  We also love it when we can depend on some awesome people to write great sport posts to help people keep up with their favorite teams and games.  The problem is, lots of infrastructure needs to be put together on the backend to support the synchronization of our algorithms with the premium content from blogs and news sources.  Superfeedr fortunately has some tricks for me to automate the process of adding and removing sources on the fly.

When a new source is added to our database, I fire off an after filter that subscribes Superfeedr to that feed.  When the source is destroyed, I also unsubscribe with Superfeedr.  The API makes it super easy.  Here are some examples from FanPulse's Rails framework:

Adding a feed to Superfeedr to subscribe to:

1 2 3

url = SUPERFEEDR_URL

private_resource = RestClient::Resource.new url, SUPERFEEDR_LOGIN, SUPERFEEDR_PASSWORD

private_resource.post "hub.topic" => @feed.url, "hub.callback" => SUPERFEEDR_CALLBACK, "hub.verify" => 'sync', "hub.mode" => 'subscribe'

I have a few constants in there that are specific to our app.  

SUPERFEEDR_URL = 'http://superfeedr.com/hubbub'

SUPERFEEDR_LOGIN = my_superfeedr_login_name

SUPERFEEDR_PASSWORD = my_superfeedr_password

SUPERFEEDR_CALLBACK = my_servers_callback_method

I'm using the RestClient gem by adamwiggins to make the calls, as you can see it's pretty darn easy.  "hub.topic" is the url for the feed,  "hub.callback" is the url I want future feeds to be pushed to using webhooks as well as subscribing internally, "hub.verify" should be set to sync since we're not doing an asynchronous call, and lastly "hub.mode" is set to subscribe since we want to add this new feed.

There are a few slight subtleties I did not mention, including the fact that your callback has to implement the basic PubSubHubbub subscribe / unsubscribe spec as usual.  That will have to be in another discussion though.

Lastly, if you want to destroy the source from your database, just make sure you hit the Superfeedr API to unsubscribe the feed as well.

1 2 3

url = SUPERFEEDR_URL

private_resource = RestClient::Resource.new url, SUPERFEEDR_LOGIN, SUPERFEEDR_PASSWORD

private_resource.post "hub.topic" => @feed.url, "hub.callback" => SUPERFEEDR_CALLBACK, "hub.verify" => 'sync', "hub.mode" => 'unsubscribe'