#restify

In this blog post, I will detail out how to build an complete web application, popularly called Single Page Apps(SPA) these days. At a high level, a SPA is one in which most of the work is done on the client side. Thus moving from page to page does not need full page reloads. It gets it's data from the backend , typically an API server, and renders it client side. 

The stack I will be using is that of my project packageindex which can be found here.

NOTE: All code snippets for this post can be found in this gist. A sample puppet manifest code for deployment can be found here.

The block diagram of the architecture looks like:

The setup above represents one of the hundreds of possible ways of implementing it. Use this as a reference and a starting point, not a bible. Also, for simplicity, all this stuff runs on one machine. Typically in your production apps you may receive much more traffic so it may make sense to run different processes on different machines within a VPC or some other scale out strategy.

Nginx

This is the web server that receives all the requests. It's blazing fast and is fast replacing popular ones like Apache. All traffic to the site is over HTTPS. Requests to port 80 are redirected to port 443 (SSL). In order to set up SSL, you will need to have the appropriate certificates set up. See my older blog post for help with that.

Here is the nginx.conf that will let you get started quickly. It's a pretty standard config. A few things to notice:

A set of security related headers

After ssl termination, all requests are forwarded to Varnish running on localhost.

Varnish

Varnish is a web accelerator a.k.a. web cache. It caches requests and if there is a cache hit, services it directly without having to go to the upstream servers. It greatly improves performance by decreasing the load on your servers. 

Varnish does not do ssl termination. Which is why it is done at nginx. Also, always ensure that varnish is listening on localhost(127.0.0.1) so that it is accessible only to the nginx running on your machine and not on all interfaces(0.0.0.0.)

Varnish is typically configured via the /etc/default/varnish file. I accepted all the defaults. I did add my own vcl file which you can find in the same gist

This varnish vcl tells varnish to cache all css/js and other static files for a day. All request are forwarded to an upstream node process.

Express (Node)

For serving the main site I used Express which is the most popular node base web framework. Using express is easy. Create the following directory structure in your favorite vcs e.g. git at $GIT_REPO:

web/

        express_app.js

        restify_app.js

        routes/

                index.js

                api_module.js

        views/

                index.html

To start the node server:

node web/express_app.js

The express_app.js contains a bunch of url patters matched with the appropriate function to call. So we could match '/' to call routes.index which in-turn would render index.html and return it to the client.

Skeleton code is available in the gist.

One thing to notice is that it also matches a bunch of links of the form /api/v1/.... These are the API requests for data which is returned to the client as JSON. These requests are forwarded to a REST API server which actually returns the JSON data. (In a future post, I will explain how to make this REST API server available to third party devs via access keys/secret keys etc)

Also note that express runs on localhost. 

In our example, the index.html contains angular code which is sent to the client the first time the site is loaded. Every time the client makes a request for data, it is forwarded to the REST API server and the response (in JSON) is sent back to the angular client which renders this data on the browser.

For cookies, the express app can forward any cookies received to the API server as a token header or something, so the REST server can know which requests are authenticated, which are not etc...

Restify REST API Server

The REST server is built using Restify. It essentially maps urls (associated with a HTTP verb) to a function which gets called and returns JSON.

Data Store - Redis

The Restify API server interacts with a data store. You can use anything you are familiar with. We used Redis. The point to note here is that the node express process never interacts with the redis data store. Any data it wishes to read/write from/to the data store is via the restify API server.

Make sure it listens only on localhost or something only the API server can access.

Angular.js client code

This is the client code which makes requests for data and it gets rendered on the browser. Describing the entire setup of angular is beyond the scope of this post, but you could google around for a bunch of tutorials that would teach you to write good client side angular controllers, services, directives etc.

Deployment

You can deploy your codebase to production via Puppet (or Chef). A simple puppet manifest to get you started can be found here. Control of the process is done via supervisord.

In addition to "The Simplest RESTful server with nodejs, restify, and mongodb", I am going to add another server-side script with schema.

It gets simpler when you use schema, but you need to stick to the schema.

The following code is to be used on the server-side of my simple wiki.

In this post I will show you how to set up an RSS feed for your blog or site using Express / Restify.

I have a site that is server via Express. It is a Single Page App that gets all it's data from an API server which runs on Restify. 

Express

First we have to set an appropriate route in Express to serve the feed. I added something like this:

app.get('/feeds/rss2', feeds.feeds);

Thus, whenever anyone visits /feeds/rss2 the feeds.js gets called which makes a query to the API server (to get the feed data) and then renders the xml to be delivered by Express. Here's a gist which shows what my feeds.js / feeds.html looks like.

Restify

The restify part hosts the API server which is queried for the feed data. I have used redis to store the last 25 updated entries. It is really easy using redis. I added a key called 'web:feeds' which is a list. Whenever an item is updated on my site, I add it to this list as well. Then I trim this list size to 25. i.e. I use the operations LPUSH / LTRIM.

In addition to The Simplest RESTful server with nodejs and restify, this is the more practical solution, but still the simplest way to build nodejs, restify, and mongodb.

First, you need to setup mongodb database and insert data into it.

How To Install & Run MongoDB (Mac OS X)

$ brew update $ brew install mongodb $ mkdir -p ~/data/mongodb $ mongod --fork --dbpath ~/data/mongodb/ --logpath /var/log/mongodb.log # START $ mongo --eval "db.shutdownServer()" admin # STOP

How To Import StarBucks Data To Your Mongodb

$ mongoimport -d starbucks -c stores --type csv --file starbucks_locations.csv --headerline $ mongo # to validate import > show dbs > use starbucks > db.stores.find({"Store ID":1006099})

Starbucks store location data is found at https://opendata.socrata.com/Business/All-Starbucks-Locations-in-the-World/xy4y-c4mk

The following is the server code part. You may need to install mongoose npm install mongoose if you did not.

var restify = require('restify'), // RESTful API Web Server mongoose = require('mongoose'); // mongodb connection /** * config server */ var server = restify.createServer(); server.use(restify.fullResponse()); // enable default headers server.use(restify.bodyParser()); // remap body contentnt of a request to the req.params server.use(restify.queryParser()); // remap query string to req.params /** * config database(starbucks) and model(Store) with collection(stores) */ mongoose.connect('mongodb://localhost/starbucks'); var db = mongoose.connection; db.on('error', function() { console.error('connection error:');} ); db.once('open', function() { console.log('connected to database');} ); // schemaless structure with no restriction to attributes var StoreSchema = mongoose.Schema({any:{}}, {strict:false}) var Store = mongoose.model('stores', StoreSchema); // $ curl -X GET http://localhost:8080/starbucks?country=AE server.get('/starbucks', function(req, res, next) { var attrs = req.params; Store.find(req.params, function(err, stores) { if(err) res.send(500, {error: err}); else res.send(200, stores); return next(); }); return next(); }); // $ curl -X GET http://localhost:8080/starbucks/51 server.get('/starbucks/:id', function(req, res, next) { var storeId = parseInt(req.params.id); Store.findOne({"Store ID":storeId}, function(err, store) { if(err || !store) res.send(404, {error: err||'Not Found'}); else res.send(200, store); return next(); }); }); // $ curl -X POST http://localhost:8080/starbucks -d "Store ID=1234567890" -d "Name=Test" server.post('/starbucks', function(req, res, next) { req.params["Store ID"] = parseInt(req.params["Store ID"]); var store = new Store(req.params); store.save(function (err, store_saved) { if(err) res.send(500, {error: err}); else res.send(201, 'created'); return next(); }); }); // $ curl -X POST http://localhost:8080/starbucks/51 --d "Name=Changed" server.put('/starbucks/:id', function(req, res, next) { var storeId = parseInt(req.params.id); delete req.params.id; Store.update({"Store ID":storeId}, req.params, function (err, store_saved) { if(err) res.send(500, {error: err}); else res.send(200, store_saved); }); return next(); }); // $ curl -X DELETE http://localhost:8080/starbucks/51 server.del('/starbucks/:id', function(req, res, next) { var storeId = parseInt(req.params.id); Store.remove({"Store ID":storeId}, function (err) { if(err) res.send(500, {error: err}); else res.send(204, 'Removed'); }); return next(); }); server.listen(8080, function() { console.log('% listening at %s', server.name, server.url); });

Here is some interesting frameworks/packages for building RESTful API with NodeJS

Fortune

Fortune is a web framework for prototyping rich hypermedia APIs that implement the JSON API specification.

Hapi

A rich framework for building web services.

Resify

Restify is a node.js module built specifically to enable you to build correct REST web services.

I'm using Hapi because it has many additional plugins (validations, cache, monitoring etc.) and you can customise it like you want.

Here's a quick how to about using JSONP with Restify.

If you have a single page app, chances are that your app will be making calls to an API server which provides content. Chances are also that your API server will be on a different host:port. Owing to this, you will not be able to make your typical HTTP calls to the API server via js owing to the 'same origin policy' restriction.

To overcome this, you can use JSONP. The <script> tags are not subject to the 'same origin policy' restriction. So the idea is to make a call to the API via the script tag and then the server returns the json wrapped in a callback which can get executed on the page.

But it is not sufficient to have the API server return JSON. It has to return JSONP i.e. the JSON wrapped around a callback supplied by the user.

Here's what you need to add to the testify server:

var restify = require('restify'),

    api = require('./routes/api.js'),

    server = restify.createServer({

      name: 'api_app',

      version: '0.1.0'

    });

//for parsing query strings

server.use(restify.queryParser());

server.use(restify.jsonp());

server.get('/foo', api.bar);

server.listen(8080, 'localhost', function() {

  console.log('%s listening at %s', server.name, server.url);

});

Now in you api.js you can add:

exports.foo = function(req, res, next){

res.send({'hello':'world'});

}

NOTE: you should use res.send and not the shortcut res.json, because the res.send will force a JSON reply.

Now you can call the api url from you website like:

http://somehost:someport/foo?callback=JSONP_CALLBACK

The response from the API server will be something like:

JSON_CALLBACK(

{'hello':'world'}

)

Pleased to announce the release of Restify 0.5.0. This release includes updates to a number of third-party dependencies along with testing with the latest version of Kohana 3.2.2.

Most notable new feature is prettification of JSON. It's very common that JSON responses come back as a compressed single line. Restify now detects JSON and properly formats for easier reading.

Check it out:

GitHub: https://github.com/morgan/kohana-restify

Live Demonstration: http://restify.io/

Demo Video: http://vimeo.com/michealmorgan/restify-overview