#node.js sockets http

The request Node.js library makes it very easy to manage your application's outbound HTTP traffic. We simply install request:

npm install request

and write a simple script:

var request = require('request'); request.get('http://www.google.co.uk/',function(err,req,body) { console.log("Got data for Google UK"); });

But what happens if we want to fetch a number of pages in parallel? Let's use the lower-level 'http' module built-in to Node.js:

var http = require('http'), start = timestamp(); var options = { hostname: 'www.google.co.uk', port: 80, path: '/', method: 'GET' }; for(var i = 0; i<25; i++) { var req = http.get(options, function(res) { res.on('data', function(chunk) { }); }).on('socket', function(e) { console.log("Socket to Google",timestamp() - start); }); }

You would expect all 25 requests to happen in parallel right?

Socket to Google 0.02200007438659668 Socket to Google 0.024000167846679688 Socket to Google 0.024000167846679688 Socket to Google 0.024000167846679688 Socket to Google 0.024000167846679688 Socket to Google 0.1360001564025879 Socket to Google 0.14300012588500977 Socket to Google 0.16000008583068848 Socket to Google 0.16000008583068848 Socket to Google 0.16100001335144043 Socket to Google 0.21700000762939453 Socket to Google 0.21800017356872559 Socket to Google 0.23200011253356934 Socket to Google 0.2330000400543213 Socket to Google 0.23600006103515625 Socket to Google 0.2740001678466797 Socket to Google 0.27500009536743164 Socket to Google 0.2850000858306885 Socket to Google 0.2910001277923584 Socket to Google 0.312000036239624 Socket to Google 0.38000011444091797 Socket to Google 0.3900001049041748 Socket to Google 0.3970000743865967 Socket to Google 0.39800000190734863 Socket to Google 0.3990001678466797

Not really. They seem to be in batches of 5. Why is that?

It's because Node.js sends all its outgoing HTTP requests through an 'agent' which, by default, is configured to pool requests to the same server and only allow 5 simultaneous requests to one domain. This can be changed:

var http = require('http'); http.globalAgent.maxSockets = 25;

Also, bear in mind that the 'https' module also has its own separate agen, so if you're making outgoing HTTPS requests then this would have to be set separately:

var https = require('https'); https.globalAgent.maxSockets = 25;

The Request module uses the 'http' module under-the-hood so it is subject to same restrictions. To get around it, simply reconfigure http/https before you require the request module: