Node.js Hints

Amazon Dash buttons are designed to be a one-click ordering mechanism for Amazon products (toilet roll, pet food, washing powder detc). They are available in the UK for £5, which is refunded to you after your first button press.

How do they work?

They are a battery-powered, wifi enabled device that wakes up when the button press and (presumably) makes an API call back to Amazon headquarters.

How can we make something happen when the button is pressed

If you take a new Dash button out of the box, configure it on your Wifi network but stop configuration at the point where you pick the product, then the button does nothing. But pressing the button does make the button join your network - if you can detect device's Mac address on your network (albeit briefly) then it you can act upon it and perform an action of your own.

Creating the script

I found the node-dash-button npm package to do the job. It has a utility for finding your button's Mac address and a function to allow you to listen for that device on your network. You can then write a simple script to make an API call to IFTTT using the Maker integration when the button is pressed:

source code

replacing the MYMACADDRESS and MYKEY placeholders. Once you're linked up to IFTTT you can configue recipies to act upon the event and perform any action you want.

References

Couchimport is a command-line tool written in Node.js that allows CSV files to be imported into Cloudant/CouchDB, with each row in the source file becoming a document in NoSQL database. A recent addition to its feature set allows the same tool to be used to bulk import JSON data too.

Imagine we want to import crime data from here:

https://data.sfgov.org/resource/tmnf-yvry.json?$select=incidntnum,category,descript,dayofweek,date,pddistrict,resolution&$limit=50

into CouchDB or Cloudant. The format of the data returned by that URL is

[ { "category" : "BURGLARY", "pddistrict" : "RICHMOND", "descript" : "BURGLARY OF APARTMENT HOUSE, UNLAWFUL ENTRY", "dayofweek" : "Monday", "resolution" : "NONE", "date" : "2005-05-16T00:00:00", "incidntnum" : "050546220" }, ... ]

We can use couchimport for this, as it handles streams of JSON data easily:

# define where the data should go export COUCH_URL=http://127.0.0.1:5984 # curl the data and pipe to couchimport, specifying # --db - the name of the database # --type - json data # --jsonpath - which part of the json to import curl 'https://data.sfgov.org/resource/tmnf-yvry.json?$select=incidntnum,category,descript,dayofweek,date,pddistrict,resolution&$limit=50000' | couchimport --db cr --type json --jsonpath “*"

If you have a text file (csv, tsv etc) and need the data in CouchDB, then the most efficient method of inserting the data is using Couch's Bulk Document API. There are no tools to do this, so it is probably that command-line scripts need to be written to do the heavy lifting.

Having written several of these myself, I have had to be careful to avoid memory shortages as the speed of inputting the data exceeds the speed of writing it to the database.

Here is another solution: "CouchImport", a Node.js (npm) script which can import flat csv/tsv data directly into CouchDB using Node's Stream framework. Features:

data expected on stdin

data is broken into lines

each line becomes an object

data is written to Couch in blocks of 500

configuration via environment variables

e.g.

 # import the contents of 'test.tsv' into Couch  cat ~/test.tsv | couchimport

or a more complicated example

 # download US crime data  curl 'http://data.octo.dc.gov/feeds/crime_incidents/archive/crime_incidents_2013_CSV.zip' > crime.zip  # unzip it  unzip crime.zip  # define the destination database  export COUCH_DATABASE="crime_2013"  # define the delimeter as comma  export COUCH_DELIMETER=","  # create a new database  ccurl -X PUT /crime_2013  # import the data  cat crime_incidents_2013_CSV.csv | couchimport

"CouchImport" also allows a custom 'transformation' function to be defined which can change the object before it is written e.g. 

remove unwanted keys

lowercase values

change the data type of keys

CouchDB's "_all_docs" endpoint gets you all the documents in database in this form

e.g

http://mycouchdbserver?all_docs?include_docs=true

{ "total_rows": 93985131, "offset": 0, "rows": [ { "id": "0000230a35e724e12b8c18a8f700065d", "key": "0000230a35e724e12b8c18a8f700065d", "value": { "rev": "1-adf8311047fcdd953543118e7d501fa1" }, "doc": { "_id": "0000230a35e724e12b8c18a8f700065d", "_rev": "1-adf8311047fcdd953543118e7d501fa1", "a": "1", "b": "2", "c": "3" } }, { "id": "0000230a35e724e12b8c18a8f7000ccd", "key": "0000230a35e724e12b8c18a8f7000ccd", "value": { "rev": "1-5ce610ff79bc1cfe62b4a1a68e5b09cf" }, "doc": { "_id": "0000230a35e724e12b8c18a8f7000ccd", "_rev": "1-5ce610ff79bc1cfe62b4a1a68e5b09cf", "a": "2", "b": "5", "c": "6" } } ] }

Notice the documents themselves are contained inside an object inside an array. In real life, the data comes out like this:

{"total_rows":93985131,"offset":0,"rows":[ {"id":"0000230a35e724e12b8c18a8f700065d","key":"0000230a35e724e12b8c18a8f700065d","value":{"rev":"1-adf8311047fcdd953543118e7d501fa1"},"doc":{"_id":"0000230a35e724e12b8c18a8f700065d","_rev":"1-adf8311047fcdd953543118e7d501fa1","a":"1","b":"2","c":"3"}}, {"id":"0000230a35e724e12b8c18a8f7000ccd","key":"0000230a35e724e12b8c18a8f7000ccd","value":{"rev":"1-5ce610ff79bc1cfe62b4a1a68e5b09cf"},"doc":{"_id":"0000230a35e724e12b8c18a8f7000ccd","_rev":"1-5ce610ff79bc1cfe62b4a1a68e5b09cf","a":"2","b":"5","c":"6"}} ]}

with each object on its own line.

If you are wanting to export the data and put it in Redshift, for example, the JSON needs to be in this form:

{"_id":"0000230a35e724e12b8c18a8f700065d","_rev":"1-adf8311047fcdd953543118e7d501fa1","a":"1","b":"2","c":"3"} {"_id":"0000230a35e724e12b8c18a8f7000ccd","_rev":"1-5ce610ff79bc1cfe62b4a1a68e5b09cf","a":"2","b":"5","c":"6"}

Solution 1 - jq

The jq utility allows JSON to be parsed and reformatted on the command-line. e.g.

curl 'http://mycouchdbserver?_all_docs?include_docs=true_' | jq '.rows[].doc'

(Thanks to Cloudant Support for this solution). Unfortunately it is not suitable for large data sets as jq requires all of the data to be in-memory.

Solution 2 - Use this docstream.js utility

DocStream takes _all_docs data in on stdin and outputs just the "doc" section:

curl 'http://mycouchdbserver?_all_docs?include_docs=true_' | node docstream.js'

This is should work with any size of data set, as long as each document appears per line.

e.g.

If you use the command line and you use CouchDB, you probably use the command-line tool 'curl' to interact with your databases. If your CouchDB is hosted, such as the services provided by Cloudant, then to create a document, you will need to perform a command like this:

curl -X POST -H 'Content-type:application/json' 'https://mypassword:[email protected]/database' -d'{"a":1,"b":2}'

Each curl command contains a lot of repeated information:

the content-type header

the protocol

the username/password

the domain name

So I wrote simple utility called "ccurl" (CouchDB Curl) which saves your fingers by storing your CouchDB credentials in an environment variable, meaning that your command now looks like this:

ccurl -X POST /database -d'{"a":1,"b":2}'

It is installed in one line (if you have Node.js) and can be found at

ccurl on Npm

ccurl on Github

If you want to ensure that your Memcached process stays running, and to restart it if it crashes, then you can use the Node.js "Forever" tool to help you.

Install Forever:

npm install -g forever

Create a script to start Memcached (e.g. start_memcache.sh):

#!/bin/bash /usr/local/bin/memcached  -u nobody -m 400 -p 11211

Make it executable

chmod +x start_memcache.sh

Run it using Forever:

If you have a Secret Santa draw going in your office, and you need to do the draw to work out which presents are going to whom, then you can either write down names on bits of paper and mess about with hats, or use the "secretsanta.js" command-line utility.

Simply download the tool here and run it as follows with the names of the people in the draw:

node secretsanta.js Rita Sue Bob "Alan Sugar"

taking care to put quotes around names with spaces. It will then return the draw for you in festive ASCII text:

Rita ------> Alan Sugar Sue ------> Bob Bob ------> Sue Alan Sugar ------> Rita

If you are publishing data to S3, then it is private by default; only you can access it. You can publish it publicly but what if you want to publish it only to people with a known link AND the link has to expire after predefined time?

Using the knox library, it's very straightforward:

var knox = require('knox'); var moment = require('moment'); var filestore = knox.createClient({ key: ##amazon key goes here##, secret: ##secret goes here##, bucket: ##bucket name goes here##, region: ##amazon region name goes here## }); // return a signed url for this file var getSignedUrl = function (path, months) { return filestore.signedUrl(path, moment().add("months", months).toDate()); } var url = getSignedUrl("mypath/myfile", 3);

Imagine you have have a large text file, with each line of data separated by a newline (“\n”) character. Our task is to read in each line, transform the data and output it to a new file. We can use the Node.js Streams api to

create an Readable Stream that points to the input file

create a Writable Stream that points to the output file

write some Transform modules that convert the data

The Readable and Writeable streams can be process.stdin and process.stdout if you wish to write Unix-like command-line scripts.

Dealing with text files, line-by-line

This blog post describes how we can create a “liner” Transform module which we can use to

receive chunks of data from the input stream

split the chunks into blocks of data that represents a line of the file

buffer any chunks that are not full lines

In a C-like programming environment, we might call fgets to get a line from a file but under-the-hood, arbitrary chunks of data are arriving which may contain multiple lines or partial lines. The liner module organises the chunks into lines and passes them on to the next process in the pipe. This makes it easy to add your own ‘transformer’ module that does your data processing and deals with whole lines only e.g.

var rs = process.stdin; var os = process.stdout; var liner = require('liner.js'); var transformer = require('transformer.js'); // pipe the input to the output, via transformation functions rs.pipe(liner) // transform the input stream into per-line events .pipe(transformer) // transform the data .pipe(os); // write the data to the output stream

Custom Transform module

Our transformer module simply needs to deal with whole lines of data:

var stream = require('stream'); var transformer = new stream.Transform( { objectMode: true } ); transformer._transform = function (line, encoding, done) { this.push( line.toUpperCase() + "\n"); done(); }; module.exports = transformer;

In this example our _transform function receives a call for every line of data, and pushes out the input data transformed into upper case, but you could put any code you like here. The done function is called when the transformation is complete.

The advantage of streams is

they can deal with huge file sizes as your script never loads all of the data into memory, it is just a production line taking data in and pushing it out

If you wish do get a list of all the "CREATE" statements required to recreate your Amazon Redshift database, and you have access to the Postgresql command-line tools, then you can do:

pg_dump -Cs --username xxxx

where xxx is a user with admin privileges.  You can pipe the result to a file:

When you are pulling data from CouchDB views with 'curl', you may wish to supply a key, startkey or endkey parameter with an array parameter. Something like this:

curl 'https://yourdomain.cloudant.com/abc/_design/extract/_list/ids/by_table?reduce=false&key=["first","second"]'

but curl will give you an error: 

curl: (3) [globbing] illegal character in range specification 

What's going on?

It turns out that curl is interpreting the square brackets as instructions to curl itself. You can switch this off by passing in the "-g" parameter:

curl -g 'https://yourdomain.cloudant.com/abc/_design/extract/_list/ids/by_table?reduce=false&key=["first","second"]'

Imagine you have a CouchDB database 'bob' and you want to do something to every document in turn. You could:

pull down the first block of 100  - /bob/_all_docs?limit=100&skip=0

apply changes and write them back in bulk

then the next block -  - /bob/_all_docs?limit=100&skip=100

Seems reasonable? Wrong. The skip parameter always starts at the first document and churns through all the documents it needs to skip, so as the value you pass to "skip" gets bigger, things get progressively more inefficient.

The solution:

ask for one more document than you need - /bob/_all_docs?limit=101

apply changes and write them back in bulk (ignoring the 101st document)

use the 101st document's _id as the startkey_docid for the next block - /bob/_all_docs?limit=101&startkey_docid=8f31b4f..

I find myself writing one-off scripts in Node.js that:

read a line from a csv file

do some work on the line

move along to the next line

Here's some example code using the csv library to parse a tab-delimited text file:

var csv = require('csv'); var csvfile = "test.txt"; csv() .from.path(csvfile, { columns: true, delimiter: "\t" } ) .on('record', function (data, index) { // do some work here console.log(data); });

But I also want to work on several lines at once, to get through the data more quickly. If the file contains many thousands of items, it is useful to control how many rows are processed in parallel with a queue. Thankfully, the async library is here to help:

var csv = require('csv'); var async = require('async'); var csvfile = "test.txt"; var q = async.queue(function (task,callback) { // do some work here // . // . console.log(task); callback(); }, 10); q.drain = function() { console.log("Done!"); } csv() .from.path(csvfile, { columns: true, delimiter: "\t" } ) .on('record', function (data, index) { q.push(data); });

In the above example, we use an async "queue" to process up to ten lines from the file in parallel. You may want to, for instance, keep the load on your database server to a reasonable level (ten simultaneous connections). The above example is trivial (it just console.logs our the queue item) but it also fails:

RangeError: Maximum call stack size exceeded

What's going on? We have exhausted the stack with queued 'callbacks'. Looking at the async issues list, other people have the same issue and commiters' response is to bark back "...STOP CALLING SYNC FUNCTIONS IN ASYNC..".

As far as I can tell, if you replace the "callback() with"

setImmediate(function() { callback() });

then the error goes away. The callbacks are scheduled to be called back asynchronously and our crash is avoided. Sometimes we don't want do any work on a certain queue item so it's tempting to just call "callback()" but we must call it asynchronously, via setImmediate

This article a tackles how to maintain uptime in your Node.js App when you are switching versions. But if you have a load balancer handling the connections, sometimes the problem is "how do I close my App without cutting shorting existing connections?".

Here is one solution:

var express = require('express'); var app = express(); // listen on the specified port var server = app.listen(8080); // serve out content app.get('/', function(req, res){ var body = 'Hello World'; res.setHeader('Content-Type', 'text/plain'); res.setHeader('Content-Length', body.length); res.end(body); }); // this function is called when you want the server to die gracefully // i.e. wait for existing connections var gracefulShutdown = function() { console.log("Received kill signal, shutting down gracefully."); server.close(function() { console.log("Closed out remaining connections."); process.exit() }); // if after setTimeout(function() { console.error("Could not close connections in time, forcefully shutting down"); process.exit() }, 10*1000); } // listen for TERM signal .e.g. kill process.on ('SIGTERM', gracefulShutdown); // listen for INT signal e.g. Ctrl-C process.on ('SIGINT', gracefulShutdown);

Our Express app listens for OS events triggered by someone pressing "Ctrl-C" at the console or using the "kill" command (as used by the forever tool). When a shutdown signal is received it instructs its server to receive no more incoming HTTP connections and it calls us back when the existing sockets have been closed, and we can safely shut down. 

The request Node.js library makes it very easy to manage your application's outbound HTTP traffic. We simply install request:

npm install request

and write a simple script:

var request = require('request'); request.get('http://www.google.co.uk/',function(err,req,body) { console.log("Got data for Google UK"); });

But what happens if we want to fetch a number of pages in parallel? Let's use the lower-level 'http' module built-in to Node.js:

var http = require('http'), start = timestamp(); var options = { hostname: 'www.google.co.uk', port: 80, path: '/', method: 'GET' }; for(var i = 0; i<25; i++) { var req = http.get(options, function(res) { res.on('data', function(chunk) { }); }).on('socket', function(e) { console.log("Socket to Google",timestamp() - start); }); }

You would expect all 25 requests to happen in parallel right?

Socket to Google 0.02200007438659668 Socket to Google 0.024000167846679688 Socket to Google 0.024000167846679688 Socket to Google 0.024000167846679688 Socket to Google 0.024000167846679688 Socket to Google 0.1360001564025879 Socket to Google 0.14300012588500977 Socket to Google 0.16000008583068848 Socket to Google 0.16000008583068848 Socket to Google 0.16100001335144043 Socket to Google 0.21700000762939453 Socket to Google 0.21800017356872559 Socket to Google 0.23200011253356934 Socket to Google 0.2330000400543213 Socket to Google 0.23600006103515625 Socket to Google 0.2740001678466797 Socket to Google 0.27500009536743164 Socket to Google 0.2850000858306885 Socket to Google 0.2910001277923584 Socket to Google 0.312000036239624 Socket to Google 0.38000011444091797 Socket to Google 0.3900001049041748 Socket to Google 0.3970000743865967 Socket to Google 0.39800000190734863 Socket to Google 0.3990001678466797

Not really. They seem to be in batches of 5. Why is that?

It's because Node.js sends all its outgoing HTTP requests through an 'agent' which, by default, is configured to pool requests to the same server and only allow 5 simultaneous requests to one domain. This can be changed:

var http = require('http'); http.globalAgent.maxSockets = 25;

Also, bear in mind that the 'https' module also has its own separate agen, so if you're making outgoing HTTPS requests then this would have to be set separately:

var https = require('https'); https.globalAgent.maxSockets = 25;

The Request module uses the 'http' module under-the-hood so it is subject to same restrictions. To get around it, simply reconfigure http/https before you require the request module:

Route 53 is Amazon's hosted DNS service. You create a zone (a domain name) and then add records to it e.g, "A" records or "CNAME" records. If you are writing your software in Node.js then Amazon's aws-sdk is here to help.

First of all, load the module:

npm install aws-sdk

and in your code:

var AWS = require('aws-sdk'); // AWS Config AWS.config.update({accessKeyId: YOURKEY, secretAccessKey: YOURSECRET}); // Set your region for future requests. AWS.config.update({region: YOURREGION}); // e.g. eu-west-1 // get the Route53 library var route53 = new AWS.Route53();

If you don't know what a accesskey, secret or region are then you need to do some readin about Amazon EC2

To work on a zone, you need to know its ZoneID. This can be discovered by fetching all your zones using the "listHostedZones" function:

route53.listHostedZones({}, function(err,data) { if(!err) { console.log(data); } });

which returns a list of your domains and their Zone IDs

{ HostedZones: [ { Id: '/hostedzone/ABC123', Name: 'mydomain.com.', CallerReference: 'SOME-UNIQUE-REFERENCE', Config: {}, ResourceRecordSetCount: 15 } ] }

We can then use the Id from the above reply to set an A record using the "changeResourceRecordSets" function:

var params = { "HostedZoneId": '/hostedzone/ABC123', // our Id from the first call "ChangeBatch": { "Changes": [ { "Action": "CREATE", "ResourceRecordSet": { "Name": "subdomain.mydomain.com", "Type": "A", "TTL": 86400, "ResourceRecords": [ { "Value": "123.123.123.123" } ] } } ] } }; route53.changeResourceRecordSets(params, function(err,data) { console.log(err,data); });

Recent reports in the Guardian pain a grim picture of the UK intelligence services tapping trans-atlantic internet cables and monitoring the traffic. Some services are plain text (POP3 email, http etc), but others are encrypted (Google, Facebook, Twitter, Gmail, etc). 

So what would a wiretapper see if they monitored the packets in my living room as I visited https://www.google.com/ (the encrypted version). This:

08:25:35.452182 IP kiki.54992 > we-in-f99.1e100.net.https: Flags [S], seq 2131462321, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 217308898 ecr 0,sackOK,eol], length 0 E..@h.@.@.!.......Bc............................... ............ 08:25:35.496038 IP we-in-f99.1e100.net.https > kiki.54992: Flags [S.], seq 4232209215, ack 2131462322, win 62392, options [mss 1430,sackOK,TS val 1385596681 ecr 217308898,nop,wscale 6], length 0 E..<....1.....Bc.........Bc?.........@......... R.. ........ 08:25:35.496104 IP kiki.54992 > we-in-f99.1e100.net.https: Flags [.], ack 1, win 8242, options [nop,nop,TS val 217308941 ecr 1385596681], length 0 E..4..@[email protected]@.. 2.I..... R.. 08:25:35.496703 IP kiki.54992 > we-in-f99.1e100.net.https: Flags [P.], seq 1:120, ack 1, win 8242, options [nop,nop,TS val 217308941 ecr 1385596681], length 119 E...u.@[email protected]@.. 2....... R.. ....r...n..Q.QoX.....&.j...lp..!...lI...5H+....9.8.5..... .3.2./............... .........................www.google.com 08:25:35.540775 IP we-in-f99.1e100.net.https > kiki.54992: Flags [.], ack 120, win 975, options [nop,nop,TS val 1385596725 ecr 217308941], length 0 E..4....1.....Bc.........Bc@...)..... ..... R..5... 08:25:35.541504 IP we-in-f99.1e100.net.https > kiki.54992: Flags [.], seq 1:1419, ack 120, win 975, options [nop,nop,TS val 1385596725 ecr 217308941], length 1418 E.......1..m..Bc.........Bc@...).....l..... ....U...Q..Q.Qj...KTRA........Z3......#.... .@[Z..8..s8V..xT&%..... .c..9Xp..... .............E...A..>...0...0......... .....0F1.0H.. ..U....US1.0...U. . 131031235959Z0h1.0...Goo..U....US1.0...U...ity0.. Mountain View1.0...U. . .........0.........E>.D.1r.,;......2..~{h/.CH.$.<+...t+v73.1/rS..^..|.2@.(x_..;....0W+..Y).4....apj..I<.$..5.Bk.(....\`...=..:..^........*..k.}.........Q0..M0...U.%..0...+.........+.......0...U....... a5...+$.{.....-..r0...U.#..0.....0..C.>g.....j..k.$0[..U...T0R0P.N.L.Jhttp://www.gstatic.com/GoogleInternetAuthority/GoogleInternetAuthority.crl0f..+........Z0X0V..+.....0..Jhttp://www.gstatic.com/GoogleInternetAuthority/GoogleInternetAuthority.crt0..............'[...G.1i#..?.e..G.70\.0N.k.a.b.v.=..{*..q................n..4.J..0...~. .#\l.>...8...2..N..K..z..1.\+..U....O{. .....0N1.0H.......U....US1.0...U..0 131231155850Z0F1.0.$Equi..U....US1.0...U.icate Authority0.. . .........0............W..>@}....qcS..yt..D.3 ..{.(...lI~._4YN z0....WE....s&..2.....;....6.oQD... Rs.&.5j....Y.+.g.9....l..%..3.G...Py..i.....Wo........0..0...U.#..0...H.h.+....G.# 08:25:35.541893 IP we-in-f99.1e100.net.https > kiki.54992: Flags [P.], seq 1419:1710, ack 120, win 975, options [nop,nop,TS val 1385596725 ecr 217308941], length 291 E..W....1.....Bc.........Bh....).....L..... .O3....0...U........0..C.>g.....j..k.$0...U.......0.......0...U...........0:..U...3010/.-.+.)http://crl.geotrust.com/crls/sec............y.F."l.1.eC`.1P..(...f.\.5.}..NY7...&..g8Z...&2.*..xP{..~..H...>.:~.`...k...*.P.w3).@.......[.4?%..LHva......P_..d_G.......~.......... 08:25:35.542015 IP kiki.54992 > we-in-f99.1e100.net.https: Flags [.], ack 1710, win 8173, options [nop,nop,TS val 217308986 ecr 1385596725], length 0 E..4..@[email protected].......).Bi............ ...:R..5 08:25:35.623283 IP kiki.54992 > we-in-f99.1e100.net.https: Flags [P.], seq 120:306, ack 1710, win 8192, options [nop,nop,TS val 217309066 ecr 1385596725], length 186 E.....@[email protected].......).Bi... ........ ....R..5.............W....M?..Y?#[email protected].&#..u...H /.&.5.O..,_..l.~i-Y.....lIgl.+..Q.,`/..b ...K.........N.*.s....^.Q......E.^..................$..+..5.1.....,0Hn.U.]., .....%`}}.rd 08:25:35.669149 IP we-in-f99.1e100.net.https > kiki.54992: Flags [P.], seq 1710:1757, ack 306, win 992, options [nop,nop,TS val 1385596854 ecr 217309066], length 47 E..c....1.....Bc.........Bi................ ...l...{M?..t..t..$.Jt.j....`...7]Q.< 08:25:35.669382 IP kiki.54992 > we-in-f99.1e100.net.https: Flags [.], ack 1757, win 8189, options [nop,nop,TS val 217309111 ecr 1385596854], length 0 E..4N.@.@.;F......Bc.........Bj............ ....R... 08:25:35.669887 IP kiki.54992 > we-in-f99.1e100.net.https: Flags [P.], seq 306:476, ack 1757, win 8192, options [nop,nop,TS val 217309111 ecr 1385596854], length 170 E.....@[email protected]... .."..... ....R........I....Uo?v.....N.N.jZE.;...2V.9...Z.=XI...H.......<yb. ...... .2n.g.$ZU...........Mg.,.).M.T1F..n...k. .F.0u..Y..t...jsv.f.~.h{=.!.-..C....gN..d..H.<...0p. .<.......9 08:25:35.718606 IP we-in-f99.1e100.net.https > kiki.54992: Flags [P.], seq 1757:2772, ack 476, win 1002, options [nop,nop,TS val 1385596903 ecr 217309111], length 1015 E..+....1.....Bc.........Bj................ R.............3.\......_#}....u.s...R.5..V..S.......Hi..=..Y*......cvx.....uM#.6c..../.....5...3m^........2....2.k.|....15.uU.o.\@.Ev..cuM.S...C..A....'.o..U...Cn.......*7I5..F5...sF).1.T...a4..k^...g..!TC...~LQ./fd...x...h........7 ..G.FR..s../H...x\y..9F..r^\2._:#9.......{..^....a......... 8..0J.....^.\b.{#.=...+V .;+..;..V...x0.+u=.r$.. .Td.....I.B}...r^i.L.. ...iYh....-:[email protected].^..q.b)d...N8....b.v{s.+..@$.;.. V.I...?.....C...zCR.. {..8d...?#4.Q.q...........M.#./.V.&D..h.......h@P^[email protected]...$U...Y.{.....b..uW,7.<..=....<.y..-.T...9R.p..=.-.....j.`.m...+mjd&J.. ..m.{.5Y...._... ..g..P..^.......T............4..V=.?....g...Z..j{"...d?F.}..idw.. .yZ$.a.].L. .....\~..P..Q...2[4v.|....w. ;..w.5.S.z.bu....V.N..0...WRUA.?8S.f.YV....b..ZRB3a....>.y$M.'..s(:[email protected]...!.........F. ..y#_c.... i......tU...D=.R.....'.(.Pv%..S/..>...ap..}..`,.......5..0#.V....W{..K.....nhb.Z..V...g...z;...Nx. 08:25:35.718839 IP kiki.54992 > we-in-f99.1e100.net.https: Flags [.], ack 2772, win 8128, options [nop,nop,TS val 217309160 ecr 1385596903], length 0 E..4..@.@..)......Bc.........Bn......T..... ....R... 08:25:35.719264 IP kiki.54992 > we-in-f99.1e100.net.https: Flags [P.], seq 476:503, ack 2772, win 8192, options [nop,nop,TS val 217309160 ecr 1385596903], length 27 E..O..@.@..*......Bc.........Bn... .6...... ....R..........[.8......3.4./h...6. 08:25:35.719347 IP kiki.54992 > we-in-f99.1e100.net.https: Flags [F.], seq 503, ack 2772, win 8192, options [nop,nop,TS val 217309160 ecr 1385596903], length 0 E..4..@.@..$......Bc.........Bn... ........ ....R... 08:25:35.761737 IP we-in-f99.1e100.net.https > kiki.54992: Flags [F.], seq 2772, ack 503, win 1002, options [nop,nop,TS val 1385596947 ecr 217309160], length 0 E..4....1.....Bc.........Bn................ R....... 08:25:35.761830 IP kiki.54992 > we-in-f99.1e100.net.https: Flags [F.], seq 503, ack 2773, win 8192, options [nop,nop,TS val 217309202 ecr 1385596947], length 0 E..4..@[email protected]... ........ ....R... 08:25:35.763465 IP we-in-f99.1e100.net.https > kiki.54992: Flags [.], ack 504, win 1002, options [nop,nop,TS val 1385596948 ecr 217309160], length 0 E..4....1.....Bc.........Bn................ R....... 08:25:35.763511 IP kiki.54992 > we-in-f99.1e100.net.https: Flags [.], ack 2773, win 8192, options [nop,nop,TS val 217309203 ecr 1385596948], length 0 E..4f.@.@.#6......Bc.........Bn... ........ ....R... ^C

Good luck with that GCHQ. There are hints of what's going on, such as the reference to 'http://www.gstatic.com/GoogleInternetAuthority/GoogleInternetAuthority.crl' which gives away the browser's negotiation with the server before it agrees to encrypt the traffic. But the body of the traffic is gibberish.

Compare this to what you see for an unencrypted request for http://www.google.com/

08:37:56.898349 IP wb-in-f104.1e100.net.http gt; kiki.55880: Flags [.], ack 146, win 992, options [nop,nop,TS val 1488458032 ecr 218040646], length 0 E..4....1.O.J}.h.....P.H....[$nc........... X..0.. F 08:37:56.901394 IP wb-in-f104.1e100.net.http gt; kiki.55880: Flags [P.], seq 1:989, ack 146, win 992, options [nop,nop,TS val 1488458035 ecr 218040646], length 988 E.......1.K.J}.h.....P.H....[$nc....D...... X..3.. FHTTP/1.1 302 Found Location: http://www.google.co.uk/ Cache-Control: private Content-Type: text/html; charset=UTF-8 Set-Cookie: PREF=ID=33d3e54116dff0d9:FF=0:TM=1371886671:LM=1371886671:S=p3NVkzfqqqf8_bzq; expires=Mon, 22-Jun-2015 07:37:51 GMT; path=/; domain=.google.com Set-Cookie: XXXXXXXXXXXXXXXXXXXXXX expires=Sun, 22-Dec-2013 07:37:51 GMT; path=/; domain=.google.com; HttpOnly P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info." Date: Sat, 22 Jun 2013 07:37:51 GMT Server: gws Content-Length: 221 X-XSS-Protection: 1; mode=block X-Frame-Options: SAMEORIGIN <HTMLgt;<HEADgt;<meta http-equiv="content-type" content="text/html;charset=utf-8"gt; <TITLEgt;302 Moved</TITLEgt;</HEADgt;<BODYgt; <H1gt;302 Moved</H1gt; The document has moved <A HREF="http://www.google.co.uk/"gt;here</Agt;. </BODYgt;</HTMLgt;

You can see that the reply from Google is easy to see (including cookies). By default, Google bounces you to the secure version, so in reality that's not much of a problem.

The question remains is "Can GCHQ decrypt https packets?".

------------------------------------------------

Update:

You can do this yourself on a Mac with:

sudo tcpdump -A -s0 -ien0 port 80

for insecure traffic and

sudo tcpdump -A -s0 -ien0 port 443