#couchdb node.js stream

CouchDB's "_all_docs" endpoint gets you all the documents in database in this form

e.g

http://mycouchdbserver?all_docs?include_docs=true

{ "total_rows": 93985131, "offset": 0, "rows": [ { "id": "0000230a35e724e12b8c18a8f700065d", "key": "0000230a35e724e12b8c18a8f700065d", "value": { "rev": "1-adf8311047fcdd953543118e7d501fa1" }, "doc": { "_id": "0000230a35e724e12b8c18a8f700065d", "_rev": "1-adf8311047fcdd953543118e7d501fa1", "a": "1", "b": "2", "c": "3" } }, { "id": "0000230a35e724e12b8c18a8f7000ccd", "key": "0000230a35e724e12b8c18a8f7000ccd", "value": { "rev": "1-5ce610ff79bc1cfe62b4a1a68e5b09cf" }, "doc": { "_id": "0000230a35e724e12b8c18a8f7000ccd", "_rev": "1-5ce610ff79bc1cfe62b4a1a68e5b09cf", "a": "2", "b": "5", "c": "6" } } ] }

Notice the documents themselves are contained inside an object inside an array. In real life, the data comes out like this:

{"total_rows":93985131,"offset":0,"rows":[ {"id":"0000230a35e724e12b8c18a8f700065d","key":"0000230a35e724e12b8c18a8f700065d","value":{"rev":"1-adf8311047fcdd953543118e7d501fa1"},"doc":{"_id":"0000230a35e724e12b8c18a8f700065d","_rev":"1-adf8311047fcdd953543118e7d501fa1","a":"1","b":"2","c":"3"}}, {"id":"0000230a35e724e12b8c18a8f7000ccd","key":"0000230a35e724e12b8c18a8f7000ccd","value":{"rev":"1-5ce610ff79bc1cfe62b4a1a68e5b09cf"},"doc":{"_id":"0000230a35e724e12b8c18a8f7000ccd","_rev":"1-5ce610ff79bc1cfe62b4a1a68e5b09cf","a":"2","b":"5","c":"6"}} ]}

with each object on its own line.

If you are wanting to export the data and put it in Redshift, for example, the JSON needs to be in this form:

{"_id":"0000230a35e724e12b8c18a8f700065d","_rev":"1-adf8311047fcdd953543118e7d501fa1","a":"1","b":"2","c":"3"} {"_id":"0000230a35e724e12b8c18a8f7000ccd","_rev":"1-5ce610ff79bc1cfe62b4a1a68e5b09cf","a":"2","b":"5","c":"6"}

Solution 1 - jq

The jq utility allows JSON to be parsed and reformatted on the command-line. e.g.

curl 'http://mycouchdbserver?_all_docs?include_docs=true_' | jq '.rows[].doc'

(Thanks to Cloudant Support for this solution). Unfortunately it is not suitable for large data sets as jq requires all of the data to be in-memory.

Solution 2 - Use this docstream.js utility

DocStream takes _all_docs data in on stdin and outputs just the "doc" section:

curl 'http://mycouchdbserver?_all_docs?include_docs=true_' | node docstream.js'

This is should work with any size of data set, as long as each document appears per line.

e.g.