#rethinkdb

I want to create an interactive website for Hearthstone tournaments with NodeJS. user shall be able to create own tournaments, and I want to host somewhat like "Ranked” tournaments. So what do we need to make that happen only with Javascript and HTML5? Well we need the following things:

a webframework: Express 4.x.x

a Database with realtime data polling: RethinkDB

(I hate mongoDB Schematiks)

realtime communication: socket.io( known for stuff like agar.io)

and something to make everything fancy: p5.js

And of course usermanagement: passportjs

If you’re wondering where the posts about my Racket RethinkDB driver went, I’ve archived them until I can find the time to reorganize them a little. The good news is that the initial code is now available on Github. It’s missing a few things yet, but there’s enough to play around with.

One of the more interesting aspects to this project was trying to design an API that closely mirrored the nice chaining style offered by the official drivers. To see an example of what I mean, here is how one might write a filter query using the Ruby driver:

r.db("library") .table("books") .filter({:author => "Doris Lessing"}) .run(conn)

What this query serializes to (i.e. what’s transmitted to the server by run) is the following:

[39,[[15,[[14, ["library"]],"books"]],{"author":"Doris Lessing"}]]

where 39 is the code for a filter command; 15, for table; and 14, for db. (See the section on “ReQL Commands” under Serializing Queries in the driver guide for more info.)

The main point here is that the query is nested “inside-out” from the order of the methods. In the most straightforward Racket, we would write something like:

(filter (table "books" (db "library")) (hash "author" "Doris Lessing"))

where each command takes the queries it depends on as arguments. But I find this aesthetically less appealing than the ReQL chaining style, and why make the Racket driver not as great as any other? So then, if we’re going to make calls in inverse order, how do we get each command to “pass itself forward” to the command that needs it?

Inverting the nesting of expressions is exactly what happens with continuation-passing style, so it seems likely that continuations are somehow the answer. In fact, each ReQL command could produce a lambda that takes a “continuation” and applies that to whatever the command itself should produce (in this case, a data structure representation of the command). This continuation knows how the rest of the query needs to be built (hence it’s called “builder”), and how to slot the current command into it. (Where it comes from, we’ll get to in a second.) So, for example, the definition of db might look something like this:

(define (db arg) (λ (builder) (builder (reql-term 'db (list arg) '#hash()))))

db is “anchored” in that it can only occur at the start of a chain, but commands that can be chained in the middle, like table, need to first produce a continuation (for their predecessor), which will in turn produce a lambda that expects a continuation. So, first the chained command receives any query terms it depends on from its predecessor, then it “passes itself forward” via the builder lambda.

(define (table arg) (λ (pred-term) (λ (builder) (builder (reql-term 'table (list pred-term arg) '#hash())))))

Finally, this is all tied together with a function that traverses this list of lambdas and applies them in succession:

(define (build-query . commands) (cond [(empty? commands) (λ (x) x)] [(empty? (rest commands)) ((first commands) (λ (x) x))] [else (let ([chain ((first commands) (second commands))]) (apply build-query (cons chain (rest (rest commands)))))]))

Key bit:

Today I have sad news to share. After more than seven years of development, the company behind RethinkDB is shutting down. We worked very hard to make RethinkDB successful, but in spite of all our efforts we were ultimately unable to build a sustainable business. There is a lot of information to unpack – over the next few months, I’ll write about lessons learned so the startup community can benefit from our mistakes.

I just installed RethinkDB 2.3.5 on a new laptop and took a few minutes to slow down and play with the product. I’m very proud of what we built alongside our community – RethinkDB’s technology more often feels like magic, and I hope it will continue to play an important role in advancing the state of the art in database technology.