#replicaset

This post summarizes what I’ve learned about setting up a MongoDB Replica Set. For this tutorial I’m using Vagrant to create the instances but you can use the information explained here and adapt it for actual machines.

VMs Setup

Since we are going to set up a replica set with 3 members, the first thing we have to do is create the VMs, one for each member.

# Create a directory for each VM. mkdir -p mongo01 mkdir -p mongo02 mkdir -p mongo03 # Create the Vagrant files. vagrant init hashicorp/precise64 --output mongo01/Vagrantfile vagrant init hashicorp/precise64 --output mongo02/Vagrantfile vagrant init hashicorp/precise64 --output mongo03/Vagrantfile

After executing the code above we end up with 3 Vagrant environments, one in each directory. Next, edit each Vagrantfile. Set the value of config.vm.hostname and config.vm.network :private_network, ip.

# The IPs and the names are up to you. # For mongo01 config.vm.hostname = "mongo01" config.vm.network :private_network, ip: "192.168.33.91" # For mongo02 config.vm.hostname = "mongo02" config.vm.network :private_network, ip: "192.168.33.92" # For mongo03 config.vm.hostname = "mongo03" config.vm.network :private_network, ip: "192.168.33.93"

Now go ahead and execute vagrant up inside each directory to create the VM. After this, execute vagrant ssh to access the VM.

Next, we need to make sure that each VM is accessible via DNS or hostname. In this case we are going to use the /etc/hosts file to set this up. Open the file on each VM and add the following lines:

# The IPs listed here must match the ones defined in the Vagrantfiles. 192.169.33.91 mongo01 192.169.33.92 mongo02 192.169.33.93 mongo03

Now you should be able to ping each VMs from one to another by using the hostnames.

MongoDB Replication Setup

Let’s install MongoDB on each VM.

sudo apt-get update sudo apt-get install mongodb

Next, edit /etc/mongodb.conf. Remove the line bind_ip = 127.0.0.1 and add the following line:

replSet = repl01 # repl01 is the name of the replica set. You can change this name for something else.

Important: The name you choose for the replica name has to be the same in all the VMs, if not, you are going to get the following error when you try to add a new replica:

{ "assertion" : "set name does not match the set name host repl01:27017 expects", "assertionCode" : 13145, "errmsg" : "db assertion failure", "ok" : 0 }

Note: Removing the bind_ip = 127.0.0.1 line allows incoming connections from all sources. Do not use this in production. Instead of removing this line, specify the IPs that are allowed to access this server.

Next, you have to restart the mongodb service to make the changes effective. For this run the following command on each VM:

sudo service mongodb restart

After this you are almost done. Before continuing, check the connectivity between instances by making sure that each MongoDBinstance is reachable by the others. For this execute:

# In mongo01 mongo --host mongo02 mongo --host mongo03 # In mongo02 mongo --host mongo01 mongo --host mongo03 # In mongo03 mongo --host mongo01 mongo --host mongo02

You should be able to establish connection between servers.

Finally, execute the following command in the mongo shell in mongo01:

# Initiate replica set use admin rs.initiate() # Verify the configuration rs.conf() # Add mongo02 as a member or the replica set rs.add("mongo02:27017") # Add mongo03 as a member or the replica set rs.add("mongo03:27017")

You can check the status of the replica set by executing:

rs.status()

The output of this should look like:

{ "set" : "repl01", "date" : ISODate("2014-10-28T23:12:29Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "mongo01:27017", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "optime" : { "t" : 1414522392000, "i" : 1 }, "optimeDate" : ISODate("2014-10-28T18:53:12Z"), "self" : true }, { "_id" : 1, "name" : "mongo02:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 15754, "optime" : { "t" : 1414522392000, "i" : 1 }, "optimeDate" : ISODate("2014-10-28T18:53:12Z"), "lastHeartbeat" : ISODate("2014-10-28T23:12:29Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "mongo03:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 7953, "optime" : { "t" : 1414522392000, "i" : 1 }, "optimeDate" : ISODate("2014-10-28T18:53:12Z"), "lastHeartbeat" : ISODate("2014-10-28T23:12:27Z"), "pingMs" : 0 } ], "ok" : 1 }

The replica set is already setup. You should be able to create a database and a colletion in mongo01. You can verify that it gets replicated by opening the mongo shell in mongo02 or mongo03 and executing:

show dbs

You can only insert, modify or delete data in the primary member. If the primary member goes down, one of the secondary members will become primary.

Note: Mongo replica set requires an uneven number of members greater or equal than 3. If you have an even number of members then you must add an arbiter. The arbiter will be responsible for breaking the ties in the election of the primary member. You can find more information about this here.

I hope you find this useful.

On a recent project for one of my customers ( whom I cannot mention for confidentiality reasons) I was pleasantly surprised by the load a small mongodb cluster can cope with running on 4 cheap virtualized servers.

Admittedly I did have to write some custom query optimization to achieve the throughput shown below.

The business use case I was trying to solve required the database to support rather complex compound equality, range, and sort queries. The solution aimed at improving the performance of IO-bound multiple range queries that needed to scan a large number of documents.

This typical recommendation for this type of complex query is to create a compound index. However,  this strategy often leads to the anti-pattern of creating multiple compound indexes in order to cover different permutations of the query constraints. 

In brief, my strategy to solve this problem was two fold:

reduce the number of elements in the compound index by creating a custom two element only index.

change the order of parameters sent to mongodb based on the cardinality of the data in the custom index named in 1 above.

As of mongodb 2.4.9 the order in which compound query constraints appear has a large impact on the query performance.