Fiesta: group email and private mailing list blog

When we announced that Fiesta was shutting down the reaction was beyond anything we could've expected. Support poured in via email, twitter, and in comments on the blog post. That support inspired us to spend the past several weeks scrambling to find a way to keep Fiesta alive, and I'm very happy to announce that it won't be shutting down after all.

A good friend of mine and one of the earliest Fiesta users, Joseph Perla, noticed the overwhelming response and reached out to us. He will be taking over the maintenance and development of Fiesta, and setting the course for its future. Joe is a great developer (and person), and I'm confident he's the right person to keep Fiesta alive and flourishing for a long time to come. Expect to hear more from Joe in the weeks and months to come.  Joe is "committed to keeping Fiesta groups alive forever as a core enhancement to email."

Sorry for the confusion/concern caused by the shutdown notice, but know that we wouldn't have been able to find a way to keep it alive if it weren't for all of your support. So, thank you, and happy Fiesta'ing!

Update (2/28/2011): We've found a way to keep Fiesta alive for good. See here for details.

We've got some sad news for you: we have decided to stop working on Fiesta and to shut the service down. If you're interested in why, there are more details at the end of this post. If not, here's the quick overview of how this affects you: we've made an export tool available on the list management page to export your lists. We've turned off the ability to create new lists, but your current lists will continue to function until March 1st. Hopefully that gives you time to find an alternate solution and coordinate with the other members of your lists. After March 1st, we'll be turning off the servers that handle incoming email (so lists will no longer function). At that point all personal data will be permanently deleted.

We owe all of you a huge debt of gratitude for using Fiesta and making it so fun for us to work on over the past 13 months. We sincerely hope that it's been a useful tool for you. If you have any questions or concerns please feel free to get in touch with me directly: [email protected]

We wish you way more than luck,

Mike and Dan

If you've read this far, it means you're curious to hear about *why* we're shutting down.

I started work on Fiesta as a reaction to a real problem - communicating with groups online is harder than it should be. I think that we've come really far with Fiesta and have solved a lot of the problems that we set out to solve. That said, it really has been a labor of love. Dan and I have worked tirelessly to get things to the point where they are now. While Fiesta has been growing rapidly, there is still a long way to go.

To be totally honest, I've lost the enthusiasm that is necessary to continue to work tirelessly on solving the problem. The reasons for that are probably too uninteresting to list here. The summary is that it seems to me that life is too short to spend time on something that I'm not 100% passionate about, so I've decided to stop spending time on Fiesta.

You may have some more questions:

Why not keep Fiesta running but spend less time on it?

Running a service like Fiesta is relatively expensive. There are financial costs (like paying for servers to host the service), but there are also time costs. Behind the scenes we do things every day like fighting spam (still partially a manual process even at really large scale services) and dealing with outages and unexpected issues. All of these things make the idea of putting Fiesta on "autopilot" impractical.

Why not sell Fiesta to someone who can keep it running?

If we found the right match, someone committed to the well-being of the users who have put their trust in us, we'd definitely consider this option. If you think you're the right person to take over (or if you know the right person) definitely get in touch.

There is a growing (and welcome) trend of web services tightly integrating email into their application workflows. One of the most useful (and frequent) types of integration is reply-by-email. Services like GitHub, Disqus and Facebook all support this.

To get reply-by-email working, services set the From header to a special address that goes back to the service in question. Here's what the From header looks like for messages from GitHub, Disqus and Facebook, respectively:

John Doe <[email protected]> Disqus <[email protected]> Reply to Comment <[email protected]>

In every case, the ... is actually a long, presumably unique, identification string.

This works well in general, but presents a couple of subtle user-experience problems. The most notable problems we've found:

Many clients (like Gmail) will automatically add addresses you contact to your address book. They'll be included in the autocomplete hints when you're composing new messages.

When searching for messages from John Doe using his email address (say [email protected]) you won't see any of the messages from him that came via GitHub.

The autocomplete problem isn't as much of a sticking point for Disqus and Facebook style headers, since they don't use the sender's name. For GitHub-style headers it's a big problem: when I start typing "John Doe" the autocomplete will include the GitHub reply address. That said, GitHub-style headers are nice as they provide more information/context about who is responsible for the message you're receiving.

To solve these problems, there should be a header that services can use to indicate that they are doing this type of From-header munging. When a client sees that header it can avoid adding the From address to the autocomplete index and, for the purposes of search, can index the message using a different address that the service optionally provides.

Let's say the header chosen is X-Indexable-From. For a service to indicate that a message's From address shouldn't be added to contacts/auto-complete, it just sets:

X-Indexable-From: false

If it's compatible with a service's privacy policy, it can also specify the address of the person who is actually responsible for the notification being received. That way, the proper name/address can be added to the receiver's contacts and search index:

X-Indexable-From: [email protected]

or

X-Indexable-From: John Doe <[email protected]>

In any of these examples, the From header stays the same as above - it's just ignored by clients for the purposes of autocomplete & search indexing.

Since we're proposing this, it may come as no surprise that we're running into similar issues at Fiesta with our implementation of Reply-Thru. We thought we'd publish this post and see if others have thoughts on this or can point out anything we're overlooking. Consider it a very informal RFC. Is anybody else interested in pushing something like this forward as an open standard? Alternatively, is there an existing solution to this problem?

One of my favorite MongoDB tricks is the ability to use an ObjectId (the default type for MongoDB's _id primary key) as a timestamp for when a document was created. Here's how it works:

>>> import pymongo >>> db = pymongo.Connection().test >>> db.test.insert({'hello': 'world'}) ObjectId('4f202e64e6fb1b56ff000000') >>> doc = db.test.find_one() >>> doc['_id'].generation_time datetime.datetime(2012, 1, 25, 16, 31, 32, tzinfo=<...>)

We're inserting a single document and then immediately querying for it. The generation_time property of the automatically generated _id gives us a datetime representing when that ObjectId was generated (precise to the second). This is great for those times when you would've otherwise added an extra "created_on" field with just a timestamp.

Going in Reverse

PyMongo's ObjectId class also has a method that let's us generate an ObjectId from a datetime, for use in querying (the other drivers have this too). Let's insert another document and give it a try:

>>> import pprint >>> import datetime >>> from bson.objectid import ObjectId >>> db.test.insert({'hello': 'a little later'}) ObjectId('4f2030d9e6fb1b56ff000001') >>> pprint.pprint(list(db.test.find())) [{u'_id': ObjectId('4f202e64e6fb1b56ff000000'), u'hello': u'world'}, {u'_id': ObjectId('4f2030d9e6fb1b56ff000001'), u'hello': u'a little later'}] >>> timestamp = datetime.datetime(2012, 1, 25, 16, 35) >>> pprint.pprint(list(db.test.find({'_id': {'$gt': ObjectId.from_datetime(timestamp)}}))) [{u'_id': ObjectId('4f2030d9e6fb1b56ff000001'), u'hello': u'a little later'}]

The call to ObjectID.from_datetime() is what let's us create a special ObjectId just for querying. If you look at the API docs you'll see a note that I wrote a long time ago about when this method is safe to use. That leads us into our next section:

Abusing ObjectIds

At Fiesta we use ObjectIds to get the timestamps to display in our new archiving UI. Recently we had to import some existing archives for a group that was migrating to Fiesta. This presents a problem: when we import the archives we are creating new documents with new ObjectIds, but we want them to have timestamps that make them look much older. There are a couple of ways we could've approached this problem. I'll start with what we did and then discuss why it's wrong and what we probably should've done instead :).

We wrote some code to generate ObjectIds with timestamps that occurred in the past, and manually generated _id values to match the messages we were importing. Here's the code:

import calendar import struct from bson.objectid import ObjectId # Current ObjectId increment INC = 0 def generate_objectid(generation_time): ''' This is unsafe. We generate fake ObjectIds. Set the five (machine id/PID) bytes to '\xFA' so we can at least recognize OIDs we generated. We don't lock around the INC, so this method isn't re-entrant. ''' global INC # Timestamp oid = struct.pack(">i", int(calendar.timegm(generation_time.timetuple()))) # Machine ID / PID oid += "\xFA" * 5 # Increment oid += struct.pack(">i", INC)[1:4] INC = (INC + 1) % 0xFFFFFF return ObjectId(oid)

We couldn't use the above ObjectId.from_datetime() because, as noted in the docs, it's unsafe for use in anything but queries. The method above is marginally more safe by virtue of using an actual increment and a canary for the Machine ID & PID (from_datetime() uses all \x00s). But it's still unsafe - if we need to do another import we need to be sure not to use the same canary. We also need to be sure that the canary never matches any of our actual Machine ID / PID bytes.

What we should've done

The great thing about mailing lists is that they make it simple to discuss things with an entire group. A problem with mailing lists is that they can make it too simple to discuss things with an entire group. Everyone has seen it or had it happen to them: you receive a message through a list and reply with a personal note to the message's sender. Your reply, however, actually goes to the entire list. Embarrassing.

The Problem

The reason this happens is because most mailing list software (including Fiesta by default) sets a "Reply-To" header pointing back to the list when a message is distributed. When you press Reply, your email client sees that header and composes a new message to the list rather than the sender. Mailing lists do this to foster group communication; if replies go to the whole group then everyone stays in the loop. Otherwise, it's very easy for discussions to become fragmented.

The problem seems intractable: we want people to be able to send personal replies, but we also want to keep group discussion on the list as much as possible (even if somebody hits Reply instead of Reply-All). To date, all mailing list software has sort of punted on the problem. The list software picks a default behavior and, at best, gives users the ability to customize the behavior themselves.

Customization might seem like a great solution, but it's bad for a couple of reasons. First, customization requires work and management. Who gets to decide how a list behaves? When? Is the setting mandatory or is it hidden away where only advanced users will find it? Second, customization means that different lists behave different ways, even on the same service. This is the same problem seen with all modal UIs; users can never be certain exactly how a list will behave without knowing how it's configured.

A New Solution: Reply-Thru

We've been working on what we think is a better solution to this problem. We're calling it Reply-Thru. Reply-Thru is enabled on all lists that are using NewFiesta.

When you get a message from a Fiesta list you can Reply or Reply-All, just like a normal email message. If you Reply-All, your message will go to the entire list. If you Reply, it will go only to the sender. What makes Reply-Thru special, however, is how that direct message is sent. It gets sent through Fiesta, and a note is added telling the recipient that it was sent directly to them. The recipient has the option of sending a Reply directly back to you, but they also have the option of hitting Reply-All and taking the conversation back to the list. They can also distribute the message to rest of the list with a single click.

Reply-Thru makes direct responses the default, but makes it dead simple to bring the discussion back to the list when that's where it belongs. Best of all, there's no configuration: every list can behave the same way. There are some issues with this approach as well, most notably is that we have to manipulate the From address of emails being sent to lists. Instead of seeing an email from [email protected], users will see [email protected]. This might present some confusion, especially to those who are used to the old behavior.

Since we launched Fiesta, one of the most asked for features has been the ability to browse list messages from the web. We've been hard at work on getting this right, and we're psyched to announce that it's available now. Here's an example thread that explains a bit more about it. For new groups it's enabled by default, but for existing groups you'll need to go to your settings page and manually enable (we explain why in that thread). There was some good coverage of the launch in Fast Company, as well.

Parts of the Fiesta API (as well as some new features we're rolling out for Fiesta itself) rely on the ability to automatically generate clean text (markdown) versions of incoming messages. Our parser tends to prefer using the text version of a message if possible, as it's generally easier to parse than the HTML version. That said, sometimes a message only contains an HTML version - we need a way to generate our canonical markdown representation from that alone. Enter html2text.

html2text is a great little Python module by Aaron Swartz that takes HTML as input and generates a markdown version as output. Here's an example:

>>> import html2text >>> print html2text.html2text('''<html><body><p>Hello World</p> <ul><li>Here's one thing</li> <li>And here's another!</li></ul></body></html>''') Hello World * Here's one thing * And here's another!

Last week we did a post with some tips for improving web app security, focused primarily on the Strict-Transport-Security, X-Frame-Options and X-Content-Security-Policy headers. Even though it was a pretty quick/simple post, the overall reaction was positive. There were some requests for more similar posts, so consider this the next in the series. This post is about a couple of options you can set on cookies to improve security.

HttpOnly

When your server sets a cookie the client's browser will include it in future requests. Cookies are probably the easiest way to maintain state across requests, so most web applications use them to store users' logged-in state, etc. That makes them a good target for attacks: if an attacker can get ahold of a user's cookie they'll be able to take actions in the app as if they were that user.

In addition to including them in future requests, the browser also exposes cookies to Javascript running on the client side (as document.cookie). This makes document.cookie a great target for exploitation if the attacker is able to find an XSS vulnerability: they can embed a script that does something evil with the values of the user's cookies.

The HttpOnly option tells the browsers that support it not to allow client-side access to the cookie. The cookie will still get sent along with future requests (so it can be used to maintain state on the server side), but won't be visible to an attacker who manages to run a script on the page. Here's what a Set-Cookie header might look like that includes HttpOnly:

Set-Cookie: x=5; path=/; HttpOnly

This option limits the surface area of XSS attacks, but (just like our discussion of X-Content-Security-Policy) is really just treating the symptoms and not the cause. The important thing is to prevent code injection altogether.

Secure

Cookies are also vulnerable when they are sent in the clear. Just like in the last post, the first step is to be sure you're using SSL for all requests. The Strict-Transport-Security header comes in handy there (n.b. we recognize that this blog doesn't use SSL - we're hosting it on tumblr. We do use both of these options on Fiesta). Cookies also have a Secure option, which tells browsers to only transmit them over HTTPS. If you're already using SSL for all requests, set the Secure bit on your cookies, too. Here's an example header:

Set-Cookie: x=5; path=/; HttpOnly; Secure

We're live-blogging MongoSV today. This is the last post, but here's a link to all of the posts from the event.

This talk is being given by Eliot (the first of his that I've seen all day!)

Going to go over HA best practices: keeping data online and safe.

What about a single node? This will have downtime. If the node crashes, intervention might be necessary. If it disappears will need a backup.

Replica set v1: Single datacenter, single switch, single power source. But automatically recovers from a single node crash. A good start but not great.

The next step up is still single datacenter, but w/ multiple power/network zones. Like EC2 in a single region but multiple AZs. Still some points of failure (datacenter / two node failure). With an arbiter, we can't do w=2 writes and remain up. With 3 non-arbiters we can use w=2 but are still vulnerable to datacenter failure.

The next step up is multi datacenter w/ a single DR (disaster recovery) node in a different DC. We can't always stay up, but we at least have a DR option now.

Now let's look at the ideal: three datacenters, five nodes. One has a single delayed slave (that helps recover from fat-finger incidents like accidental db.drop). The other two DCs each have 2 active nodes. We can lose an entire DC and still have a majority w/ the other two DCs. Can do `w={dc: 2}` to guarantee write in 2 DCs.

Moving on to HA sharding

Each shard needs to be a replica set - same rules apply as above. Balancing can be run in a window (this is cool!) can set an activeWindow to only run the balancer at night. Sweet!

Config servers need to be on at least 2 different power/network zones. Ideally just put in three separate DCs. Use host names rather than IP addresses: much easier to move a config server. Take backups of config servers. Important note (saw this earlier too in Richard's talk): not a replica set. To bring new nodes online you need to manually move the data. Not a problem if a config server is down for a day or something: just won't do splits/migrates.

Run one mongos per app-server. Don't need to worry about scaling mongos. Saves a network hop for many ops. If you really don't like this, run a pool per power region with a load-balancer in front.

Application-level tips:

Handle spikes: queue non-synchronous writes, isolate components and features. Can your site handle going into a read-only mode? That helps a lot when dealing w/ issues.

Monitor! Load, disk, CPU, but most importantly I/O (iostat). Alerts go hand in hand w/ monitoring.

Have good procedures for backups, adding replica set members, adding shards, etc. Practice (in staging) (which ought to be the same as prod). Randomly shut down boxes & load test as much as possible.

Recap

We're live-blogging from MongoSV today. Here's a link to the entire series of posts.

Presented by Greg Brockman from Stripe

The actual title of this talk is "There's a Moster in My Closest", but I thought the subtitle would be more elucidating. This talk is packed! Actually, all of the talks so far today have been pretty packed - great crowd here.

Monster is the name of the event processing system Greg built for Stripe. Been using it in production for a few months now, and it's built on top of MongoDB. The concept of event processing is that you want to glean some information from lots of real-time events that are happening (incremental stats, real time analytics, trending topics, etc.). Stripe uses it for fraud detection, dashboards, and more. Now we're going to get a live demo!

He's showing a blog-post generator that he's written, going to use Monster to monitor the content of the posts that it's spitting out. Live coding a "model", which looks like sort of a quanta of reporting. Logging a new event per-sentence that gets generated. Now we need a consumer to actually do something with the events. The consumer gets streamed events and just needs to "do something". Doesn't worry about storage, generation, etc. Registers for classes of events and has a `consume()` method. Pretty simple, but flexible. Consumer is logging when generated sentences are "too long".

Question: Monster vs celery/beanstalkd/resque? Answer: when using a job queue the act of logging implies an "action"/job. With Monster/event queuing the goal is to totally decouple logging from performing actions on logs. Can add new consumers later, etc. Events persist, not ephemeral.

Consumer uses polling to get new events.

Now we're hearing why they chose MongoDB. Replica sets are a major reason, for HA. They also wanted a document store: easy to use, so developers will all use it. They need atomic operations (talking about things like findAndModify). Seems like a lot of the talks today have been mentioning findAndModify. They like automatic collection creation, from a deployment perspective. No migrations, etc. Finally, background index building is really important for Stripe. Can create new indexes w/o compromising availability.

Tradeoff: no transactions. This is the one thing they'd really like for Monster (mainly for DR). The particular case that they need it for is what they call a Stateful Consumer - can modify the state of an event while consuming it. They basically build transactions at the application layer here.

We're live-blogging from MongoSV today. Here's a link to the entire series of posts.

Presented by Luke Gotszling

About.me uses MongoDB for different pieces of infrastructure, but this talk is just about queuing.

Originally ran a 3-node RabbitMQ cluster, without disk persistence. Were having trouble diagnosing issues at scale. Looked at some other AMQP options, but decided on MongoDB.

Benefits: async ops, per-message (document) atomicity, batch processing, periodic processing, durability, sharding, operational familiarity (n.b. that would be the big one for me!). One drawback: AMQP push model needs to be emulated with MongoDB polling. To model topic matching, they're using a regex. One thing they don't (can't) do with Mongo: fanout.

Use a capped collection? It has better performance but is limited to a single node and FIFO. They use an uncapped collection: can shard. Can get semi-FIFO but not strict.

Implementation:

Each message is a document. To create a message, just insert. The document has a queue field (string id) and a payload (serialized data).

To consume a message they use a findAndModify to grab and remove a document atomically. They index on (queue, _id).

That's pretty much it! This would be pretty simple to implement in any language (he's showing an example in the shell + in Python).

Benchmarks they ran showed MongoDB outperforming RabbitMQ for message creation by 19% (this is a single-node benchmark on a laptop, FYI). For consumption MongoDB again does very well (outperforming RabbitMQ for different levels of concurrency).

FindAndModify is blocking, so you will see high lock % w/ lots of concurrent consumers.

Pros and Cons

Pro: familiar, sharding, durability/persistence, low operational overhead, optional use of advanced queries.

Presented by Richard Kreuter

We're live-blogging from MongoSV today. Here's a link to the entire series of posts.

These are cautionary tales, but also an opportunity for schadenfreude. All true stories.

1. The first example is of a user who ran w/o journaling. Important to note that journaling isn't a panacea: OSes can have bugs, hardware can fail, etc. Journaling is good, but replication is even more good.

So. No journaling. All of the mongod processes were in the same data center. Lost power and everything shut down. When things came up they needed to run `repair`. (This is like fsck for MongoDB). They were warned that it would take a long time (about 12 hrs in their case). They decided to go back into production w/o repairing to avoid the 12 hr wait.

After a few days, things weren't happy. Mongod was stable but some cursors were aborting when they saw corruption (they were seeing "Invalid BSON" in the log files). They didn't notice the log file problem but just saw queries aborting. One of the cursors that was aborting was the replication cursor. The corruption was such that mongod could run, but queries were failing after a certain corruption point.

Note: this isn't guaranteed operational behavior, just the way this particular corruption played out. Ultimately, the fix was to take the system offline and repair. It fixed the problem but took 12 hrs.

tl;dr Use journaling unless you're absolutely sure you don't need it.

2. A user decided to shut down a config server (in a sharded deployment) and delete all of it's data files. The config server is one of three independent servers that use 2PC to maintain identical copies of their data.

Once the config server was down (because 2 were still up) reads and writes still worked but data rebalancing stopped. At some point they brought down the second and third config servers as well. At the time they had some mongos servers running that probably had config server data cached, but that data isn't persisted. They had two options:

  1. Dump each shard's data. Set up a new cluster and re-load all of the data.

  2. Analyze data on each shard to figure out what the actual ranges of data were on each shard. That could reconstruct the data that was lost. This is a very tricky/hard program to write.

Either of these approaches could lead to a problem because sometimes there are multiple copies of data across shards (during migrations from one shard to another, after an aborted migration, etc.). This is normally the job of mongos/config servers. They ended up having to resolve those conflicts manually.

tl;dr Don't delete your config servers!

3. User was seeing replication lagging and write ops taking forever on the primary. Application was basically unresponsive. Primary was showing large #s of page faults and a high lock % (time spent holding the write lock).

As it turned out, 100% of their write load consisted of pushing elements onto lists. Number of pushes grew with the square of the number of users. They also didn't have indexes on userId. So 100% of writes and reads were table scans, and each write action was N^2. This load was saturating the primary, so secondaries couldn't keep up.

Adding an index fixed immediate issues, but they needed schema/application logic changes for long-term fix.

tl;dr Pay attention to your indexes!

4. A user deployed but vastly underestimated their uptake. Needed to shard within 3 hours of launch.

Going from N shards to N+1 shards is easier for large N. In any case, it's hard if you're already overloaded - can't migrate data. Luckily, they hada few collections, each equally written to.

All they needed to do was set up some replica sets and dump out specific collections to move them off. Then they set up config servers manually. Once they had headroom, they added shards and used a more conventional sharding setup.

Presented by Kyle Banker

We're live-blogging from MongoSV today. Here's a link to the entire series of posts.

Kyle's strategy is to start with a normalized representation and then embed for simplicity and optimization. This reminds me of our data-modeling post.

1. Hierarchies

Two strategies; first is to store the canonical hierarchy in a single document. That allows atomic updates of the hierarchy, but can make other ops difficult. The second is to store a list of ancestors in each document; that's the one we'll talk about. Each node has a parent_id with the immediate parent. Each also has an array of ancestors. It's easy to display a single node, but it's also easy to display all descendants of a node (by looking inside the ancestors array). What about updates?

There's a new helper to compute ancestors for a node and overwrite the ancestors array (in case we insert in the middle of the hierarchy). When we do an insert in the middle we'll need to run the helper on each descendant, since they all have a new ancestor.

tl;dr use arrays and multi-key indexes (and the positional operator) to deal with hierarchies.

2. Ticketing

What about ticketing, don't you need transactions? How can you do transactions w/o RDBMS style transactions? Distributed systems, long-running transactions, and contentious environments might require different strategies (see paper: your coffee shop doesn't use two-phase commit). Let's see an example:

Two collections. One is a map of available seats for an event. The second has one document for each seat, with a state (available or not) and price.

Use find-and-modify as a TAS operation to update the state of seats to "Cart". If it succeeds all of the seats then we're good to go. If not, we can manually roll-back state. Basically do each state-change manually and be prepared to roll-back if needed.

This isn't right for everything but it works for some.

3. Feed reader example

Four collections: users, feeds, entries and buckets.

Users have username and an array of feeds. Each feed in that array is a document with an ID and denormalized name. We'll index on username.

Feeds collection each has a URL, name, subscriber count (denormalizing here again) and date of last entry.

To add a feed, do an upsert to insert if missing or increment subscriber count otherwise (good use of upsert). Also need to do an $addToSet on the user to add the feed to the feeds array.

Removing means a $pull from the user's feed array and a $inc of -1 on the feed's subscriber count.

Entries documents each have content for a single entry, with ID of feed it belongs to.

We need a way to show an individual user's feed: that query can be expensive if done naively. That's what bucketing is for: only query for latest entries and then store them in a bucket.

Presented by Kenny Gorman

We're live-blogging from MongoSV today. Here's a link to the entire series of posts.

Kenny is getting started, talking about performance tuning based on experience at Shutterfly. They have 8 MongoDB clusters in production with ~30 servers. Not cloud based: all own hardware and datacenters.

MongoDB performance tuning is similar to traditional RDBMS tuning. Looking at queries, indexes, etc. If performance isn't good on a single server than don't look to sharding, reading from replicas, etc. Single server performance is critical.

Modeling is key. Schema design can be really important for performance (recommends talks later on by Eliot & Kyle).

Know when to stop tuning: prioritize what is important/adequate for the business/application. What needs to be fast? Build tuning into dev. lifecycle, don't wait until there's an issue. Tuning is "personal": need to know your problem/domain.

MongoDB is really fast when read only, writes start to impact performance. Important consideration during design phase.

The profiler. Writes to db.system.profile collection. Recommendation is to turn it on and leave it on: low overhead. Look for full scans (nreturned vs nscanned) and updates (ideally you want fastmod - in place updates. Look for moved & key updates).

Should graph response times over time (from the system.profile collection). Shows performance over time of db. To look at the profiling data just do `show profile` from the shell.

Showing examples of data from the profiler: here's an example where nscanned is 10000 and nreturned is 1: we need an index! Another example where need to move the document due to an update (keyword "moved" in the profile doc.). Now showing an example using $inc - you'll see "fastmod" in the profile document - that's good!

Now talking about explain(). Use during development, don't wait. This actually runs the query when you call it. When you find a bad op using the profiler, run explain on it to get more info: shows index usage, yields, covered indexes, nscanned vs nreturned. Another recommendation: run explain() twice to see difference when data is in memory. Showing the difference between a query w/ and w/o an index in terms of explain.

Now talking about covered indexes: need to do a projection that says we don't need _id: `db.test.find({userid: 10}, {_id: 0, userid: 1})`. When you don't need _id it's possible to respond to the query using the index only.

Architecture tips: split on functional areas first to different replica set clusters, then worry about sharding those (possibly). Do reads off of slaves when you can, but be sure your app can handle inconsistent reads first. Also, use slaves for maintenance (index compaction, etc.). Move reports & backups to slaves, too. One mongod instance per machine: keeps things simple for introspection.

Emphasizing the importance of minimizing writes.

Now we're talking about data locality. When you're doing a query it's best if the results are as dense as possible (as few blocks on disk). How do you maintain this? Here's an example of how to see this: need to include `$diskLoc` in your query document, and finish with a `.showDiskLoc()` (analogous to `.explain()`).

Total performance is a function of write performance. Keep an eye on lock % and queue size: how much is the DB waiting for writes. A trick (for pre 2.0 when data > RAM) is to do read before write: spend more time in read lock rather than write lock. Tune for fastmod's: reduce moves (maybe by pre-padding documents). Evaluate indexes for key changes, minimize # of indexes if unused. Look for places to do inserts instead of updates.

What about scaling reads? They scale easily if writes are tuned. Identify reads that can be performed on slaves. Make sure you have enough RAM for indexes - can check the mongostat "faults" column for cache misses. Minimize I/O per query (back to data locality).

Tools: mongostat (look for faults & lock % / queue lengeth). currentOp() to see what's waiting. mtop to get a picture of current session level information. iostat to see how much physical I/O is going on. Do load testing before going live. Use MMS (or some other monitoring system).

I made the trip to Santa Clara for 10gen's big MongoDB conference, MongoSV, which starts in just a few minutes. It should be a great event, and I thought it'd be fun to report back on the event as it happens for those of you who couldn't make the trip.

I'll be trying to update this blog with posts from each session I attend, so subscribe or refresh to get the latest. If there's a specific session you'd like to hear about, let me know in the comments and I'll make an effort to get to that one!

Here's a link to the entire series of posts.

MongoDB's Storage Engine Bit by Bit

Presented by Mathias Stearn

Mathias is set to present about MongoDB storage internals and the room is filling up. Somebody asks a question: what is his favorite Mongo feature?. He's partial to find-and-modify but his favorite is just general ease-of-use. He's also talking a bit about the new aggregation framework which should be coming out in 2.2. Declarative aggregations: much easier than map reduce and better performance. Another question: what's the most common mistake? One: thinking MongoDB is the same as a relational DB. Two: thinking MongoDB is totally different from a relational DB. Basic concepts still matter but also can't think "too relational". The room is now SRO and the talk is starting.

Mathias is talking about the structure of the files in /data/db. You'll see mongod.lock, which is a lock file that just contains the PID (a few bytes). There's also a .ns file per DB, which is 16MB and is basically just a big hash table (more details later). There are also data files: dbname.0, dbname.1, etc. Those files grow up to 2GB, and there's always a pre-allocated empty file: this is why an almost empty DB will still use >200MB of space. Trading off space for speed.

Now we're hearing about mmap: an OS feature that maps a file on disk into virtual memory. A common confusion point is that "MongoDB is an in-memory DB". Mongo maps to virtual memory: doesn't need to fit in RAM. Now he's showing a diagram of virtual address space: kernel space, stack, heap, program text, etc. Program text is mmap'ed just like Mongo data files. Showing where the mapped data files live in the virtual address space.

Now Mathias is talking about the 32-bit limitation caused by relying on mmap: roughly 2.5GB of space left over in the address space after kernel, heap, stack, program text, etc. (start w/ 4GB: 2^32). With journaling this is effectively even smaller: don't use 32-bit. With journaling you still effectively have 64TB of capacity w/ 64-bit.

Describing a doubly-linked list: data structure used a lot in internals. Each element has both a next & prev pointer to allow traversal in either direction.

Now we're talking about the .ns file: Namespace Details. 16MB divided up into ~1KB buckets. Inside each bucket we have the name of a collection, some stats (size, count, etc.), pointers to first extent & last extent (doubly-linked list), free list (actually an array of bucketed free-lists), and index data. There are namespaces for collections and also one for each index. There's one special namespace: $freelist for free extents. ~16000 namespaces per database, which is usually enough but can be configured if more is needed.

The pointers are stored as DiskLocs, which is a data structure with a fileNum and an offset: two ints (a 64-bit struct). Note that max offset is 2^32 - 1: same as max file size.

Now we're talking about data files (.0, .1, etc.). Each data file is broken up into extents: contiguous blocks w/in a file that is owned by a single namespace. Each extent has a location, pointers to the next and previous extents, pointers to the first and last records in the extent, and the length of the extent. Then, obviously, there's a data section. The first and last record pointers are just 4-byte offsets into the data section. Extents are nice because they keep data local per namespace.

Now on to records. Each record has a length, offset w/in the extent, and then offsets of the next and previous record. Lastly there's data (BSON objects, b-tree buckets, etc.). How big is the data section? In the BSON case it's bigger than the object it contains: there's some padding that gets pre-allocated to try to avoid moves when documents grow. That padding factor is dynamic and calculated by how often moves end up occurring. The more documents move, the larger the padding factor will be. It's local to the namespace (i.e. index collections never grow). Padding factor can vary between 1 (no padding) and 2 (double data size), starts at 1. The extent offset is necessary to go from a record back to the extent header data.

Raw queries ($natural order) just walk over the next/prev links directly.

Talking about the "swiss-cheese problem". If documents change sizes a lot you can end up with gaps that are too small to be used by any new documents: hence the new "compact" option to remove those gaps. This is a blocking operation: best to run on a secondary node and then step-down the master. Right now there's not great metrics about fragmentation: can get some clue by looking at dataSize vs storageSize. If storageSize - lastExtentSize >> dataSize then there is a lot of empty space that compact would clean up.

On initial RS sync or re-sync you'll basically get compaction too.

Now we're talking about the B-tree implementation. Regularly B-tree w/ a few modifications. Each bucket is 8KB, w/ keys in sorted order. One modification is the way splits happen: rather than always splitting in the middle of the node Mongo will do a 90/10 split if you're doing incremental / ordered insertions. That can save a lot of space.

Update: here's the next post in this series.

It seems like web app security has entered the public conscious recently, probably as a result of the press covering the activities of groups like Anonymous and incidents like security breaches at several CAs. Here are a couple of quick security tips to improve the security of your web apps. Think of these as low-hanging fruit, not as a substitute for thorough analysis of your app's security. If there's interest in this topic we can do more posts, too - let us know in the comments!

Prologue: SSL

Your app already forces all traffic over SSL, right? If it doesn't, it should. There are CAs that are accepted by all major browsers and will give you a cert for free. It's almost a sure bet that any overhead of SSL is drowned out by time spent in your app code. Forcing traffic to use SSL isn't a panacea (more on that later), but it's a great place to start.

Strict-Transport-Security

Assuming your app forces traffic to use SSL, there's still a chance that a plain-text connection happens: the user might forget to type *https* instead of *http* in their location-bar. For example, let's say Fiesta forces all traffic to use SSL (it does). When a user types "http://fiesta.cc" (or just "fiesta.cc") into their browser bar the Fiesta server responds with a redirect to "https://fiesta.cc". When that initial plain-text request is made, however, the browser includes any existing cookies. That enables attacks like the one that Firesheep made so easy earlier this year. It also opens up the possibility of a man-in-the-middle attack.

Strict-Transport-Security is a header that a site can set to tell browsers that it always wants connections to use SSL. If a server includes the Strict-Transport-Security header, browsers that support it will remember that information. The next time the user visits the site the browser will automatically switch to the SSL version of the site, without making any requests. That means there is no opportunity for cookies and other information to be sent plain-text, or for MITM attacks. Here's an example of how to set this header using Nginx:

add_header Strict-Transport-Security max-age=15768000;

The `max-age` parameter tells the browser for how long to cache the knowledge that this app wants SSL connections only. In this case, we're telling the browser to remember for half a year.

X-Frame-Options

Another type of attack that has gained in prominence in recent years is called Clickjacking. Clickjacking is a technique where an attacker hosts their own site, but includes a frame with content from your app. On top of the frame, they overlay carefully constructed content that tricks the user to click in certain areas (maybe it's a whack-a-mole game (click on the moles as they pop up) or a link "click here to win a VCR"). When the user clicks, the click is actually passed through to the underlying frame where it takes some undesirable action in your application.

The standard approach to fight clickjacking is to include Javascript in your app that causes it not to load when it is embedded in an frame. This is called frame-busting. The problem with frame-busting is that attackers have devised ways to counteract many different types of frame-busting logic. Knowing to include the code and keeping up to date with the state of the art is more than can be expected of most developers. That said, browsers can't just disable framing altogether - much of the web depends on it.

The X-Frame-Options header is a mechanism for app owners to tell modern browsers whether or not their content should be allowed to get framed. Our approach is to never allow framing of Fiesta content. Here's how we set X-Frame-Options with Nginx:

add_header X-Frame-Options DENY;

If there are specific pages that we want to be frameable, we don't set the header for those pages.

X-Content-Security-Policy

This last tip is a bit less trivial than the previous two. The basic idea is that we can limit the damage from XSS attacks by allowing app owners to specify trusted sources of content for a given page. If a site happens to have a vulnerability that allows a XSS attack the attacker won't be able to load arbitrary resources. This is obviously "treating the symptoms and not the cause": the best approach is just to be sure that your site never allows arbitrary content injection. That said, an extra layer of protection is always a good thing.

An X-Content-Security-Policy header tells the browser (as of now I think only Firefox supports this) what sources are allowed for different types of content. The value of the header is a complicated declaration of those sources. Here's an example that allows all content from the host domain, plus scripts and images from Google Analytics:

add_header X-Content-Security-Policy "default-src 'self'; \ script-src 'self' https://ssl.google-analytics.com; \ img-src 'self' https://ssl.google-analytics.com";

CSP also has a reporting mechanism to alert you when any violation occurs. This makes it useful as a warning signal of XSS vulnerabilities, even if only a fraction of your users are on a browser that supports the header. There's also a "Report-Only" mode that will send you reports but not block any content: perfect for testing and getting started with CSP. As I said, this one is a bit more complicated, but also quite powerful. Definitely worth a look.

Going Forward

As I mentioned in the Prologue, the recent attacks on CAs have proven how vulnerable SSL really is: it's really only as secure as the least-secure of all the CAs. Defaulting to SSL isn't enough, at some point we'll need to move away from the current CA system. Public-key pinning is a good first step, and something like network perspectives / Convergence is an interesting path for the future. If you're interested in this stuff I'd highly recommend checking out ImperialViolet and Moxie's Blog for more. They're both interesting and easy reading.

Does this happen to anybody else besides me?

$ import datetime -bash: import: command not found

I'd say at least once a day I fire up a new shell and start typing Python code without remembering to launch the interpreter first. Luckily, that Python code usually starts with an import statement. Here's a quick hack I just put in my .bashrc to fix the problem:

fake_import() { python -ic "import $1" } alias import=fake_import

Now I can just type import datetime and a new Python interpreter will launch with the datetime module already imported:

$ import datetime >>> datetime.datetime.utcnow() datetime.datetime(2011, 12, 2, 16, 15, 29, 614887)