Making Change.org

At some companies, user research is a lot like flossing: we know it’s good for us but we aren’t doing it enough. Here at Change, we floss all the time - and we make sure to floss together.  This means that when we talk to users, we make sure that all members of the product team hear what they have to say. This includes the engineers.

There are many benefits to having engineers participate in user interviews, but they mainly fall into three buckets:

It creates empathy toward your users

Every developer has looked at error reports or logs and wondered aloud, “what the hell is this idiot user doing? They must be daft.”  This is wrong.  The best antidote to this toxic thinking is actually meeting your users and learning about how they use your product.

You start to have thoughts like, “oh, that link is kind of hard to find, isn’t it?” and “ouch, I never noticed how confusing that navigation is - I always shortcut it and manually enter the url.”

Your goal should be to have a mindset where you look at the error reports and think, “well, given that this user is not an idiot, what did we do to mess up their experience?”

It creates a common language among your team

Hearing feedback directly from a user’s mouth cuts through any mental blinders we may have when we only read research reports. No more can we say, “oh, I bet they didn’t mean it that way.”

In our case, both the lead engineer and product manager attend user interviews. This means that as we plan features, we can easily talk about “users like Ray & Ruth” or situations like “the Schlossberg dilemma.”

It energizes you!!

One good user interview can leave you with a “I want to build stuff for them!” attitude.

Having looked a user in the eyes and chatted with them can help you keep a high level of quality when you are working through particularly hard problems.  I like to take it one step further and end the meeting by giving the interviewees my business card - then I’m definitely on the hook to build a good product.

Operational tips

I’m a fan of hacking my own habits to make myself more productive, so I wanted to include some guidance for any engineer attending an interview.

Be the note-taker

The person running the interview is focusing on the interviewee and digging out their wants and needs - they shouldn’t be taking notes.  Typing up notes during the interview keeps you in the moment but too busy to speak much (see the next tip).  Post interview, cleaning up & annotating the notes is a good way to review what was said and will help you retain the findings.

STFU - seriously.

User interviews are all about listening. Resist the urge to offer solutions, especially if the user isn’t using your product “correctly.” This is usually the hardest part for everyone (especially engineers) to do in practice.  

If you have to speak, don’t stray far from “that’s interesting, tell me more about that.”  If you see that the user is missing some important piece of functionality and you feel compelled to offer a solution, at least wait until the very end of the interview.

After a long hiatus, we finally put up the video we recorded at the SF JavaScript meetup on RendrJS. Watch Josh Callender and Mark Dimas from our Platform team talk about why we use Rendr, how we architected it, and any gotchas we ran into.

We use online surveys on our website to learn our users and their goals, identify areas for improvement, and measure user satisfaction. Also, by surveying users, we can test content, information, architecture, visual and interactive design, etc. However, surveys are not always the best tool. Sometimes the data (meaning feedback) may be obtained through other resources and approaches, such as interviews (formal/informal), census figures, observation, and etc. We prefer to use a survey when we need a quick and efficient way to get information, reach out to a large number of people, get statistically valid data about a large number of people, and this information isn't available through other sources.

There are 3 types of surveys:

1. Case study surveys

provide only specific information about the group

collect information only from a part of a group

2. Sampled surveys

ask a sample portion of a group to answer the questions

the results for the sample reflect the results of the entire group

3. Census surveys

go to every member of the group

give the most accurate information about the group

are not practical for large groups

We don’t conduct census surveys at Change.org. The most common and practical approach for us is running a sample survey. Sometimes we also conduct a case study survey, but it depends on the research goal and audience.  

The goal and audience

Defining the audience and the goal is one of the most important stages. It defines the survey structure and its form. First, we consider the purpose of research and then the target.

Goal

Focus on survey goal first: what are the major decision points or areas of uncertainty in the usage of the product? Focus on those areas and clearly define what you hope to discover. Inform or assign the stakeholders in the development of the survey and set up a deadline.

Audience

The next step is clarifying who has the answers to your questions. Which specific group? The audience defines your survey type. The most complicated stage in conducting the survey is to compile your sampling frame.

The Sample (or sampling frame) is the group of people who is selected to be in our research. It’s hard to set up its size. A sample size should be large enough to represent the overall group. It should be proportional to the group. (Sometimes statistics agencies use a sample size calculators to figure out how big the sample should be). A general rule is that the larger the sample size, the more accurate a reflection of the whole it will be.

Caveat: bias and self-selected responders should be consider as a separate group. Otherwise, the surveys can cause more harm than good.

And, don’t forget about margin of error and standard deviation! In analyzing the results, we should consider the margin of error in a sample. When in doubt, go find yourself a statistics nerd to help talk you through these details.

Margin of error is a confidence interval. When we conduct a survey, we're making an assumption about what the larger population thinks. If a survey has a margin of error saying 5%, it means that if you ran that questionnaire 100 times — asking a different sample of people each time — the overall percentage of people who responded the same way would remain within 5% of the original result.

Standard deviation - the amount of variation or dispersion from the average (or how far from the normal). A low SD indicates that the data points tend to be very close to the expected value. A high SD indicates that the data points are spread out over a large range of values.

To summarize: defining the audience and the right user segmentation for the survey may be complicated, because you have to keep in mind a lot of different terms to make your sample correct and reliable. The main rule is to remember that asking more people in one survey helps reduce the margin of error, and analyzing the multiple forms gives a more accurate view of what people really think.

Gathering questions and compiling the survey

Before gathering questions and compiling the survey remember an important rule of human nature: the more questions you ask, the fewer respondents who start a survey will complete the full questionnaire. Research shows that the survey works best with less than 35 questions (source). Talking from our experience, the best response rate we got from the surveys with less than 12-16 questions. However, the most important criteria is not the number of questions. It is the total time users spend on filling the survey. It should not exceed 2-3 minutes.

There are many guidelines on how to formulate questions and how to lay out surveys to make them “respondent friendly”. The answers from user surveys must be relevant to the issues that are important. Therefore, if you can get the data from other sources (internal and external), avoid asking overlapping questions in the surveys. Create a survey that asks the right questions to meet your research objective. Make the form short, simple and clear as possible. Follow the “4 Be” rule: Be Brief - Be Objective - Be Simple - Be Specific.

No matter the main survey purpose and issue areas, in every our survey we want to cover the User Experience Honeycomb (by Peter Morville), which covers user experience (UX) basics. In order to provide a valuable and useful UX, the content on our website must be: useful, usable, desirable, findable, accessible, and credible. Therefore, we ask a block of questions like How clear are the guides and instructions on our website?, How difficult it is to navigate at the bottom of each our survey.

We also try to follow a few general-accepted rules:

Use closed-ended (yes/no, multiple choice, rating scale, etc) questions rather than open-ended (comments and essays). However, keep a few open questions. Closed-ended questions give you the better response rate, however open-ended provide the more valuable feedback. So, you have to juggle a bit here.

Consider what questions to use and when it is appropriate to use them. The sequence of questions should help to create a certain flow through the survey. I prefer to ask interesting questions in the beginning of the survey to catch the respondant’s attention. According to our observations, if a user started to answer the survey - he/she will the most likely make it to the end. Leading with interesting questions may help to stimulate interest. The most answered (the highest response rate) or interesting questions on Change.org include: Do you use your real name?, Why did you start your petition?, Would you create another petition in the future?, Would you use "I don't support" button to some campaigns?

Develop the concept of trust between yourself and the respondents. Maintain a neutral tone in all your questions. Notify users how much time it will take to complete the form and how you’ll use the results. Stay away from assumptions, personal statements, and complicated question structures.

Avoid leading questions, loaded questions, jargon or specific internal terminology.

The common practice is to place demographic and/or other sensitive questions at the end of the survey. If they are in the beginning, participants may opt out early. However, we tested a few surveys with different demographic block placement, and it performed much better (and the survey overall) being placed at the beginning.

Group similar questions together or in the same area of the survey. At Change.org we create different pages where we post different blocks of questions. The result is a demographic block, close-your-campaign block, start-your-petition block, etc. We name them accordingly: “Tell us about yourself”, “Your petition on Change.org”, “Please, share your experience with us”. When users see the title and a new page with questions, they can prepare themselves for a certain theme or topic.

Ask if a respondent is willing to answer more questions or provide additional feedback.

There are a lot of rules and techniques how for compiling your survey in the most efficient way. I summarized only a few important rules.

Finalizing and sending the survey

Customizing the survey design according our brand and theme makes the survey look professional. The right color fonts and the background make the survey visually appealing and user-friendly. Think how you going to spread the survey:

as a weblink shared through email or social media

embedding the survey into the website

invitation popup asking to take the survey when someone visits a specific page on the website

a survey window popup, containing the survey when someone visits a specific page on the website.

Each one of the forms above has its own specific, advantages and disadvantages. For instance, the survey window popup is the most convenient survey form, but works best only for a few short questions. The embed survey form looks very nice, but it might have its own formatting or interactivity limitations.

It is fairly common to offer incentives for completion in order to drive the participation rate up. A lot of companies use the subject line to emphasize the benefit to customers. Tempting subjects like “Get your holiday free” or “Win your chance..” instead of “Can you fill out our survey?” improve the response rate. We don’t offer incentives, and prefer to emphasize that the survey is easy to fill out (“it only takes about 3 minutes”). In the absence of incentives, implying that the cost or effort is very low can have a similar effect.

Before sending out the survey, we want to make sure the survey design and settings are working as expected. Therefore, we test the survey before it goes live. A small sample of test respondents (10% of sampling) can help verify if the survey is working properly. If something goes wrong on this stage it’s easy to revise and edit questions and/or the survey design. At this stage we can start analyzing results, revising or correcting the questions. If there are often-skipped questions, we can rephrase or get rid of them.

The final full send goes out to all our sampling list, excluding the first test send. At this stage we can’t revise the survey or change its format. Usually, over 70% results come in the first few days after the full send. We wait a week for the final and official survey analysis. That being said, some results can come even in a few months if your survey form isn’t expired.

Learnings

No matter what tool you choose for conducting the survey, you should be able to share and export results in different formats. First, we share results with the stakeholders, and then disseminate them with the rest of the teams. Once we have gathered the data, we can use it to evaluate the UX of the website, recommend improvements, discuss the recommendations, and etc.

In general we can conduct an online survey at any stage of the development process. We’re using the two main methods of conducting a survey, which depends on the long-term or short-term survey purpose. Once per year we launch a broad and global survey to cover mainly product questions and to ask about general experience. In addition, a few times per quarter we can survey a very specific set of users with only a few questions.

One of the most important survey outcomes is the establishing a relationship with our users. The next time we conduct the survey, the results should be better, because the best response rate comes from the audience that knows you.

Olga Chernenko

Following our recent launch of the Decision Makers feature, decision makers are better equipped than ever to respond and work with petition creators to help improve their circumstances.  Helping people target their petitions at the appropriate decision makers has become more important than ever.

Unfortunately, targeting decision makers is difficult because many leaders, politicians and other decision makers share names with different people.  A 2009 RapLeaf study found that there were over 10,000 people on Facebook named Juan Carlos, including the current King of Spain, two Argentinian politicians, a Pennsylvanian restaurant owner and a former Colombian minister.  We need a system to identify which Juan Carlos we intend to petition if we want a response.

Our solution:  Autocomplete.  Based on what our user has typed, we display a list of the most likely decision makers they might want to petition.

Showing a list of relevant decision makers, whose emails we have confirmed in our database, accomplishes two goals:

A better user experience.  The user of our platform sees visual reassurance and feels more confident about their petition reaching the decision maker.

The petition is delivered to the correct decision maker, who will receive notifications about petition activity and is more likely to respond. The petition has a better chance of achieving victory.

Our autocomplete solution:

We created a searchable index of our decision makers table using a full-text search engine, Sphinx, and its Rails gem, Thinking Sphinx.  We tell Sphinx which data fields to index, which other fields to include in the results, and how to weight the results.  For example, publicly-visible decision makers appear earlier in the results than non-publicly-visible ones.  In the future, we plan to prioritize local decision makers depending on the user’s locale.

The user starts typing in the “Whom do you want to petition?” field, which triggers an AJAX request to the Node.js server for autocomplete suggestions.  In the future, we plan to cache the results of previous such queries in the browser's local cache, to reduce the load this would cause on our servers. 

The Node.js server then requests search results from Sphinx (using the Node sphinxapi module), and sends the JSON list back to the web browser.

The browser displays the appropriate choices, along with hidden fields that precisely identify that petition target in our database.

We repeat this with every keystroke the user types to adjust their search.

If the user wants to petition somebody who is not in the list, the user can still add her or him as a new decision maker, whom we will store in our database.

We expect the autocomplete feature to increase the percentage of petitions that reach their intended decision makers, to increase the number of decision maker responses to petitions, and ultimately to help drive more victories.

Michael "Marick" Arick and Letitia "Tish" Lew Web Engineers

I'm a bit sentimental. Perhaps that's not the best way to be in an industry where just about everything is etherial and temporary. Over the years I figure that most of what I've created has been edited, refactored, deprecated or outright binned when something different or better came along.

Even so, the process of creation is what I most enjoy, and looking back on my work is more about the evolution of a codebase rather than being able to look at HEAD and stake an "I built that" claim. So when that sentimental mood struck me recently, I created a Gource visualization of our main Rails application, spanning my time at Change.org (a little over 3 years).

Gource generates some pretty beautiful stuff and it's fun to watch. As a participant, though, it also dredges up all sorts of memories. I see our team grow from 5 to 25, our jumps from Rails 2.3 all the way through 3.2, the pivot away from being a blog platform, the brutal initial i18n effort, and a surprising amount of effort from "YOUR NAME HERE" (which I have to assume is the software equivalent of underpants gnomes).

But mostly I see that evolutionary development process happening. The super-cool thing I wrote one day became the annoying legacy code later. I still take pride in every curly brace and variable name today, but I also find it useful to remember that, viewed longitudinally like this, it probably won't matter. What matters is that I'm participating, solving problems, and pressing forward. Having a big ego about that piece of code today is a little silly because, in a functioning engineering organization, it won't be "your" code for long at all.

You probably already recognize the value of user research—testing your product with users can confirm suspicions, illuminate difficult interactions, and even spur new product ideas. (Not convinced? Check out Erika Hall’s new book, Just Enough Research, to catch up.) Even so, the process of conducting user interviews can seem dauntingly official, expensive, or time-consuming. It doesn’t need to be any of these. Follow these six basic steps to get started.

Identify subjects. First, consider your primary audience. Are you targeting a particular age group, geographic location, or level of education? Focus your search as necessary. Then:

Ask people you know. Your housemate, neighbor, or coworker can offer surprisingly insightful feedback and you can’t beat the convenience. At Change.org, we occasionally offer unfilled interview slots to internal employees on other teams. They understand our product decisions and we identify their needs.

Head to coffee shops. If your interview is casual, ask people sitting alone at coffee shops. Most folks are eager to share their opinions, especially when offered a gift certificate (see step 2). At Change.org, we use this tactic frequently since it requires little planning but still gauges response.

Reach out to existing users. If you already have a user base, contact those people directly. Explain the value of their reactions, invite them to your office, and provide flexible time slots. When we want feedback from a particular group, like petition creators, we email our users directly.

Incentivize. While it’s not required, we like offering a simple incentive, like a gift card. Take a friend to lunch, support the coffee shop in which you’re interviewing, or consider a product-related discount. Our San Francisco office is near a Whole Foods, so we treat visiting users to gift certificates there.

Set up and record. We limit the group size to three Change.org employees plus the interviewee. Two-on-one is a better balance, but it’s often helpful to have engineers or product managers attend. Plan your questions in advance, but be willing to meander from the script. Designate a primary note-taker so at least one person can make eye contact. Capturing a video and screencast recording of the interview can be vital to share with your wider team. We like Screenflow and Silverback for this purpose.

Our design team captured this screencast of Morgane, an internal marketing employee, walking through a prototype.

Warm up. To start, introduce everyone in the interview. Offer your guest a beverage. Remind your interviewee that all feedback, especially critical, is helpful. Encourage him/her to verbalize every thought. If you’re concerned about confidentiality, sign a nondisclosure agreement. (Don’t get stuck on NDAs. Use the Shake app to create one instantly.) Begin your interview with general questions: “What do you do for a living?” “What kind of mobile device do you have?” “How many hours per day do you spend online?”

Get to the meat of it. Whether you’re asking general questions, showing mock-ups, or walking through a feature, resist the temptation to talk about yourself or your product. This is not a conversation—you should not explain how your product works or offer solutions if your user gets stuck. Simply acknowledge that you’re listening. To wrap up the interview, explain answers to previously asked questions. Pull for any additional feedback. Thank them for their time.

Dissect your results. Supplement your notes from the interview while it’s fresh, highlighting the takeaways. If you recorded the session, bookmark key moments for easy reference. Share your notes and recording with your team. While one session is helpful, user research is most valuable when trends emerge. Continue collecting interviews and referencing notes. Adjust your product as necessary; rinse and repeat.

Lauren P. Adams @laurenpadams Product Designer

I remember the olden days of change.org, back when we were a four-person engineering team at a company that hadn't yet figured out whether it wanted to be a blog or a platform for petitions.

Back in those days, we hired an outside contracting company to deliver our empowerment with a technology called "Flash" (you may have heard of it). The result was an embeddable widget that people could put on a Blogger site or AngelFire homepage. At the time, it was a really powerful and engaging experience that helped us bootstrap the Change.org community by being featured on everything from AlterNet to the tiniest mommy-blog.

Since then, we’ve built Change.org into the world's largest platform for change, our team has grown dramatically and we constantly challenge ourselves to focus on the most effective features for our community. Knowing when to throw away old things is essential and is actually one of my favorite parts of engineering and company evolution.

That time has come for the Change.org embeddable widget (a.k.a. "e.change.org"), which we will disable in January 2014. It no longer works on modern browsers, we heard from our users that it was slow, and it creates a tiny sub-fraction of the traffic on our site. So it’s time to remove the embed code from your pages, and raise a glass to the widget-that-was, which will soon be relegated to the archive known as our git commit history.

We are always looking at better ways to have ways for people to embed our content and share change.org campaigns (have you seen our API?). If you have ideas for what you'd like to see for embeddable tools to help your movement, leave a comment or send us a tweet!

Move fast and break things but don't let them get to production

We had the pleasure of attending Mobile Developer Day at Facebook’s Menlo Park headquarters. Parse and it’s tools were the main feature along with a hackathon, but the highlights for us were the great talks on design and product that were given including an impromptu visit from Mark Zuckerberg. We thought it would be great to walk through some of the things we learned with some excerpts from the talks.

Why is mobile so important?

Mobile devices have become personal items that we carry with us everywhere we go. And when I say everywhere, I mean EVERYWHERE. Surveys suggest 54% of average users use smart phones in bed, 30% use them at the dinner table and 39% use them in the bathroom. Their adoption is growing at an exponential rate and the devices are ubiquitous.

Why mobile fails & how can we make it better?

Most companies have very small mobile teams - generally 1 or 2 people or worse yet, none. It’s understandable for resource strapped startups to not have the necessary resources allocated to mobile, which is considered a secondary market.

But companies should realize by now that this is no longer the case. If you put together the overall usage of mobile & tablets, it easily surpasses PC usage.

MVP or MDP

The desire to release quickly has always led to discussions on what is the MVP (Minimum Viable Product) but what is the MDP (Minimum Desirable Product) which pushes the product forward and executes on core features.

To achieve great design and desirability, allow space for creativity and let your design be optimistic and ambitious. Let designers craft the best case scenarios then critique the designs and align the scope.

Use the space to engage the user completely as your product should be able to create the desired emotion in the user.

Help the users narrow the choices, but do not force your choices on them. Since many times this involves an ad or conversion point, embed these in the regular flow and take them through an experience ending where the user needs to be.

Release

Some companies have a vast user database and they can leverage beta users to test new products and get feedback. With startups it’s difficult, but that should not mean you test in production. If you release internally, followed by releasing to a small user base and use A/B testing you can craft a great experience which is vital since first impressions mean a lot.

Release regularly; there is no magic number. With websites, this could be twice a day, but with apps this could be every 4 weeks. Continuous deployment and scheduled release cycles can help you achieve this.

With regular releases it is easy to get small set of features out at a time and you can observe its effects, not just the desired ones but also the side effects.

Use the data to understand your user, but don't drive your product entirely based on the data.

Lastly, use intuition, user feedback, and most importantly, the interests of the people developing the product into consideration when developing your product.

Praneeta Mhatre http://github.com/Praneeta Web Engineer

We've standardized on using promises to manage most of the asynchronous interactions in our JavaScript codebase. In addition to really enjoying the extra expressiveness and aggregation possibilities offered by promises, we're benefitting greatly from richer error handling. We've had some surprises though, and this post explains some things which caught us out and some guidelines we're following to avoid similar situations in future.

We use the excellent when.js library, however this post should be relevant no matter what implementation you're using.

image via http://heartofaleader.wordpress.com/

1. Reject what you would normally throw

Rejecting a promise should be synonymous with throwing an exception in synchronous code, according to Domenic Denicola. We've previously built some promise interfaces which can reject with no value, or with a string error message. Rejecting with a string should be discouraged for the very same reasons as throwing strings. An empty rejection is the async equivalent of throw undefined, which (hopefully) nobody would consider doing.

These practices cause us additional problems with the likes of Express and Mocha since they expect their next and done callbacks to be invoked with an Error instance to trigger the failure flow, otherwise they consider it a successful operation. We, quite reasonably, are in the habit of chaining promise rejections straight onto these (e.g. .then(onResolve, next) or .catch(next)). If the promise rejects with no arguments then it won't signal a failure when used like this! 

Guideline 1: Always reject promises with an Error instance. Do not reject with no arguments. Do not reject with non-Error objects, or primitive values.

It's easy to feel like rejecting a promise is less 'severe' than throwing an exception, this impulse can lead to promises being rejected where normally you would not throw an exception (e.g. validation checks). If the above guideline is difficult to follow in a specific case because you can't think of an appropriate Error type to reject with, then perhaps it shouldn't be a rejection after all.

2. Understand how .catch works

We have run into trouble with catch (a.k.a. otherwise/fail for non-ES5 browsers). The documentation for it might lead you to think that somePromise.then(onResolve).catch(onReject) is equivalent to somePromise.then(onResolve, onReject). This appears true at first glance:

function onResolve(result) { console.log('OK', result) } function onReject(err) { console.log('NOT OK', err.stack) } // This: somePromise.then(onResolve, onReject) // Is equivalent to: somePromise.then(onResolve).catch(onReject)

They differ, however, in how they respond to errors thrown in callbacks. If onResolve throws and we are using the .then(onResolve, onReject) form, onReject is not invoked, and the outer promise is rejected.

var outer = resolvedPromise.then(function onResolve(result) { throw new Error('this is an error') }, function onReject(err) { // Never gets invoked }) // outer is rejected with the 'this is an error' error

If onResolve throws and we are using the .then(onResolve).catch(onReject) form, onReject is invoked, and the outer promise is resolved.

var outer = resolvedPromise.then(function onResolve(result) { throw new Error('this is an error') }).catch(function onReject(err) { console.log('FAILED', err) // => FAILED [Error: this is an error] }) // outer is resolved

So, throwing inside either handler will reject the outer promise with the thrown error.

Guideline 2: Anticipate failures in your handlers. Consider whether your rejection handler should be invoked by failures in the resolution handler, or if there should be different behavior.

3. Use .finally for cleanup

.finally (.ensure for non-ES5 environments) allows you to perform an action after a promise completes, without modifying the returned promise (unless the finally handler throws).

var outer = resolvedPromise.then(function onResolve(result) { throw new Error('this is an error') }).finally(function cleanup() { actionButton.enabled = true }) // outer is rejected with the first Error, as if the finally handler wasn't even there

var outer = resolvedPromise.then(function onResolve(result) { console.log('resolved') }).finally(function cleanup() { throw new Error('failed during cleanup') }) // outer is rejected with the 'failed during cleanup' Error

As hinted at in the example, this could be especially useful on the browser for re-enabling action buttons once an operation completes, regardless of the outcome.

Guideline 3: Use finally for cleanup

4. Terminating the promise chain

Guideline 4: Either return your promise to someone else, or if the chain ends with you, call done to terminate it. (from the Q docs)

This has bitten us many times when using promises in Express handlers, and resulted in hard-to-debug hung requests. This handler will never respond to the user:

function handler(req, res, next) { service.doSomething().then(function onResolve(result) { throw new Error('this is an error') res.json({result: 'ok'}) }, function onReject(err) { res.json({err: err.message}) // Never gets invoked }) }

Changing the then in the above code to done means that there will be no outer promise returned from this, and the error will result in an asynchronous uncaught exception, which will bring down the Node process. In theory this makes it unlikely that any such problem would make it into production, given how loudly and clearly it would fail during development and testing.

An alternative is to add a final .catch(next) to the promise chain to ensure that any error thrown in either handler will invoke the Express error handler:

function handler(req, res, next) { service.doSomething().then(function onResolve(result) { throw new Error('this is an error') res.json({result: 'ok'}) }, function onReject(err) { res.json({err: err.message}) // Never gets invoked }) .catch(next) // next is invoked with the 'this is an error' error }

This goes against the above guideline, since we are creating a new promise rather than ending the promise chain. You could argue that we trust next not to throw and as such there is no chance for the outer promise to reject. In addition there is no possibility of a hanging request or an uncaught failure (unless you throw undefined in either handler!).

If the idea of done bringing down a Node process makes you uncomfortable enough to ignore this 'golden rule' of promise error-handling, then perhaps this is a good option. The important thing is that errors do not hang requests, or get quietly transformed into successes.

Summary

We've seen 4 simple guidelines which should feel familiar if you've dealt with synchronous exception handling best practices. In fact they could almost be generalized to apply to both sync and async exception handling:

Throw meaningful errors (and a string is not an error)

Be aware of downstream effects of errors on the flow of execution

Cleanup as soon as the error occurs

Bubble exceptions up to a top level handler

This is a powerful feature of promises - letting us deal with errors in a stye that is more natural to us, as long as we are actually mindful of this, and remember to follow similar rules.

Jon Merrifield http://github.com/jmerrifield Engineer

Dian Andamari Rosanti Tjandrawinata* just joined Change.org as a Product Manager. Previously, Dian was at Flipboard, where she worked on both mobile and web products (and a ton of localization).

The Bay Area has been her home for the last 8 years, but Dian was born and raised in Jakarta, Indonesia. This means she LOVES spicy food and hot, humid weather. She also likes Southern food, baking, reading, making silly faces, singing her sentences, and long, quiet road trips.

Bob Tucker is our new Infrastructure Engineer. Bob joins us from BazaarVoice/PowerReviews where he was a Senior DevOps Engineer. He lives in Oakland with his dog Hiro, an Asteroids machine and an elderly Jeepster all next door to a urban digerati-hippie co-op complete with bees, turkeys, chickens and ducks.

Ernest hails from a sleepy little town just south of SF called Los Angeles. Previously, Ernest was awesome-izing the moviegoing experience as a designer at Fandango. Like Chelsea, Ernest made his way to Change.org via Designer Fund. He was really drawn to Change.org's mission, its ridiculously good-looking design team, and the opportunity take the mobile experience to the next level.

On nights and weekends Ernest makes every attempt to stay active -  dancing, hiking, biking, and swimming his way to a healthy cardiovascular system.

Last week I gave a talk at Data Week held here in San Francisco at Fort Mason: "ETL Validation with Cascading on Elastic MapReduce" (hey, I'm an engineer not a copy-writer!). It focused on how Cascading provides a useful abstraction atop of Hadoop that enables us to apply Test Driven Development in building and, more importantly, maintaining large, complex data transformations.

  The basic idea is how Cascading abstracts complex, multi-step map/reduce jobs into pipes, the individual units that actually do some processing, and taps, that connect data to your pipe assemblies.  This lends itself to a practical TDD-approach where you build up larger applications from individually isolated and tested components.

Developers from the agile world will already be familiar with the productivity boost that comes from being in "the flow" of TDD's red/green/refactor cycle.  But if it takes 90+ minutes to get feedback from your Hadoop job, your cycle feels more like a crawl. Additionally, requiring a local or network-accessible Hadoop cluster in order to test your latest changes makes integrating with 3rd-party Continuous Integration (CI) services a lot trickier. By providing a local platform runner which can run both unit and integration tests, Cascading shortens your cycle time to a much more manageable level, and makes integration with 3rd-party CI services as easy as pushing code.

Additionally, by using Gradle to handle testing and building our applications, we get Continuous Deployment in the Hadoop space. This may sound a bit frightening until you get used to the peace-of-mind afforded by good test-driven design. We integrate with CircleCI which allows us to have after-test hooks. A green build on master executes a Gradle command to build a jar for Hadoop and upload it to S3. The next time our ETL pipeline runs, the data verification step will use the new jar when initiating a job on Elastic MapReduce.

We're continuing to use Cascading here at Change.org for the safety and ease-of-use it provides in developing and maintaining complex Hadoop jobs over our increasingly large production data set. If problems like this and others in the distributed computing/applied data science space get you excited, we'd love to talk!  And come see us at Amazon Web Service's re:Invent where I'll be presenting about how we built an automated machine-learning driven targeting application for email campaigning using aws-swf, our lightweight Ruby framework for distributed applications on Amazon SimpleWorkflow. We'd love to meet you!

Vijay Ramesh Software Engineer, Data Science

Wilson joins the Product Team at Change.org from Yammer - an enterprise social network where he was a Product Manager.

In his varied past, Wilson co-founded a social enterprise in Uganda selling leather products, helped launch a global gap year program called Global Citizen Year, and spent a year cooking professionally around San Francisco.

It’s a really special day when your small website grows up and starts having to deal with large amounts of data and the challenges that come with it.

We have recently begun to experience some of these growing pains as our site has accrued more members. Where we were once able to get by with easy idioms that pulled entire SQL result sets into memory, we're now seeing our processes get killed by the operating system.

I personally got to feel the pain of these failures.

An incredibly important task, which was normally handled by Resque and was getting killed by the OS due to memory usage, was now being manually run by me in a Rails console. This worked as the process wouldn’t fall victim to the OOM killer, a monitor in Unix based systems that will kill processes that consume too much memory. This hadn’t been pleasant for me and it was actually a good motivator towards finding a solution.

The problem

The work needing to be done consisted of comparing two large sets of MD5 hashes. The intersection of these sets and also the areas where they don’t overlap both serve business purposes. Each of which have a few successive tasks that needed to be run shortly after the job finished.

The original job pulled resources from MySQL.

We started by initially looking at ways to optimize the queries. Our first optimization was to chunk the queries and compare them. This worked as a stopgap, but as the size of the hashes continued to grow on both sides, the amount of time it took to run the job increased to an unsustainable level.

We noticed another problem with this approach which seemed to be a memory leak. Every time a new batch was fetched and processed the memory footprint seemed to grow larger. We examined where this could be occurring and manually tried to flush what would have the potential to grow, but even this didn’t solve the issue. At this point the job was taking a minimum of forty minutes to run and in some instances upwards of four hours when it would even finish at all.

We needed to find a more sustainable solution.

Enter the wonderful world of key-value stores. Our first choice was Redis - a key-value store that hangs out in memory and provides very fast reads/writes. So with a little napkin math we figured our database should be able to hold the values we’d need in memory. We estimated it would be somewhere around 1.5 GB in size.

Fast forward two weeks, after everything had been implemented utilizing Redis, and we were in the midst of pushing to production. We ran our script for filling up Redis with the data needed and realized it had consumed 6GB of memory - four times the expected amount! The extra load would necessitate a more expensive server or require a dedicated server, neither of which is cost-effective.

Lesson Learned: Don’t rely on napkin math, do your due diligence and spike the proposed solution.

Knowing the efficiency of Redis storage, the 6GB size of the dataset didn’t seem like it was going to change drastically if we attempted to optimize the data some way. Some other solution needed to be found. We tried pushing the data with its more efficient structure into MySQL to try it out even though we already knew it probably wasn’t fast enough. Our suspicions were correct as it was quite slow and wouldn’t suffice. This is when we decided to look at something outside of our current stack.

We needed something that could handle lots of non sequential reads/writes with a hashed dataset and didn’t want it to be constrained to in-memory systems like Redis. We also didn’t want to sacrifice performance in accommodating these requirements. After looking at a few different technologies that met our criteria such as Riak and Berkeley DB, we choose LevelDB.

Working with LevelDB

The infrastructure required to get our solution operational was something we mulled over when deciding what to choose. We were able to get LevelDB running with a fraction of the setup one would normally associate with setting up a database server. It's simplicity also means it doesn’t provide all of the niceties that we’ve come to expect from other database servers.

A problem that we couldn’t avoid tackling was backing up the database if the server were to go down. There isn’t redundancy built in so we had to account for this ourselves by doing a basic cron based backup of the directory where the database lives. Also, with no consistency mechanism provided, we decided to only deploy it on one machine and have that be the workhorse for performing LevelDB jobs.

Our implementation backed by LevelDB has been churning through the jobs we throw at it for about 4 months now and it hasn’t fallen behind or been the victim of the OOM killer, which is great news. The jobs that previously took 40 minutes to 4 hours are now finishing in 10-20 minutes. in the future, we may choose something like Riak or Cassandra but for now my days are free of the interruptions the original problem caused. 

Eric Nicholas Web Engineer

Chelsea comes to Change through the DesignerFund Bridge program and previously crafted user experiences for TaskRabbit and WordPress.com.  One of her missions in life is to design behavior for social good, so Change seemed like a pretty good fit. She has an under-used degree in Journalism and takes pride in well organized CSS.

When not pushing pixels, she spends her nights surfing karaoke joints (trying to nail down that perfect performance), learning to play the mandolin, and playing in the San Francisco Berlin-style Ping Pong League. Her favorite animal is the red panda.