7 Key Lessons on Early Testing & Development
Now that the Acquaintable Alpha is in the rearview mirror, I figured I’d share a few thoughts and reflections on what I’ve learned from the Acquaintable Alpha. I endeavor not to be didactic; my goal is to hopefully inspire a few entrepreneurs to think about and consider the lessons below.
1) Users are dumb (and I mean this in the nicest way possible, my beloved Alpha Testers):
Thanks to the likes of the amazingly brilliant entrepreneurs and technologists that paved the Web 2.0 and mobile revolutions everyone is god-damned spoiled (I’m looking at you, Google and Apple)! Users expect things to be drop dead easy, simple, and intuitive. We quickly realized that we undoubtedly would be (and should be!) held to the same standard as the best in the business. While we are innovating to create a novel user experience and a completely re-imagined way of social networking, we strive to make our user experience inherently intuitive as possible. One of the phrases in Apple’s genius marketing campaigns for the iPhone and iPad is the phrase “and you already know how to use it”. We are far from this, but usability has come a long way thanks to your help!
2) It’s not rocket science, it’s just science:
I went to a math and science high school, and 4 years of physics, 4 years of chemistry, and 3.5 years of biology every weekday ingrained the scientific method in the way I develop knowledge. Product design is no different. The Alpha was all about developing well-reasoned hypothesis, testing them, and determining whether they were valid or invalid. There are a few things here in agile development that weren’t obvious, that I’ll attempt to explain:
Isolate variables: Just as in science class, if you don’t isolate variables, you can’t meaningfully prove or disprove your hypothesis. This is why it’s good to iterate frequently, change one or a few things at a time (hopefully, that are not related to each other), so you can determine the effect of the changes without diluting the efficacy of the experiment.
Quickly change the experiment: However, unlike science class, the pace of agile development, especially when you are just starting out and proving different ideas, should be fast – so the whole process needs to speed up accordingly. The faster you can draw conclusions and quickly and change the experiment the better. You don’t have to be right (there is no “right”), you just have to be closer to the right answer than you were before, then on to the next one.
3) Visibility/metrics are key:
Jack Welch famously said “you can’t manage what you can’t measure.” What is true for a Fortune 500 company, so is true for a 3 person startup. We are addicted to metrics – I mean addicted. When I wake up, before I go to sleep, whenever my mind is not occupied by some other pressing task, I’m pretty much checking metrics and trying to glean insight from user data. I don’t just look at metrics every so often, I watch them to see how they are changing as we push code, and every day. I think it’s crucial to develop a “feel” for how users are using your site by monitoring usage and endeavoring to understand how and why metrics are changing over time.
4) Engage and listen to your users!
Just as I have one finger on the pulse of the metrics, I have the other on the pulse of the user – actually talking to users every day. Further, getting real-time feedback is key – I mean on the phone, or sitting next to users as they experience the application. It’s important to not only ask questions, but to watch people use the product and to have them speak out loud to describe their thoughts as they use the product. I’ve learned more from the knee-jerk, stream of conscious thoughts of users than I could have ever doing post-experience interviews or surveys. It’s easy to engage 25 Alpha users, but I aim to weave the theme of “engaging and listening to our users” into the culture at Acquaintable.
5) Leverage your development team:
Engineering time is the most scarce and valuable resource in most tech startups. My goal every day is to make our development team as good and efficient as possible. It’s the first thing we talk about when we meet, it’s how we prioritize tasks, it’s the basis of how we run the business. Development efficiency is close to Gospel at Acquaintable. If you are a business person working with a technical team – simply refuse to be the bottleneck. A whole bunch of work has to be done thinking through the vision, listening to user feedback, designing features, wire-framing, prioritizing tasks, etc., upfront – but this NEEDS to get done, and done well, in advance of the development team’s task queue being empty or being filled with incomplete or unimportant tasks. It’s a colossal disaster if development time is idle, or worse, wasted because a feature wasn’t thought through enough, not enough feedback was collected, or a test was not designed well. I can’t understate how important planning and prioritization is to enable this process to happen efficiently. No matter how talented and hardworking you are, if you are not working on the right (and most time-sensitive) tasks, then your work is futile. Bottom line: Whatever you can do to make your engineering team more efficient, do so.
6) Ship at 80%, then iterate and test:
“Ship at 80%” quickly became a mantra at Acquaintable. When designing features, we realized that getting to 80% was easy, and that the last 20% to get to “perfect” (I say that quotes because there is no such thing as idealistic “perfect”) would take twice as long as the first 80% and be totally useless given the rapid pace of iteration. Classic 80%/20%, but a great application of it. Don’t strive for perfection, strive to get to 80% as quickly as possible, and iterate from there. Which leads me to my last point (a related notion)…
7) Go with your gut: In any startup that’s early, there is never enough data, so you have to shoot from the hip. More importantly, you have to get comfortable doing this. I’ve heard Elliot Katzman, a General Partner at Commonwealth Capital Ventures, advise some of his portfolio companies by saying “Making the wrong decision is sometimes better than not making a decision at all.” I think this rings even more true with companies in their “startup” stage. You have to pull the trigger, make a decision (any decision!), and if your testing/iterating process is honed, you’ll get to the right answer sooner than you would have if you had just thought about a bit more.
I realize that many people smarter than I have written about similar things. I have not had time to properly research and cite competing or complementary posts or arguments. If you know of any blog posts, articles, etc. that touch on early testing & development and incorporating user feedback through rapid iteration, please post them in the comments and participate in the discussion. It would be great to have a few different viewpoints on the topic. Thanks!
