dSetia UX musings

Introduction

Over the weekend, I created a tool called Relative ROI Calculator to help Product Owners (PO) of Scrum teams to quickly prioritize user stories based on their ROI (compared against one another).

This tool is meant to provide a framework for team discussion to answer the question: What are the most important things that we should we work on next?

The problems with current backlog prioritization methods

Most Scrum teams are good at estimating the amount of effort to complete each story, but not very good at estimating the value that the completed story will produce. This causes the product backlog prioritization to be misaligned from its mission and purpose of the product. 

Few typical prioritization methods: 

Based on ease/cost of development (low story points first)

Based on Product Owner’s preference first (i.e. flashy feature that’s trendy now)  

Based on his stakeholders’ demand (i.e. the ones that would make the stakeholder’s number look good). Essentially the PO is using product development as a way to gain favor from stakeholders.

Based on team consensus. The team argues for what should be prioritized higher. In this case, the loudest and most articulate voice wins.

The first problem that could arise is the team gets demotivated. If the prioritization is dictated by the PO based on what he considers important without an understood set of criteria, the team can lose trust and get demotivated. The same goes to using the team consensus method. The team members who can’t articulate their arguments very well or not vocal may be discontent. To motivate a Scrum team, you need to create an alignment of purpose. If everyone in your team understood why the backlog is prioritized a certain way, which should be aligned with the bigger purpose of the product, then they would be motivated to fight to get the stories done.

The second problem is that the end-users (customers) are not considered. None of the methods mentioned above take the end-users into account. Without this, how can we honestly say that our backlog is prioritized based on what deliver value for our customers (1st Agile principle) ? And if we don’t deliver value for our customers, how can we capture value from them as a business?

Using Relative ROI  to Prioritize Your Backlog

To calculate ROI on the story level, we need both the cost and the value that the story will give us. For the value side of the equation, we need both the value for the business and the customers.

This concept is based on Goal Centered Design Process by Alan Cooper and the Design Thinking approach of IDEO.

The Relative ROI Calculator, has all 3 pieces :

Business : Conversion funnel metrics based on Dave McClure’s Pirate’s Metrics

Customer : How many users are affected and how often based on Dr. David Travis’ Red Route Analysis

Cost  : Story points 

All you have to do is answer the questions (filling out a short form) then the calculator will give you a Relative ROI score. After you do this for few stories, you’ll see how they compare against one another in terms of the ROI.

The Relative ROI score doesn’t mean anything in isolation. It's meant to be used as a tool to compare one story against another. Similar concept as the story points.

When to use this tool?

This tool is meant to be used as part of tactical planning, when you already validated the customer segment (persona) and their problem/motivation (needs/wants). You also have validated that your product does solve your customer’s problem either through prototypes/vaporware or if you have an existing product.

You have a product backlog populated with user stories as a result of doing the strategic work of gathering insights/requirements from customers and business stakeholders. Now you need to prioritize them so your team can start working on them.

How to incorporate this tool into your workflow?

1. PO working solo. It can be used by the PO independently to prepare the business case for the user stories that he will then present to the team during planning to back up his prioritization.

2. Workshop with team. The recommended use is to use this tool as part of a workshop with the team to create an alignment on what they collectively consider important. The fields in the tool’s form would give 

Feedback?

User Research

Surveys: Google Form, TypeForm

Recruiting: UserInterviews.co , craigslist

User interviews: Google Hangout , Skype

Getting quick directions: UsabilityHub

Remote usability study: Usertesting.com

Video editing: iMovie

Workshops

White board, marker, pen, paper, dot stickers, timer, masking tape, post-its

App demo (competitive analysis): Reflector 2

Remote affinity mapping: Stormboard

Diagraming/Visualization

Omnigraffle 

Prototyping

Visual Design: Sketch, Photoshop

Animation/transition: Keynote

Flow: InVision, PoP

Micro interactions: Pixate, FramerJS

Mobile preview: Skala

Workflow/Project Management

Automate design specs: Zeplin

Visualize user tasks/kanban board: Trello

Agile board, bug tracking: JIRA

Distribute in-dev build: HockeyApp

Distribute Beta build on iOS: TestFlight 

File sharing: Google Drive, Dropbox

Communication: Slack

Validating Demand

Marketing landing page: Launchkit

Generate traffic: Facebook ad, internal ad

Customer Insights

Bug reporting: Instabug

Session recording: Appsee (app), HotJar (web)

Conversion funnel: MixPanel

Top level analytics: Google Analytics

NPS: Usabilla

Similar to how vanity metrics are the metrics that look good, but don’t lead to actions, vanity UX deliverables are those cool looking UX outputs that don’t lead actions.

This article is written with other UX professionals in mind with the intention to elevate the value of what UX delivers (not just deliverables) to the business.

Why create UX deliverables?

There are only 2 purposes why a UX professional would create a deliverable/artifact:

1. To understand 

When we design a product/service, at the heart of it we’re dealing with systems that involve customers’ behavior . Visualizing the system we’re planning to build and how to it’d interact with existing systems is a great way for ourselves to understand how these systems work. For this purpose, the visualization needs to be comprehensive because we want to identify potential gaps in our concept. 

2. To communicate 

The second purpose of creating a deliverable is to communicate the concept to other people in the company (team members, executives). Why do this? So the other person on the receiving end can be on the same page on the stage of the project and take the necessary actions (i.e. creating new stories for the backlog, approving a budget, implementing a design etc.). For this purpose, the deliverable needs to be as simple as possible to make sure the message we’re trying to deliver is not lost and gets delivered while we still have the recipient’s e

How to avoid creating vanity UX deliverables

Start with the persona

This is what we do day-in and day-out as UXers. We research and empathize with the end-users to create something that they can use. Having the deliverable’s user in mind would help you choose how to design it: what to include, the terms and symbolism to use, how comprehensive it needs to be.

Optimize for task completion

As part of thinking about your deliverable’s persona, ask yourself “What is the task that they need to complete when they use your deliverable?” For a developer, it could be to build a feature. For a product manager, maybe to prioritize stories for the next sprint. For executive, probably to decide whether to fund the project. The goal is to communicate just enough information to allow the persona to get to task completion with minimal effort.

Focus on core task

Don’t try to create a deliverable that everybody can use. Just like designing for anything else, this is doomed to failure. The more personas you need to design for, the less effective your communication effort will become.

Focus on communicating one concept for helping one persona complete one task. By doing this, you’ll be able to remove all extraneous information that’s irrelevant to the target persona (and would only confuse them).

This means, potentially you could end up creating multiple flavors of the same deliverables for multiple personas. 

Test it

And lastly, as with everything that we do in Lean, test the deliverable’s effectiveness by showing it to your colleagues and asking them to describe what they see.

Take away

If you find yourself creating a type of deliverable that doesn’t help with the process, stop doing it. At The Economist, we no longer create wireframes for this reason. It made our team look smart and comprehensive, but in reality we could move faster by moving from sketching to prototyping.

Congratulations! You have reached product/market fit and are getting good traction. 

The funding you just received gave you an additional validation from the investors and officially kicked off the growth stage of your startup.You’re expected to scale up. Scale up your customers and with that, revenue. 

What’s the role of UX in this chapter of your startup’s adventure?

Identify risks

UX’s role here is to make sure that there’s no major usability issue in your current product. Your UX team can do this by visualizing scenarios, user flows and task lists. 

This would allow your team to identify and prioritize the tasks to address highest risks of churn. This can be quickly verified by running a remote usability study.

Optimize conversion

After getting the product to the place where you’re confident that it’s usable, now you need to optimoze your acquisition and activation funnel. This way, when the leads start to come - as the result of your marketing effort - you’ll be sure you’re converting them into customers.

The UX team can create mini task flow to zoom in on the activities that potential customers do on your landing page. They also can test and optimize this page using usabilityhub’s various tools to see if it looks credible, communicating the right message/impression, is missing any key info etc. 

On the activation side, your UX team can watch session recordings of customers who became regular users to identify patterns of behavior that contribute to turning them from somebody who’s checking out the product to a believer.

Again, these should be further validated by doing in-person interviews. They could piece the 2 (acquisition and activation) together as a single flow during the test to see if there’s any issue that cause drop-offs.

Identify new growth opportunities 

Now you’ve got a robust product that converts leads to customers. What’s next?

More growth of course. 

Your UX team can now do research to identify ways to grow your customers. A couple things that they can explore:

New customer segment who have the same/similiar problem

New problem for the current customer segment that you can easily solve

They can potentially use your current product to test the new hypotheses during user interviews. This way, you don’t even need to build anything to start exploring new opportunities.

Conclusion

So you've done your research and have a winning product idea that you and your team are ready to build (let's say you just finished a design sprint). How do you get it out?

Here's how:

1.Concept Prototype

Users: usability test participants recruited based on persona

Create a realistic looking prototype to test the concept of your solution. Nothing in code at this point. Use InVision, Keynote and POP or other tools that allow you to fake the experience without having to build anything in code.

This method will allow you to iterate on your concept quickly and cheaply. Once you're sure the solution is usable and getting the reaction that you aim for, move on to building a walking skeleton.

2. Walking Skeleton

Users: product team -> internal -> super fans

A walking skeleton is a very basic build that includes only the essential features needed to measure the product's effectiveness (is it solving the problem we set out to solve?). You'll need to experiment with many ideas so make sure your dev team doesn't build a production-ready code that limits your ability to swap in-and-out features you want to experiment with. Hack the back-end: Use manual process or hard-coded content so you don't need to spend time building a system that may not work.

Naturally, you'll need to have the tools to measure both qualitative and quantitative data installed. Additionally, you'll also need a bug reporting tool at this stage.

You'll release it first to the product team to catch bugs. After you've addressed major bugs, recruit help from internal people in your organization who are not familiar with the project to help you identify unaccounted use cases (and catch more bugs). Next, reach out to your super fans and gather feedback from them as the first group of real customers.

3. MVP

Users: Super fans, early adopters (or small % of users for existing product)

At this point, you know your product is solving your user's problem and you've added more features to make this product viable (i.e. authentication, checkout system, etc.).

You'll get your super fans to continue providing feedback and recruit more people who can be chosen randomly or self-selecting (i.e. recruited via sign up form or survey)

As you implement more features, make sure you continue to monitor the effectiveness of your core tasks.

4. Robust product

Users: everyone that fits your persona, 100% of user base

Your product is now public and marketed heavily.

At this point, you'll be monitoring the end-to-end customer journey beyond the user experience within the product: Marketing landing page, app store page, onboarding until conversion.

We recently finished a design sprint for a new product concept that aims to solve a big problem that our existing products have.

Design sprint recap

After doing some extensive pre-sprint research, both talking to internal stakeholders and our target audience based on our personas, we booked 2 days with key stakeholders in London to do a workshop. 

We got a good team of stakeholder together representing editorial, marketing, product and advertising. We discussed and defined the problem together, we charted the customer journey, sketched and ended up with a storyboard that we felt good about.   

Then we spent a couple days prototyping the concept and preparing for usability tests in New York that we’d scheduled before we left for London.

We did 3 days of usability testing with a total of 8 participants. between the first and second day of testing, we iterated on the prototype based on the glaring reaction we observed during the study.

At the end of the usability tests, we were convinced that the solution we came up with addressed the perception issues that we discovered during our early user research. We also saw that all participants were able to complete the tasks tested without major issues. 

Now what?

Does this mean we’re ready to build the product, ship then market it? Not quite.

We have a usable solution in our hands, but we still don’t know if it’s useful or not. Just because it’s well-designed and easy to use, doesn’t mean that people will actually use it.

People will use your product only if they see value in doing so.

How do we figure out if our product delivers value for them? We ask them to pay for it.

This means, we need to have something real they can use outside of the controlled lab environment. Something that they can use at home when no one is looking. 

We need to have an alpha build.

It’s a real (but hacky) version of the product. It only includes the features that we need to start measuring what users actually do on our app.

Quant and Qual running parallel 

Once we have that, we’ll release it to a small group or closed alpha users then experiment with tons of features to increase the value of the product and ,on the other side of the equation, different ways of making them pay for the value.

In parallel to this, we’ll have a monthly usability study so we can investigate the behavior that we see further. In other words, the data from analytics will show us what they alpha users do, the usability study will give us the opportunity ask them why.

Few years ago, at SXSW, I had the opportunity to hear Scott Jenson speak about UX. He boiled it down to UX = value vs cost.

Then a couple years after that, I heard Steve Cohn spoke about Lean product development at Lean UX conference in NY. The main thing that I still carry with me is the way he talks about validating ideas by making people we test our product with pay with : time, reputation, or money. One more that I’ll add to that list of cost is “enjoyment”. Unless it costs them anything, they won’t give us a reliable response.

What I just realized recently is that the two are connected. An product user experience is a transaction where people pay with something (cost) to get something that they want (value).  

For example:

1. Uber passenger | cost: money , value: time/effort

2. Instagram | cost: time/effort , value: reputation