Have you ever bought a lottery ticket? I admit, I’ve played a few times. You won’t be surprised to learn I never did win the big jackpot. Seeing winners on the evening news gives the false impression that anyone could win, but the odds of winning are long—very long.

Rod Wolfe knows a thing or 2 about long odds. His friends call him “Lightning Rod” because he’s been struck by lightning not once, but twice. What are the chances? Well, you’re more likely to be the next Rod Wolfe than you are the next lottery winner.

The software industry has a lot in common with the lottery. We see big winners in the news everyday—Facebook, Uber, Airbnb. Their success bolsters our ambitions of making the next big product. Our ambitions are big and we act fast hoping to beat competitors to the market.

Software success hinges on a lottery-like collection of variables: the right product with the right features for the right audience in the right market. If you’re even a little bit off in your planning, you can end up wasting time and resources, and potentially put your company in a very difficult situation.

In this excerpt from the documentary DESIGN DISRUPTORS, veteran designers Daniel Burka and Mike Davidson talk about the problems of waiting until you ship a product to get customer feedback.

Optimistic that they already understand how to design a winning product and eager to get to market, many companies dive straight into production without spending time learning about customers and their needs. They base their designs on guesses that make the odds of success long.

---

If you’re going to solve a problem, you want sufficient information to solve it.

Erika Hall

Mule Design

“If you’re going to solve a problem, you want sufficient information to solve it.” @mulegirl

Guesses make messes

Buffer, a popular publishing platform for social networks, found itself in financial turmoil in part because they’d over-invested in products and features that weren’t relevant to their customers.

---

We have a bias toward action at Buffer, and believe that moving fast and being bold are important. Optimism has seen us through a lot of mistakes at Buffer, like the countless new features and products we spent months building only to realize we need to scrap them. Content suggestions and our Daily iOS app are just a couple.

But after a certain point in a company, the mistakes we make don’t just affect the product features. They affect people’s lives.

Joel Gascoigne

CEO at Buffer

“But after a certain point in a company, the mistakes…affect people’s lives.” @joelgascoigne @buffer

Buffer had to let 10 employees go and made painful budget cuts to recover. The good news is they’re starting to get back on track, but optimism and assumptions almost took them down. If you’ve spent time in the software industry you know Buffer’s story isn’t unique.

So many companies base their strategies on optimistic guesses and get it wrong far too often.

Guessing is expensive—if you’re wrong you could be out of business.

Guessing puts you at a competitive disadvantage—when you know little about the customers you serve, you know little about how to succeed.

Guessing is arrogant—you’re lying if you think you understand your customers without studying them first.

There’s a way to tweak your odds of succeeding, though. Rather than making assumptions about customers, we can start to learn from them. Customer research is easy to do and can be folded into any workflow—Sprints, Agile, Lean, whatever! As you start to think about customer research, you’ll probably find you have a lot of data already on hand that can inform your work—you just need to bring it to the surface.

---

You win a race at the finish line, not the starting block.

Laura Martini

Google

“You win a race at the finish line, not the starting block.” @martinibot

Guessing makes your odds of success long. Let’s stop playing the product design lottery and start getting the insights we need to make great products.

Here’s how to do it.

Research fast and make things

Customer research fits into every workflow, every role, and every company size. Whether you’re a designer, project manager, or director, the goal is to guess less and work from a position of being informed and confident.

---

When designers don’t know which problems to solve, we spin our wheels. We make products prettier when we could be solving customer’s needs and generating real value. So any company that’s serious about design should get equally serious about listening to customers.

Braden Kowitz

Google Ventures

“When designers don’t know which problems to solve, we spin our wheels.” @kowitz @GVTeam

There are 2 types of research you can do to learn about your customers:

• Quantitative: These are the things we can measure. Examples include analytics that communicate customer behavioral patterns and aggregate stats about customer cohorts.
• Qualitative: These are things that tell us about the qualities of a product or experience. Customer interviews, for example, give us insights about how a customer feels, which can provide a lot of insight into what motivates their behavior.

Think of quantitative and qualitative research as the Wonder Twins. They each have incredible powers, but they’re much more useful when they work together. Relying on 1 can sometimes lead you down the wrong path.

For instance, a couple years ago the user research team at MailChimp stumbled upon an interesting piece of quantitative data: many customers connected their Facebook accounts to their MailChimp accounts.

Based on the quantitative findings, the product team started to consider how to further the MailChimp-Facebook connection, but the qualitative findings from customer interviews told a different tale. Most customers only connected to Facebook because it seemed like something they should do given the social network’s popularity, but they never actually did anything meaningful with the integration. The product team changed course once the qualitative findings clarified the motivations behind the customer behavior.

Quantitative and qualitative findings

Numbers give the illusion of certainty, but they can be misleading if not verified with qualitative findings.

Surveys that impact product design

Surveys are a handy way to learn about your customers and can be conducted ad hoc or even automated. There is a host of different surveys you could run, but use them sparingly, as too many will alienate customers.

There is an art to creating effective surveys, and the Google Ventures team has a wonderful guide that will help you avoid rookie mistakes as you begin this practice.

Tips for building effective surveys

• Start simple with clear goals about what you want to learn.
• Keep your surveys as short as possible to get better response rates. “Nice-to-know” questions should be cut, as they just increase the length of your survey.
• Never ask respondents for information you could get yourself. For example, don’t ask when someone signed up for your service if you already have that info in your database.
• Randomize answers to question to avoid response order bias.
• Conclude with an open-ended question like, “Is there anything else you want to tell us?” to give respondents an opportunity to surface interesting issues that may surprise you. This is a great way to find good candidates for interviews.
• Run a pilot test of your survey with a small sample of people before you send it to everyone. This will help you find questions that may be missing response options or identify places where things aren’t clear.
• Spend time carefully writing the email asking customers to take your survey, as it will greatly influence your response rate.

Automated surveys

Who they help: Everyone!

Why they’re useful: After you set up an automated survey, data keeps streaming in, giving you fresh insights regularly.

Data kung fu

Use a tool like Zapier to forward all survey responses into a shared Google Sheet, or even an Evernote account where your team can search through all responses.

Types of automated surveys you might send

• Net Promoter Score: Learn about your customers’ loyalty to your brand. Delighted is a lovely tool to run regular NPS surveys.
• After sign up: Find out why customers signed up, and from which competitor they’re switching. This is useful for marketing teams as it helps identify the language that motivates buying behavior. Send this within a few days of sign up while their memories are still fresh.
• After account closing: Find out why they’re leaving. Is it you or just circumstance? Are they switching to a competitor? Link to your survey on the page in your app confirming their account is closed.
• Topic specific: Use a tool like Ethnio or Qualaroo to deliver a micro-survey to specific customers. By placing a small code snippet on a carefully selected page in your Knowledge Base, you can find a customer with expertise on almost any topic.

Ad hoc surveys

Who they help: Teams doing a deep dive on a feature or topic.

Why they’re useful: They can give you an aggregate view of customers’ thoughts on a topic, and help you find outliers who may make for good interviews.

Smarter response filtering

Ask a question or 2 at the beginning of the survey to help you filter responses later. For example, a question like, “How old is your business?” or, “About how many people are in your organization?” can expose different responses from various customer cohorts.

Ad hoc surveys can be conducted in many ways. You can send an annual survey to collect customer data to inform projects throughout the year. You could also send a survey to gather insights or guide development on a specific feature or new product.

You needn’t survey all of your customers to get the results you need. Surveying too many people will produce lower response rates and introduce unwanted noise into your data. Instead, use your customer data to target the right people for your study before you send.

For example, want to learn more about customers who sell things online? Find a segment of those customers who have a shopping cart platform, like Shopify, integrated with your app. Want to hear from customers who are highly engaged with your product? Segment by ‘times logged in this month.’

Netflix, Airbnb, and Intuit have all used automated and ad hoc surveys to inform their work.

Customer interviews

Customer interviews deliver a wealth of information that will help you design more successful products. They’ll give you a glimpse into the emotions that drive customer behavior, help you understand your customers’ workflows, and let you hear the language people use when describing your product. This is essential stuff!

Finding your next product idea

Customer interviews are the best way to find the idea for your next product or the vision for the future version of your product. Getting a glimpse into the daily lives of your customers will show you where they struggle and the opportunities for designing products that will solve their problems.

But your time is limited and you probably can’t spend weeks talking to dozens of customers. How can you find the people with the most insight? The answer lies in your survey responses!

Your survey generated data from a variety of customers who can help you better understand how to design your product. Drop your survey response data into Excel and filter to find any of these types of customers:

• People nearby you can visit in person
• People who just signed up
• People who just closed their account
• People with interesting traits, behaviors, or off-the-wall responses
• People who’ve said they would or would not recommend your product to a friend

When you’ve found customers of interest, send them a short, personal email asking to learn more about them. Interviews by Skype or Google Hangouts can be conducted in a conference room where your whole team can listen in—they’ll comprehend the feedback more easily if they hear from the customer themselves. About 20-30 minutes is all that’s needed for a phone interview. It’s always a good idea to record interviews so you can reference them later.

Visiting customers in person takes a bit more time, but can be eye-opening. You’ll get to see the hardware they use, the distractions of their office, the flow of their day, and meet some of their colleagues. The entire experience will be a vivid reminder to you and your team that you’re designing products for real people.

Tips for conducting customer interviews

• Customer interviews needn’t always be connected to a project. You can dedicate a day or 2 per month to talk to customers to keep your team in the habit of learning.
• Limit the number of people conducting the interview so you don’t overwhelm your participant.
• Assign a person to take notes so the person asking questions is free to drive the conversation.
• Watch for signs of an energy change from the subject, raised voice, the use of profanity to punctuate a story, leaning in to emphasize a point—these indicate what’s important to your customer, and directs you to ideas for refinement or even new products.
• Bring a voice recorder to capture the interview so you don’t feel compelled to furiously capture every word.
• Each interview will yield 1 or 2 golden insights. Don’t get lost in the details—train your ears to listen for the meaningful insights.
• Use the Switch Interview technique to learn from people who just bought or just left your product.

While at MailChimp, my team noticed a small trend of customers departing us for more complex and more expensive competitors. Using surveys, we recruited customers who had just recently closed an account and cited a competitor’s platform as the reason. We set up 60-minute calls with a handful, and spent 2 days interviewing.

The survey pointed us in the right direction, but the interviews provided the missing link: these customers weren’t leaving because of the app’s shortcomings; they were leaving because of a perception problem. They mistook the simplicity of the app for a lack of sophistication. These former customers were looking for a complex tool to make them feel like the accomplished professionals they are. It was eye-opening, and helped illuminate a new product direction for MailChimp.

Existing data

Sometimes guessing less simply means becoming aware of the data you already have. That’s exactly what happened at Bambora, a new global payments company based in Stockholm. Creative Director Anders Färdigh and his design team craved more insight to guide their work, but the thought of building a dedicated research team felt premature. Maybe there was a simpler starting point?

During a meeting with their COO Patrik Göthlin, Anders discovered that much of the insight his team needed was already being gathered. Patrick’s operations team was doing extensive Net Promoter Score research, surveying Bambora’s customers to determine their loyalty to the brand, and following up with detractors to learn where they were falling short. They’d even been visiting customers in person to capture feedback about their products. There was so much information already on hand to help the design and product teams prioritize their work.

In large organizations, it’s hard to know what research is already siloed within other teams. That’s why it’s important to spend time talking with colleagues on other teams to learn about their work and the research already underway.

Get started by talking to people in these teams

• Sales: These folks talk with customers all day. They’re collecting insights about product shortcomings and data about every potential customer. You may find that the data the sales team tracks in Salesforce could help you identify interesting customers to interview.
• Marketing: Analytics often falls to the marketing team to track. They can give you access to Google Analytics and other tools that may help you understand customer pathways and raise questions about interesting customer behaviors. Marketers are often at events talking with customers, and may have insights to share with you.
• Customer service: Few teams have as much actionable information for refining your product as the customer service team. They hear the struggles of your customer daily, and they know what themes are strongest. Make a habit of talking to many customer service agents to get the broadest perspective on your customers’ pain points.
• Data science: Your customer database is a goldmine of information. If you have a data science team then chances are they’re already querying that database to find customer cohorts. This team will be your most treasured ally as you dig deeper into customer research.
• Engineering: You’re probably already working closely with the engineering team on the product, but you should also be talking to them about the data they could be logging for you. Curious about which integrations customers connect first to your app, or failure rates of a particular workflow? Your engineering colleagues can probably log that data for you and have the app email you a report.

Learning in the background

There are a number of handy tools that can collect data for you behind the scenes, giving you yet another source of information to tap when you need it most.

Fullstory

Have you ever wished you could just watch your customers using your product to see where they stumble? Fullstory gives you that super power. It’s like a DVR for your app or website—it captures every session in your app and lets you play them back.

Want to see sessions of every customer who’s signed up in the past week and clicked the help button? Or maybe you want to see sessions where a specific error happened so you can diagnose the problem. Fullstory has incredible search options to help you filter sessions to find exactly what you’re looking for.

Intercom

Intercom’s customer feedback features can help you get in touch with customers who have taken specific actions, so you can then ask them about their experience. Once you’ve defined the rules for the customers you want to speak with, you can create automated messages to be sent via email or in-app. All of the feedback lands in a shared inbox, where your team can tag and organize messages to identify patterns.

Builtwith

You can learn a lot about your customers just by examining the tools they use, and Builtwith will help you discover every technology your customers use to power their website. When a new customer signs up for your product, pass their URL over to the Builtwith API to retrieve and store their tech stack data for your research.

Knowing that a customer is using Shopify, MailChimp, and Zendesk gives you clues about their business—they sell stuff online and have marketing and customer service teams; they’re using tools that are very design focused; and they’re using tools that are DIY, so they may not have developers on staff.

As you look at the tech stacks of your customers you’ll begin to read them like tea leaves, giving you clues that can guide you to the right interviews and aggregate data to understand customer cohorts.

Against all odds

You know why the lottery is fun despite being a losing game? There’s not much at stake. I know the odds are wildly against me, but when I lose I’ll only be out a few bucks. Rent will still get paid.

Product design also has tough odds, but the stakes are much higher. If your product fails to get traction with your customers, you and your colleagues could be out of a job.

“Fail fast” is the mantra of the software industry. The surest way to achieve that goal is by basing your product design on guesses. Successful, design-driven companies are doing just the opposite—they’re succeeding fast by guessing less. They’re finding ways to inform their work each step of the way, tapping into existing data, setting up systems to continuously gather feedback, and they’re talking to their customers all the time.

Research isn’t some monolithic, academic process. It can fit into every workflow, every role, and every company size. Whether you’re a designer, project manager, or director, the goal is to guess less and work from a position of strength by being informed.