Best Practices Tech

What did we learn after running 2,000 experiments for Fortune 500 product teams?


The ‘Lean’ movement has taken the corporate world by storm, but there are still countless obstacles for product groups that seek to undertake its experiment-driven ethos and make selections informed by buyer knowledge. That’s why two years ago we began building Alpha, a platform for Fortune 500 product teams to show hypotheses into buyer perception inside 24 hours with out having to tap any inner capabilities or navigate compliance obstacles. In the process, we’ve discovered a substantial quantity about corporate tradition, the character of consumer research, and product administration processes.

At this time, our shoppers embrace forward-thinking product teams from AT&T, Capital One, PwC, Aetna, and lots of others. Just lately, they collectively surpassed 2,000 experiments on our platform! After producing roughly 660 prototypes, they obtained feedback from almost 400,000 customers. The outcome: 46,000 minutes of video from moderated and unmoderated interviews, 6,500 charts, and lots of of undoubtedly sensible — and informed — product selections.

We spent some time mining our databases (for what we name ‘experimetrics’) and reflecting on shopper conversations to extrapolate what we’ve discovered along the best way. Under are the seven most meaningful and actionable insights we discovered:

1. Change is troublesome. The previous adage is painstakingly true. As a startup, we now have to keep in examine our myopic perspective of the world — speaking to clients may be an natural part of our job, however, as we discovered, that’s not often the case at a big organization.

Regardless of long believing in the worth of speedy prototyping and experimentation, Fortune 500 product managers usually operate in environments with many competing priorities. Consumer research is usually expensive and executed by inner groups or businesses in month-to-month or quarterly cadences. The power to show around analysis in lower than every week, but alone a day, is totally unprecedented.

And whereas ‘on-demand consumer insights’ sounds appealing, in follow it challenges many company conventions, probably the most entrenched of which is the bias to overplan. When research cycles take months, it’s critically necessary to make it possible for every facet is rigorously crafted and vetted. But once you speed up that process to a matter of hours or days, iteration eliminates the necessity for exhaustive planning.

Our knowledge illustrates how troublesome this modification in mindset and conduct may be. At full capacity, particular person product groups execute about 8–12 experiments per 30 days on our platform. Even with workshops and in depth onboarding, it takes anyplace from three to six months for shoppers to succeed in that bandwidth. Positive, some of that time is spent figuring out how you can shortly turn knowledge into selections. But the overwhelming majority is consumed as a product workforce culturally and practically shifts from waterfall to agile experimentation, recognizing that deliberate research pales in comparison to iterative analysis. Spending two weeks outlining customer analysis that may inevitably be flawed is not any match for six iterations that can be executed in the same timeframe.

On our podcast, That is Product Administration, Cindy Alvarez, Director of Consumer Experience at Yammer, echoed one of the crucial widespread sentiments about working towards ‘Lean’ and ‘Buyer Improvement’ inside a large organization. She urged listeners to stop planning and just go begin talking to clients as a result of it’s inconceivable to get better at doing it otherwise.


She’s absolutely right and it’s a technique we are closely invested in. We began pre-populating new shopper accounts with analysis executed for them, including customer insights into aggressive benchmarks and usefulness throughout their respective merchandise. Thus far it’s been a useful ignition for product teams to begin iterating.

2. Typically, formality trumps informality. Continuing the theme from the previous perception, we’ve discovered that, even once shoppers hit full velocity, it doesn’t quite resemble the cadence of how startups follow experimentation. We initially designed the product in order that any stakeholder might easily submit an experiment on an ad hoc foundation, which is analogous to how we operate. As an alternative of operating impromptu experiments though, our shoppers submit experiments in batches, typically weekly.

And it turns on the market’s a great cause for this. Whereas a fluid workflow is sensible in a startup, it sometimes doesn’t within a large organization that has numerous stakeholders with totally different (and sometimes competing) aims and tasks. Product managers diligently seek the advice of these stakeholders when explaining customer suggestions and deciding on next steps. A predictable and recurring cadence is usually essential to hold everybody on the same web page.

That’s why concepts like the ‘design dash’ have taken off: they allot time for stakeholders to get aligned. We’re embracing the position that formality plays right here, and now encourage shoppers to arrange ‘experiment periods’ on a daily and constant basis, so long as those periods finish with testable hypotheses.

three. Product experiments may be grouped into discrete categories. Before we might create a platform and workflow to speed up consumer research processes, we had to better perceive the varieties of research product teams want in the first place. That’s why, earlier than writing a single line of code, we carried out the primary 500 or so experiments manually using third-party instruments.

We discovered that consumer analysis experiments involving prototypes (as opposed to doing experiments in a production surroundings) usually fall into one in every of six discrete categories. Considered one of them, usability testing, has a extensively accepted definition. We had to delineate the others though, and while our definitions are under no circumstances gospel, they suffice surprisingly properly, requiring solely modest ongoing revisions. Each category is accompanied by ‘guidelines of thumb’ and a set of configurable experiment templates, which you’ll be able to read about in ourguide to prototyping, however here is an summary of every:


Here’s a breakdown of the popularity of each check run on our platform:


We now have lots more research to do, but these working definitions allow consumer researchers in our trade to take nearly any shopper request and turn it into an executable research within minutes.

4. All analysis is biased. Our providing primarily consists of testing in what we name a ‘simulated surroundings.’ The users who present feedback know that they’re a part of a research and are paid for their time. They interact with high-fidelity, interactive prototypes, and usually understand that the products have not been engineered and launched to the market.

We focus on this sort of testing as a result of product teams can study an incredible amount from it whereas complying with their group’s present processes and danger tolerance. No inner engineering or design assets are required; no valued buyer turns into the victim of a half-baked product; and no legal division must be consulted. In fact, the info isn’t as reliable as what you’d study from delivery a product.

All research, together with ours, suffers from a level of bias. But acknowledging such isn’t an excuse to keep away from doing consumer analysis altogether. It’s an argument for the other: to fervently do much more analysis and attempt to attenuate the bias across it. Considering in any other case is lacking the forest for the timber.

One of the core rules of the scientific technique is the idea of replicability — that the results of any single experiment may be reproduced by one other experiment. We’ve far too typically seen a product workforce wielding a single ‘statistically vital’ knowledge level to defend a dubious instinct or pet venture. But there are a selection of things that would and virtually all the time do bias the outcomes of a check with none intentional wrongdoing. Mistakenly asking a number one query or sourcing a sample that doesn’t adequately characterize your goal buyer can skew particular person check outcomes.

To derive worth from individual experiments and buyer knowledge factors, product teams have to follow substantiation via iteration. Even when the outcomes of any given experiment are skewed or outdated, they are often offset by a strong consumer analysis process. The safeguard towards pursuing insignificant findings, if you’ll, is to be aware not to think about knowledge to be an actionable insight till a pattern has been rigorously established.


That’s why we make it possible for for nearly each experiment, qualitative and quantitative research is carried out. Further, we attempt to generate insights which might be comparative — it’s not often enough to study what users think of a prototype in a vacuum. In the actual world, users have an array of choices to fulfill any given need, so we be sure that feedback on an answer is all the time relative to an alternate. Combining and optimizing these two approaches has significantly minimized bias, and sometimes results in a plethora of knowledge from which to determine patterns and insights. And, in fact, we stress the importance of incorporating different knowledge inputs, like conventional market analysis and in-app analytics.

5. Consumer feedback never ceases to surprise us. You’d assume that after generating knowledge from lots of of hundreds of customers, we’d have ‘seen it all’ on the subject of feedback and insights. But that isn’t even close to true. We proceed to be stunned by what we see each day, primarily with regard to…

…the difference between what users say and what they do.

It’s been nicely established that humans are fairly dangerous at predicting their future conduct. We’ve researched the psychology of that dynamic extensively. However it’s nonetheless shocking once we find nearly unanimous help for a function in a survey and subsequently discover completely little interest in the function as soon as it’s prototyped. Placing a visual stimuli in entrance of your goal market is completely important for substantiating findings.

…the sincere emotions expressed.

Market tendencies change rapidly and product teams are in a continuing hustle to keep up. Few things get them to drop what they’re doing and sit silently as well as watching a video of an emotional consumer interview. We’ve witnessed a senior citizen cry profusely as they work together with a prototype that invokes nostalgia. We’ve giggled as a Millennial described how much they hated a product concept and all of the issues they’d relatively use as an alternative of it. We’ve been shocked by a gentleman who opened up about how a brand new product might help him rebuild relationships together with his youngsters. Consumer analysis is actually an emotional rollercoaster.

…the validation of passionate enthusiasm.

One of the crucial widespread questions our shoppers ask is: “How do we all know once we’ve validated a product idea with clients?” While we don’t have any hard-and-fast guidelines, we’ve half-joked about making use of the “Pokémon GO Benchmark.” For enjoyable (and since we’re addicted to the game), we executed research towards a couple of hundred users of the cellular recreation. The responses have been impressively enthusiastic and exemplified patterns to look for when assessing validation. Players gave detailed suggestions to open-ended questions, spent vital time partaking with prototypes, and routinely provided to pay for brand spanking new features we designed. Clearly, each product doesn’t must be a meteoric hit to seek out success, however evaluating outliers like Pokémon GO serves as a strong anecdote.

The important thing takeaway is that even once we assume we know a consumer phase rather well, research findings are not often predictable or obvious. You simply can’t underestimate how troublesome and rewarding having empathy may be.

6. Shorter iteration cycles unlock deeper insights. When our initial shoppers finally starting rapidly operating experiments on Alpha, it turned clear why producing significant customer insights is usually so elusive for corporations that take months to execute research. Velocity in and of itself is the important thing.

When iteration cycles are sluggish, product groups prototype and experiment until they generate promising results. The second they get the slightest sense that they’ve struck gold, they start engineering a solution (in the event that they haven’t already began). In essence, they study ‘what’ resonates properly, however they don’t have the time to study ‘why.’

But once we accelerated the analysis process to days, we found that shoppers have been not content material once they validated a product concept. They finally had the time and bandwidth to ask ‘why’ a prototype was perceived as extra invaluable that earlier iterations or options. To keep up, we had to construct out an in depth qualitative workflow so that we might return to a sample of users that tested a product and ask them open-ended questions. In doing so, we have been capable of unlock ‘deep insights.’

We define a deep perception as a comprehension of a buyer persona that’s so strong that its value transcends the individual challenge that a product staff is engaged on. It’s helpful to anybody within the organization who is concentrated on delivering value to the same market. As an alternative of merely understanding that clients favor your prototype with an costly one-time purchase compared to an affordable month-to-month subscription, you conduct interviews to study why and uncover that clients are actually afraid of forgetting to cancel their subscription. That’s an insight that’s so significant, it can be utilized to different products in your group’s portfolio. And it’s made attainable by velocity.

7. Knowledge is a way to an end. It’s straightforward to get misplaced in the buzzwords du jour fairly than to do the arduous work of discovering worth and driving ROI. We discovered shortly that to build a successful platform, we’d should deliver to product groups more than the power to be ‘knowledge pushed.’

Initially, our assumption was that knowledge that shoppers generated inside Alpha would translate immediately into higher product decision-making. That’s true to an extent and it definitely issues to the organization as an entire. However once we really investigated what was happening, we discovered that being knowledge pushed isn’t actually what product managers want or want.

We pay attention intently to how our shoppers communicate the worth of our platform and experimentation to peers at other organizations. Often, they mention how it aligns their workforce around hypotheses moderately than opinions. As an alternative of two hour meetings full of debates, the workforce spends 15 minutes placing hypotheses into Alpha and then 15 minutes reviewing the findings once they’re prepared. One product supervisor discussed how he makes use of Alpha just because the info provides him a purpose to e-mail his director an update as soon as every week. Another spoke about how thrilled he is to influence other departments to recognize the value of iteration and learning.

In fact, knowledge is important to enabling all of these advantages, nevertheless it’s a way moderately than an end. And that matters because it informs our product roadmap. For example, early on we didn’t put much effort into the info visualization of analysis findings. But now we perceive that presentation is simply as, if not more, essential than the underlying info, because it’s going to be shared and used to influence stakeholders. Recognizing how product managers must handle upward, sideways, and downward led us to prioritizing options like reporting and sharing.

We’ll proceed to replace this listing as we study more. When you’re as passionate as we are about experimentation and customer insights, be a part of our group. Or give Alpha a spin and start making smarter product selections.