A / B Testing E-Book

Dan Siroker

• Herausgeber: John Wiley & Sons
• Kategorie: Fachliteratur
• Sprache: Englisch

Seitenzahl: 165

Veröffentlichungsjahr: 2013

Beschreibung

How Your Business Can Use the Science That Helped Win the White House The average conversion rate--the rate at which visitors convert into customers--across the web is only 2%. That means it's likely that 98% of visitors to your website won't end up converting into customers. What's the solution? A/B testing. A/B testing is the simple idea of showing several different versions of a web page to live traffic, and then measuring the effect each version has on visitors. Using A/B testing, companies can improve the effectiveness of their marketing and user experience and, in doing so, can sometimes double or triple their conversion rates. Testing has been fundamental in driving the success of Google, Amazon, Netflix, and other top tech companies. Even Barack Obama and Mitt Romney had dedicated teams A/B testing their campaign websites during the 2012 Presidential race. In the past, marketing teams were unable to unleash the power of A/B testing because it required costly engineering and IT resources. Today, a new generation of technology that enables marketers to run A/B tests without depending on engineers is emerging and quickly becoming one of the most powerful tools for making data-driven decisions. Authors Dan Siroker and Pete Koomen are cofounders of Optimizely, the leading A/B testing platform used by more than 5,000 organizations across the world. A/B Testing: The Most Powerful Way to Turn Clicks Into Customers offers best practices and lessons learned from more than 300,000 experiments run by Optimizely customers. You'll learn: * What to test * How to choose the testing solution that's right for your organization * How to assemble an A/B testing dream team * How to create personalized experiences for every visitor * And much more Marketers and web professionals will become obsolete if they don't embrace a data-driven approach to decision making. This book shows you how, no matter your technical expertise.

Leseprobe

Contents

Cover

Title Page

Copyright

Chapter 1: How A/B Testing Helped Win the White House—Twice

Optimization for Everyone

Part I: Lessons Learned from 200,000 A/B Tests (and Counting)

Chapter 2: What to Test

Step One: Define Success

Step Two: Identify Bottlenecks

Step Three: Construct a Hypothesis

Step Four: Prioritize

Step Five: Test

Chapter 3: Seek the Global Maximum

Break from the Status Quo: ABC Family

Learn Your Way to the New Site: Chrome Industries

Rethink the Business Model: Lumosity

Test Through the Redesign, Not After: Digg and Netflix

Chapter 4: Less Is More: Reduce Choices

Every Field Counts: The Clinton Bush Haiti Fund

Keep It Simple: SeeClickFix

Hide Options: Cost Plus World Market

Remove Distractions and “Outs”: Avalanche Technology Group

Lower the Slope: Obama 2012

Chapter 5: Words Matter: Focus on Your Call to Action

Reconsider “Submit”: The Clinton Bush Haiti Fund

Find the Perfect Appeal: Wikipedia

Why versus How : Formstack

Nouns versus Verbs: LiveChat

Framing Effects

Chapter 6: Fail Fast and Learn

Prime Real Estate versus Familiar Real Estate: IGN

What's Good at One Scale Isn't Always Good at Another: E-Commerce

What Buyers Want Isn't Always What Sellers Want: Etsy

When a Win Isn't a Win (Is a Win): Chrome Industries

Part II: Implementing A/B Testing: Play-by-Play Guide

Chapter 7: Choose the Solution That's Right for Your Organization

Option One: Build

Option Two: Buy

Option Three: Hire

The Choice Is Yours (and Your Team's)

Chapter 8: The Cure for the HiPPO Syndrome

The HiPPO Syndrome Has a Cure

Winning Over the Stakeholders

Communicate Findings and Value to Your Team

Evangelize and Galvanize: Lizzie Allen at IGN

Chapter 9: The A/B Testing Dream Team

The Centralized Team

The Decentralized Team

The Three Key Ingredients of a Scalable Testing Strategy

Chapter 10: Iterate, Iterate, Iterate

Multivariate Testing and Iterative Testing

There Are No Universal Truths: Always Be Testing

Redesigning Your Redesign: CareerBuilder and the Optimizely Website

Part III: Advanced Topics in A/B Testing

Chapter 11: How A/B Tests Can Go Awry

Testing without Traffic

The Beginning of the Funnel versus the End: UserVoice

Off-Brand Testing: Dell

Chapter 12: Beyond the Page: Non-Website A/B Testing

The What and the When : Prezi

Price Testing

Testing the Perceived Price: The Last-Minute Discount

Anchoring in Action: Judy's Book Club

Testing the Billing

Testing the Actual Price

Chapter 13: Personalize, Personalize, Personalize

Targeting versus Segmentation

Using Segmentation to Drill Down into Test Results

Geo-Targeting, State by State: Romney 2012

When to Personalize and When Not to: Wikipedia

Conclusion

Appendix 1: 60 Things to A/B Test

Appendix 2: Metrics and the Statistics behind A/B Testing

Acknowledgments

Index

Cover design: Ryan Myers

Copyright © 2013 by Dan Siroker and Pete Koomen. All rights reserved.

Published by John Wiley & Sons, Inc., Hoboken, New Jersey.

Published simultaneously in Canada.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, (978) 750–8400, fax [phone], or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748–6011, fax [phone], or online at www.wiley.com/go/permissions.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with the respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor the author shall be liable for damages arising herefrom.

For general information about our other products and services, please contact our Customer Care Department within the United States at (800) 762–2974, outside the United States at (317) 572–3993 or fax [phone].

Wiley publishes in a variety of print and electronic formats and by print-on-demand. Some material included with standard print versions of this book may not be included in e-books or in print-on-demand. If this book refers to media such as a CD or DVD that is not included in the version you purchased, you may download this material at http://booksupport.wiley.com. For more information about Wiley products, visit www.wiley.com.

Library of Congress Cataloging-in-Publication Data:

Siroker, Dan.

A/B testing : the most powerful way to turn clicks into customers / Dan Siroker, Pete Koomen.

pages cm

Includes index.

ISBN 978-1-118-53609-4 (cloth); 978-1-118-65917-5 (ebk); ISBN 978-1-118-65920-5 (ebk)

1. Organizational effectiveness. 2. Multimedia systems–Social aspects. 3. Application software–Testing. I. Koomen, Pete, 1982- II. Title.

HD58.9.S5447 2013

658.8′3402854678–dc23

2013016038

Chapter 1

How A/B Testing Helped Win the White House—Twice

The $57 Million Button

It was 2007 when then-Senator Barack Obama was running for President, and no one but the Des Moines Register seemed to think he had a chance of winning the Democratic primary.

DAN: I was a product manager at Google at the time, and I'd seen Obama speak at our headquarters several weeks prior to the primary election. “I am a big believer in reason and facts and evidence and science and feedback—everything that allows you to do what you do. That's what we should be doing in our government,” Obama told the packed auditorium. “I think that many of you can help me, so I want you to be involved.” He probably meant that he wanted donations, or maybe votes, but I took him literally. I took a leave of absence from Google initially and eventually quit my job to move from California to Chicago to join the campaign.

I joined what was being called the “new media” team. They used the phrase “new media” because it encompassed everything that didn't typically fit into traditional political campaigns: email, social media, blogging, SMS, and the web. The team had competent bloggers, designers, and email copywriters; I wondered where I might be able to make an impact.

One thing stood out to me: a red button.

Online donations to the campaign came from subscribers to the email newsletter; subscriptions for this came from the campaign website's signup form; and the signup form came as a result of clicking a red button that said “Sign Up.” This was the gateway through which all of Obama's email supporters had to pass; it all came down to one button. So, one simple, humble question immediately became pivotal.

Is This the Right Button?

Is this our best chance to get every single supporter, and every single dollar, that we possibly can?

I had zero political experience at the time, and little clout within the organization. I didn't have a politico's intuition about what the button or the image above it should look like—nor the persuasive rhetoric required to run any proposed improvements up the chain of command. All I had was one insistent question: Is this button the absolute best? —and the desire to find the answer. There was only one way to know for certain.

Knowing little about politics or why certain words and images might be more moving or more effective than others, I suggested experimenting to figure out what worked to drive the most signups. Our team tested four different labels for the button (“Sign Up,”“Sign Up Now,”“Join Us Now,” and “Learn More”) and six different media (images and videos) above it to see which combination induced the most visitors to engage and sign up.

Our team took bets on which variation (Figures 1.1 through 1.3) would perform best at garnering email signups. Most folks put their money on “Sam's Video,” a compilation of some of the most powerful moments in Obama's speeches. We assumed any video—with not just Obama's image, but his voice and message—would lead more people to enter their email addresses than a simple static image would.

Figure 1.1 The original splash page we set out to optimize at the Obama campaign in 2008.

Source: Optimizely.

Figure 1.2 The button variations we tested.

Source: Optimizely.

Figure 1.3 The media variations we tested.

Source: Optimizely.

Boy, were we wrong.

In fact, not only “Sam's Video” but every video dramatically underperformed every image. Even more dramatically, one image-and-button combination in particular stood head and shoulders above the original (Figure 1.4).

Figure 1.4 A side-by-side comparison of the original and winning variation of the splash page at the 2008 Obama campaign.

Source: Optimizely.

A combination of the “Family Image” and the “Learn More” button improved the signup rate by a staggering 40.6 percent. Over the course of the campaign, that 40.6 percent lift in signups translated to 2.8 million more email subscribers, 288,000 more volunteers, and—perhaps most important of all—an additional $57 million in donations.

Obama went on, with an enormous lead in dollars and supporters raised online, to win the election. He was buoyed by a team willing to test everything and to listen to the data even when it surprised them the most.

A small, simple question about a small red button had been answered conclusively with a straightforward experiment. But in its place loomed another question, just as simple and just as insistent:

“Why aren't more people doing this?”

The Age of Testing

The answer, in short, was that the commercially available tools at the time required heavy involvement from software engineers to run experiments. For all of the spectacular gains that website testing enabled at the Obama campaign—as well as at big-tech players like Google and Amazon—it was still a highly technical practice. It was simply out of reach for most businesses that didn't have the know-how and a dedicated in-house team, and prohibitively difficult even for many that did. But why did this have to remain the case? Why couldn't every organization have access to these tools?

I joined up with my fellow Google product manager, Pete Koomen, and in 2010 the two of us struck off on our own to help do just that. What we built as a result is Optimizely, a website optimization platform that makes it easy for any organization, from a one-person startup to a Fortune 100 firm, to do what the Obama team did on the road to the White House—with no degrees in statistics or dedicated engineering team required.

Over the past several years, a range of new tools has emerged to make this online testing and optimization practice—A/B testing, as it is known—easier and faster. The concept of A/B testing is simple: show different variations of your website to different people and measure which variation is the most effective at turning them into customers. If each visitor to your website is randomly shown one of these variations and you do this over the same period of time, then you've created a controlled experiment known as an A/B test. A/B testing has gone from a secret weapon within the purview of only a handful of tech companies to an increasingly ubiquitous and critical part of doing business online.

This sea change in the way companies are conducting online business and marketing is perhaps best illustrated by taking a glance at the election cycle that came next: the 2012 presidential race. There were some key differences between 2012 and 2008: the Obama campaign team had an intense testing program in place from day one and didn't need to be persuaded as to the mission-critical value of A/B testing. The other key difference: so did the opponent, Mitt Romney.

Leading publications from TIME to The Atlantic to Businessweek to Forbes wrote about the 2012 presidential campaign fundraising machines as being the most sophisticated, data-driven, and efficient organizations that politics had ever seen. And at the heart of this new reality was A/B testing.

Optimization for Everyone

Whether or not you have plans to run for office in the near future, whether you come from a huge organization or a team of one, and whether your background is in computer science or marketing, you can make these same changes happen in your own workplace.

In this book we'll tell the story of a number of different companies—what they tested, what they learned, and how testing has grown to become part of their mindset, values, and culture. We at Optimizely have been fortunate to witness this transformation take place time and again. Part of what we want to change is the notion that testing is something that only IT staff or developers can do, because nothing could be further from the truth. Anyone can conduct A/B testing, and it can affect—and infect—an entire organization.

This book is for anyone interested in building a culture of testing within his or her organization by being curious enough, brave enough, and humble enough to ask simple questions like “Is this red button the best?” and to experiment to find the answer. Through our experiences building Optimizely and through extensive contact with our own customers, we've come to understand very well the challenges that this group faces—a group that includes the following members at its core:

Digital marketers and marketing managersDesignersProduct managersSoftware engineersEntrepreneursCopywritersGrowth hackersData scientists

No matter your role or experience level, your seniority or budget, you'll come away from this book with a host of new ideas, as well as a seasoned understanding of the challenges—and advantages—involved in using data to drive your organization's growth and revenue. To help you get going, we'll start with some of the most common questions:

“How should we choose what to test?”

“How can my team and I adopt a sustainable testing process?”

“Where do we begin?”

This book takes you through dozens of real-world examples that show specifically how teams like yours have embraced A/B testing as a fundamental part of not only their day-to-day workflow, but their creative process as well. We examine dramatic successes, subtle differences, and surprising failures, and we reveal a wealth of ideas and actionable insights by looking through the eyes of the people and teams who discovered them.

Each case study serves to highlight different best (and worst) practices, and we've grouped our studies broadly based on common underlying themes. You can read them one after the other, or pick and choose targeted examples based on what you're looking for. At the end of each chapter, we've included a brisk “TL;DR” (too long; didn't read) condensation of the chapter's main points and key takeaways.

The tools are out there, and they're not hard to get or hard to use. All you need to do is start asking questions. Who knows where they might lead you?

Part I

Lessons Learned from 200,000 A/B Tests (and Counting)

Having worked with thousands of different organizations on hundreds of thousands of different A/B tests, we've been able to distill some of the key principles and insights that any tester or testing team should have in mind as they make their first forays into testing. Part I addresses some of the typical questions that you might have about jumping in, starting with perhaps the biggest: “Where do I begin?”

And that's exactly where we pick up next.

Chapter 2

What to Test

Optimization in Five Steps

The hardest part of A/B testing is determining what to test in the first place. Having worked with thousands of customers who do A/B testing every day, one of the most common questions we hear is, “Where do I begin?”

A mistake that some companies make is to start moving a bunch of levers around without clear planning upfront for what they're trying to optimize—and what will be impacted by those changes. It's tempting to just dive in and start changing parts of your homepage, or your product page, or your checkout page, without truly understanding the value that it's generating (or not generating) for your business.

Instead, we advise a purposeful and deliberate five-step process:

Step One: Define success

Step Two: Identify bottlenecks

Step Three: Construct a hypothesis

Step Four: Prioritize

Step Five: Test

This process begins with the most important question of all: What is the purpose of your site?

Step One: Define Success

Before you can determine which of your test's variations is the winner, you have to first decide how you're keeping score. To start A/B testing successfully, you need to answer a specific question: What is your website for? If you could make your website do one thing better, what would it do?

If the answer to that question isn't completely clear to you, there's a trick that might help. Imagine the following dialogue:

ALICE:“What do you want to achieve with A/B testing?”

BOB:“We don't know. We don't know what we want our website to do.”

ALICE:“Why don't you take it down?”

BOB:“Of course not! We need our website because it—”

And then Bob has the aha! moment that crystallizes his website's raison d'être : He can see reasons for the website deeper than “Everyone else has one, so we need one, too.”

Defining success in the context of A/B testing involves taking the answer to the question of your site's ultimate purpose and turning it into something more precise: quantifiable success metrics. Your success metrics are the specific numbers you hope will be improved by your tests.

It's fairly easy for an e-commerce business to define its success metrics in terms of revenue per visitor (though there are complexities and “gotchas” we'll discuss later), and for a fundraising website to define its success metrics in terms of average donation per visitor. Depending on your business model, defining your success metrics may be trickier.

For instance, Google's search engineers measure what they call abandonment, which is when a user leaves a search results page without clicking anything. Abandonment can be bad—perhaps none of the results looked helpful—but it can also be good—perhaps the results page itself was so informative that there was no need to click through to any of the pages.

Figure 2.1 lists some of the most common success metrics for particular site types. Here we've broken websites down into four broad categories.

Figure 2.1 Table of conversion goals by site type.

Part of building out your testing strategy is identifying what constitutes—and does not constitute—a “conversion” for your particular site. In online terms, a conversion is the point at which a visitor takes the desired action on your website. Pinpointing the specific actions you want people to take most on your site and that are most critical to your business will lead you to the tests that have an impact.

Macroconversions, Microconversions, and Vanity Metrics

Author and digital marketing evangelist Avinash Kaushik makes the distinction between what he calls macroconversions —the metric most closely aligned with your site's primary raison d'être, as we discussed earlier—and microconversions —the other actions that users take on your site. While microconversions (things like clicking one button in a signup funnel, watching a video, or commenting on a blog post) may not be as immediately valuable or monetizeable as macroconversions, they can provide a tremendous amount of indirect value (provided they're not optimized at the expense of macroconversions).

A quick word of caution: sometimes a business can be lured into chasing “vanity metrics” that end up being distractions from the actual goal.