Beware the Black Box: A/B Testing, Randomized Controlled Trials, and Experimentation

Beware the Black Box: A/B Testing, Randomized Controlled Trials, and Experimentation

More and more companies are relying on online and offline experiments, in the form of A/B tests and randomized controlled trials, to assess the effectiveness of different treatments (e.g., design choices, marketing promotions, product descriptions). However, the ability to draw causal inferences from these tests hinges on a number of key assumptions that may not always be met. This is particularly true for online platform studies (e.g., Facebook A/B tests, Google AdWords) whose underlying algorithms—often a “black box” to experimental researchers—can result in skewed or “divergent delivery.”  

In this talk, Dr. Isaac will discuss the importance of properly determining whether studies can provide causal versus correlational insights, particularly given the unique value that experiments offer as part of the marketing research “toolbox.” He will also share best practices in experimental research and offer guidance on how to position, design, conduct, and report offline and online studies. The discussion should benefit seasoned researchers who are experienced in running and analyzing experiments, as well as practitioners who are less familiar with this methodology.

Professor of Marketing, Chair of the Marketing Department

Mathew Isaac obtained his PhD in Marketing from the Kellogg School of Management at Northwestern University. At the Albers School of Business and Economics at Seattle University, Dr. Isaac teaches Introduction to Marketing, Sales Management, Brand Management, and Marketing Strategy to undergraduates and MBA students. His research focuses primarily on consumer judgment and decision-making, examining how contextual and motivational factors influence product evaluations and purchase intentions. His work has been published in leading scientific journals and featured in business and popular press, including Harvard Business Review, Time, Forbes, Wall Street Journal, The Atlantic, and Fast Company. Prior to entering academia, Dr. Isaac advised media, healthcare, and private equity clients as a Consultant and Manager for Bain & Company and ZS Associates.