The fruit fly of organizational decision-making.
It's advertising
I am teaching a new PhD level course this semester, and designing this course has prompted me to ponder what business school research is worth teaching. Although the scope of my class is quite broad, including pricing, AI, and digital platforms, I’ve decided to include two weeks on advertising.
Why advertising? A goal of business school research is to devise ways in which we can evaluate and improve decision-making. It’s hard to know whether a decision was good unless we know its effects. The best technology we have for measuring causal effects is the randomized control trial (RCT), but most decisions are not evaluated via RCT. Advertising is an exception. For a variety of reasons, it is the decision that is most frequently experimented with by firms.1 As a result, advertising is especially suited to measuring whether and when organizational decisions are good.
What have we learned about organizations from rigorous advertising research?
1. Firms frequently make large and preventable mistakes.
Economists often assume that firms behave optimally. Verifying optimality is quite difficult, but advertising experiments can help. Suppose that a firm turns off all advertising and almost nothing happens to sales and other outcomes. Well, in that case, we would say that a firm was making a mistake by advertising. This is indeed what numerous research papers have found about large advertising campaigns by major e-commerce companies2 and consumer brands.3 The amount of money wasted by some of these firms easily escalated into the hundreds of million of dollars. At the same time, not all firms over-advertise. In fact, some firms are able to achieve positive ROI.
2. One reason for continued mistakes is that managers have incentives to preserve budget, even if that budget is being wasted.
Marketing managers often have large budgets, which rationalize their own pay as well as the pay of their reports. Running an experiment that could potentially show that marketing has little, or no effect is therefore a threat. As a result, there is often resistance to large-scale experiments. Even when advertising experiments show highly negative ROI, advertising often continues.
3. No matter how rigorous a quantitative analysis is, there are always some blind spots. People will use these potential things to rationalize not making rigorous decisions.
Suppose a company runs an experiment where a treatment group sees an ad and a control group does not. Suppose also that the difference in sales between these two groups is small, and does not justify the ad spending. Namely, the return on investment of the advertising is negative. Marketing professionals will oftentimes resist this result by pointing out hypotheticals such as people in the treated group telling their friends (word of mouth) or brand awareness that can increase purchases in some far away future.
4. Actions have different effects depending on context. Heterogeneity abounds.
A striking feature of ads is that most ads don’t work well but some do. We know relatively little about how to predict which ads do and don’t work. Below, I show some plots demonstrating ad heterogeneity from recent papers:
Notice the wide range of effects across many contexts, including even political ads. Not only do effects vary, but the standard errors on these effect sizes are large!
5. For many decisions, the amount of data needed to be confident in a decision is prohibitive. We are stuck with educated guesses.
Given that ads have highly variable effects, can we actually measure these variable effects well enough to make the right decision? To make the right decision, we need to know whether there is a positive return on investment, accounting for the costs of advertising. In a highly influential paper, Lewis and Rao (2015) showed that the amount of data required to demonstrate that advertising is positive ROI is substantially greater than the amount of data required to merge show that it increases sales. In particular, many millions of observation are often required to precisely conclude that ROI is positive.
Similar calculations are likely to hold for many other decisions that firms make. For example, an A/B test between two ranking algorithms may fail to reject the null hypothesis that the two arms are different, even if the true effect size is large and important to the company and its customers.
To summarize, advertising research teaches us that making rigorous decisions is very difficult, both for statistical and organizational reasons. Given this, it must be the case that many organizations in the world are far from the optimal. While this is unfortunate, it means that the world can become a much better place than it currently is through better decisions.
Other decisions such as pricing and product design are less amenable to experimentation. Pricing is hard to experiment with due to fairness concerns, while product design is something one needs to do prior to testing the product on the marketplace. Advertising on the other hand is something that can be continuously and cheaply varied, making it more amenable to return on investment analysis.
Blake, Thomas, Chris Nosko, and Steven Tadelis. 2015. “Consumer Heterogeneity and Paid Search Effectiveness: A Large-Scaled Field Experiment.” Econometrica, 83 (1): 155- 174.
Shapiro, Bradley T., Günter J. Hitsch, and Anna E. Tuchman. "TV advertising effectiveness and profitability: Generalizable results from 288 brands." Econometrica 89.4 (2021): 1855-1879.