Persuasive Advertising - J. Scott Armstrong [8]
Non-experimental data analyses have proven useful for assessing advertising principles that involve only one or two minor conditions, especially when they have a strong effect, such as the use of a “brand-differentiating message.” However, analyses of non-experimental data suffer from the same problem that was noted in the studies of female hormone replacement drugs. Namely, the more informed advertisers are aware of expert advice and they may employ a set of recommended procedures, some of which are of no value, but, as they are used along with procedures that do matter, they appear to help. Based to some extent on my research for this book, I have become skeptical about the value of non-experimental data in complex situations.
Non-experimental data are often used by advertisers. For example, a study by Ogilvy and Raphaelson (1982) used responses from viewers of 951 TV commercials analyzed by the Mapes and Ross research firm. This approach took a giant step forward thanks to Stewart and Furse’s (1986) analysis of 1,059 TV commercials that had been tested by Research Systems Corporation (RSC). It used before and after viewer responses from many thousands of viewers to assess the effects of over 160 features of commercials on recall, comprehension, and persuasion. It examined such data for 356 brands from 63 firms in 12 product categories. It was a massive undertaking. Their study inspired further studies of tested TV commercials, including Stewart and Koslow (1989), with an additional 1,017 commercials from RSC; Laskey, Fox and Crask (1994), with an analysis of data on 1,100 30-second commercials for fast-moving food and household items; Stanton and Burke’s (1998) analysis of 601 commercials; Phillips and Stanton’s (2004) analysis of 5,000 commercials; and Walker’s (2008) analysis of 1,513 commercials that had been tested by Ipsos ASI.
At first glance, the findings from these studies seem disappointing. Few variables mattered. Why was that? The primary reason is that different approaches to advertising are used in different situations. Another reason is that there are an enormous number of conditions, and what works under some conditions may be harmful in others. As a result, what seem to be large sample sizes of over 1,000 commercials and a vast number of viewers were woefully inadequate. Massive sample sizes would be required so that the analyses could be done for ads facing similar conditions. In short, as a general strategy for learning how to persuade through advertising, the analysis of non-experimental data falls short. However, this was in itself an important finding. The analysis of non-experimental data has been widely used in advertising, and there are studies showing that even when done on a large scale by highly competent researchers, the findings can be misleading.
Another important conclusion is that for simple situations, the non-experimental findings to date have been useful. In this book, I have limited the discussion of non-experimental data to situations with few conditions. These findings are useful where experimental data are lacking. Even then, I advise caution. We conducted an analysis of 24 principles where non-experimental data seemed relevant and where we had experimental evidence. The findings from the non-experimental analyses agreed with those from the experimental data on 67 percent of the principles (Armstrong and Patnaik 2009). Better than chance (50 percent), for sure, but obviously calling for caution.
Non-experimental findings, when relevant, are provided at the end of each section on evidence. The reports are brief. Those who are interested in details can find them in the annotated references for the cited authors as well as in the original studies.
I believe that