• Choose a category
• All
• Classic-Fiction

One of the difficult requirements of a before-after research design is that only one variable can change at the moment that divides the longitudinal case neatly in two. Campbell and Stanley emphasize that the values of the observed variables should not be examined only immediately before and after the event, but also well before and well after it. As David Collier writes: “Causal inferences about the impact of discrete events can be risky if one does not have an extended series of observations.”332 Campbell and Stanley suggested, and subsequent research has demonstrated, that when this type of quasi-experimental research design is imaginatively and carefully employed, it can be extremely useful in policy evaluation research.

The most common challenge for the before-after design is that for most phenomena of interest, more than one variable changes at a time. It is therefore important to do process-tracing not just on the main variables of interest that changed at a particular time, but also on the other potential causal variables that changed at the same time. This can help establish whether the variables of interest were causal and whether the other variables that changed in the same period were not, or at least that they do not account for all of the change in the outcome. Such process-tracing can focus on the standard list of potentially “confounding” variables identified by Campbell and Stanley, including the effects of history, maturation, testing, instrumentation, regression, selection, and mortality.333 It can also address whatever other idiosyncratic differences between the two cases might account for their differences.

Interesting examples of the before-after research design include Robert Putnam’s Making Democracy Work. Putnam argues that the Italian government’s reform of 1970 established regional governments for the first time in Italy, providing a type of natural experiment. Other socioeconomic and cultural variables could be held constant for the most part while the structure of political institutions was abruptly altered by the reform. This historical development gave Putnam an opportunity to evaluate the impact of this structural reform on the identities, power, and strategies of political actors.334

A more complex form of the before-after design or pathway analysis is employed by the Colliers in Shaping the Political Arena. They develop “critical junctures,” defined as periods of significant change, to serve as a common framework that is hypothesized as producing distinct regimes.

THE USE OF A COUNTERFACTUAL CASE OR MENTAL EXPERIMENT

Another way of attempting to achieve controlled comparison when two historical cases closely resembling each other cannot be located is to match the given case with an invented one that does.335 The case is, of course, a hypothetical one derived through counterfactual analysis of the existing case or, as it is sometimes referred to, the “mental experiment.” As James Fearon, Philip Tetlock, Aaron Belkin, and others have noted, resort to counterfactual analysis, either explicitly or implicitly, is a common practice in many types of research. Fearon asserts that “the common condition of too many variables and too few cases makes counterfactual thought experiments a necessary means for strong justification of causal claims.”336 The use of mental experiments in the service of theory development has a long and distinguished history, including Albert Einstein’s development of relativity theory.

However, counterfactual analysis, though frequently employed, has lacked strong criteria and standards for distinguishing good practice from the highly speculative and less disciplined uses of the method. Additional discussion of standards for counterfactual analysis appears in Chapter 10, but several criteria can be stated here. First, since a counterfactual case necessarily builds upon an existing case, it will be difficult to invent an acceptable one unless the investigator has already constructed