Case Studies and Theory Development in the Social Sciences - Alexander L. George [58]
Problems in Evaluating Case Studies
Case writers should become familiar with the variety of critiques their work may face. The importance of understanding the history and context of a case makes the difficulties of critiquing qualitative research different from those of assessing quantitative work. Readers cannot easily judge the validity of the explanation of a case unless they possess a degree of independent knowledge of that case. This requires that reader-critics themselves possess some familiarity with the complexity of the case and the range of data available for studying it; knowledge of the existence of different interpretations offered by other scholars and of the status of the generalizations and theories employed by the case writer; and an ability to evaluate the case writer’s use of counterfactual analysis or to provide plausible counterfactual analysis of their own. These are tough requirements for readers who must evaluate case studies, and simply to state these desiderata suffices to indicate that they are not easily met. Our own commentaries of case study research designs in the Appendix, “Studies That Illustrate Research Design,” should be read with the caveat that we are not theoretical or historical experts on all the subjects of these studies. This is a problem also for those who review these books in academic journals.
Let us discuss some of the problems likely to be encountered by readers who attempt to evaluate case studies. Much of the preceding discussion is relevant to the task of evaluating case studies, and a few additional observations can be made.
The task of evaluating case studies differs depending on the research objective of the case. When the investigator’s research objective is to explain a case outcome, the reader-critic must consider whether the case analyst has “imposed” a favored theory as the explanation. Have alternative theories that might provide an explanation been overlooked or inadequately considered? When the case writer pursues the different research objective of attempting to use case findings to “test” an existing theory, there are several questions the reader-critic has to consider in deciding whether such a claim is justified. Does the case (or cases) constitute an easy or tough test of the theory? Do case findings really support the theory in question? Do they perhaps also support other theories the investigator has overlooked or inadequately considered?
Reader-critics must consider the possibility that the case-writer has overlooked or unduly minimized potentially important causal variables, or has not considered the possibility or likelihood that the phenomenon is subject to multiple conjunctural causation or is affected by equifinality.
These and other problems in using case studies to develop or test theories are also discussed in Chapter 6. They are referred to here in order to emphasize that case writers should be familiar with the variety of criticisms that can be and often are made of their work.
In addition, we urge that case writers accept the obligation to assist readers in evaluating whether their case analyses have met relevant methodological standards. To meet this requirement, case writers should go as far as reasonably possible to make the analyses they offer transparent enough to enable readers to evaluate them. Transparency of case studies must be closely linked with standards for case studies. These standards include (but are not limited to) providing enough detail to satisfy as much as possible the criteria of replicability and of the validity and reliability of the way in which variables are scored. Certainly these standards are often difficult to meet in case study research, but case writers can often do more to at least approximate them. We strongly concur with the admonition of Gary King, Robert Keohane, and Sidney Verba that “the most important rule for all