causation « Real Evaluation

Outcomes, impacts & causal attribution – handouts from Auckland anzea symposium

Here’s a link to the handout for the anzea symposium session today entitled:

Outcomes, impacts & causal attribution: Affordable, practical, feasible options for the real world

Abstract:

If a reported â€œoutcomeâ€ is not caused by a programme, it is not an outcome at all; it’s a coincidence. Simply measuring variables that may or may not be causally related to a programme (i.e. could just be coincidences – who knows?) doesn’t tell you anything about the quality or value of the programme, therefore it can’t be referred to as outcome evaluation – it’s just measurement.

Isn’t causal attribution heinously expensive, almost never feasible, and doesn’t it require some form of experimental design? Not necessarily. In this interactive seminar, Jane will use case examples to illustrate eight strategies for inferring (or ruling out) causal links between programmes and suspected outcomes: (1) Ask those who have observed or experienced the causal effect, (2) Check if the content of the intervention matches the nature of the outcome; (3) Look for distinctive effect patterns (modus operandi method), (4) Check whether the timing of outcomes makes sense, (5) Examine the relationship between program â€œdoseâ€ and â€œresponseâ€, (6) Use a comparison or control, (7) Control statistically for extraneous variables, and (8) Identify and check the causal mechanisms. These strategies are outlined in Jane’s (2004) book, â€œEvaluation Methodology Basics: The nuts and bolts of sound evaluationâ€ (Sage).

Past Publications (pre-2010)

The following is a list of selected publications, some of which are accessible online. In addition, many more recent short pieces are available on Jane Davidson and Patricia Rogers’ joint blog, http://GenuineEvaluation.com, as well as on this site.

Davidson, E. J. (2010, in press). Strategic evaluation of the workplace assessment program. In J. C. Scott & D. H. Reynolds (Eds.), The Handbook of Workplace Assessment: Evidence-based practices for selecting and developing organizational talent. SIOP Professional Practice Series. Jossey-Bass/Pfeiffer.

Davidson, E. J. (2010). “Process values” and “deep values” in evaluation, Journal of MultiDisciplinary Evaluation, vol. 16(13), 182-192.

Wehipeihana, N., Davidson, E. J., McKegg, K., & Shanker, V. (2010). What does it take to do evaluation in communities and cultural contexts other than our own? Journal of MultiDisciplinary Evaluation, vol. 16(13), 182-192.

Davidson, E. J. (2007). Unlearning some of our social scientist habits. Journal of Multidisciplinary Evaluation, 4(8), iii-vi.

Davidson, E. J., & Martineau, J. W. (2006). Strategic uses of evaluation. In J. W. Martineau, L. Merritt, & K. Hannum (Eds.), Leadership development evaluation handbook (pp. 433-463). San Francisco, CA: Jossey-Bass.

Davidson, E. J. (2006). The RCTs-only doctrine: Brakes on the acquisition of knowledge? Journal of Multidisciplinary Evaluation, 3(6), ii-v.

Davidson, E. J. (2006). The â€œbaggagingâ€ of theory-based evaluation. Journal of Multidisciplinary Evaluation, 3(4), iii-xiii.

Davidson, E. J. (2006). Some brief thoughts on organisational complexity and evaluation. Evalueringsnyt, 11, 22-26.

Davidson, E. J., Howe, M., & Scriven, M. (2004). Evaluative thinking for grantees. In M. T. Braverman, J. K. Slater, & N. A. Constantine (Eds.), Foundations and evaluation: Contexts and practices for effective philanthropy (pp. 259-280). Jossey-Bass.

Davidson, E. J. (2003). Linking organizational learning to the bottom line: Methodological issues, challenges, and suggestions. The Psychologist-Manager Journal, 6(1), 54-67.

Rose, D. S., & Davidson, E. J. (2003). Introduction to program evaluation. In J. E. Edwards, J. C. Scott, & N. S. Raju (Eds.), The human resources program evaluation handbook (pp. 3-26). Sage.

Sanders, J. R., & Davidson, E. J. (2003). A model for school evaluation. In D. L. Stufflebeam & T. E. Kellaghan (Eds.), International handbook of educational evaluation (pp. 807-826). Dordrecht, Netherlands: Kluwer.

Davidson, E. J. (2003). The program theory evangelists vs. the â€œtheoro-skepticsâ€: Who’s right, when and whyâ€”and where to next? Mechanisms. [Newsletter of the Program Theory and Theory-Driven Evaluation TIG, American Evaluation Association.]

Davidson, E. J. (2002). Organizational evaluation: Issues and methods. In R. L. Lowman (Ed.), Handbook of organizational consulting psychology (pp. 344-369). San Francisco, CA: Jossey-Bass.

Davidson, E. J. (2002). The discipline of evaluation: A helicopter tour for I/O psychologists. The Industrial-Organizational Psychologist, 40(2), 31-35.

Davidson, E. J. (2000). Ascertaining causality in theory-based evaluation. New Directions for Evaluation, 87, 17-26.

Sathe, V., & Davidson, E. J. (2000). Toward a new conceptualization of culture change. In N. M. Ashkanasy, C. P. M. Wilderom, & M. F. Peterson (Eds.), Handbook of organizational culture and climate (pp. 279-296). Thousand Oaks, CA: Sage.