What are some useful ideas and tools for schools working to accelerate student achievement in literacy (and/or other areas), particularly for Maori and Pasifika students? This presentation was delivered to school leaders, literacy leaders and their literacy facilitators (external PLD providers) participating in the Secondary Literacy Project (with overall design and national coordination from the University of Auckland’s Woolf Fisher Research Centre).
In it, I share some thinking about:
interpretation of progress data from e-asTTle, using …
a range of comparisons that can be made depending on the evaluation question one asks
multiple perspectives on effect sizes and their interpretation
the use of evaluative rubrics for interpreting accelerated student progress and achievement, and
for inquiring into key drivers of student progress such as effective teaching, educational leadership and school culture, case management and implementation, and many more
Click to download a PDF of the presentation (including several slides that were omitted from the actual session).
It was a reflection on part of the theme for the 2010 American Evaluation Association conference – Evaluation Quality – with a focus on Ernie House’s (1980) concepts of truth, beaty and justice in evaluation (click the link to access PDFs of the key chapters in House’s book, from the AEA e-library).
Although House wrote that truth trumps beauty and justice trumps them all, Jane argued for the critical importance of ‘beauty’ (the coherent, persuasive, compelling story or argument) – with the ‘values’ and the reasoning that guide conclusions about quality and value made clearly visible, transparent and understandable – as a hugely important path to both truth and justice.
On December 2nd 2010, Jane delivered a keynote address for the Self Assessment for Quality conference, hosted by the Otago Polytechnic in Dunedin, New Zealand.
The audience were representatives from tertiary (=higher) education organisations from throughout Aotearoa New Zealand working to implement a new evaluative approach to quality assurance, where they ask and answer questions about the quality of their offerings and services and the value of their outcomes for learners and other key stakeholders (such as employers, communities and iwi).
The theme of the conference was Self-Assessment for Quality: How do you know good when you see it?
If a reported “outcome†is not caused by a programme, it is not an outcome at all; it’s a coincidence. Simply measuring variables that may or may not be causally related to a programme (i.e. could just be coincidences – who knows?) doesn’t tell you anything about the quality or value of the programme, therefore it can’t be referred to as outcome evaluation – it’s just measurement.
Isn’t causal attribution heinously expensive, almost never feasible, and doesn’t it require some form of experimental design? Not necessarily. In this interactive seminar, Jane will use case examples to illustrate eight strategies for inferring (or ruling out) causal links between programmes and suspected outcomes: (1) Ask those who have observed or experienced the causal effect, (2) Check if the content of the intervention matches the nature of the outcome; (3) Look for distinctive effect patterns (modus operandi method), (4) Check whether the timing of outcomes makes sense, (5) Examine the relationship between program “dose†and “responseâ€, (6) Use a comparison or control, (7) Control statistically for extraneous variables, and (8) Identify and check the causal mechanisms. These strategies are outlined in Jane’s (2004) book, “Evaluation Methodology Basics: The nuts and bolts of sound evaluation†(Sage).
Actionable Evaluations: A Bootcamp for Commissioners, Managers and Evaluators
E. Jane Davidson & Nan Wehipeihana
Abstract:
Let’s face it. Every year there are evaluations that bitterly disappoint client organisations by failing to produce actionable answers to important evaluative questions. Weak evaluations frequently lack incisive evaluation questions; employ the wrong methodology for the questions; get lost in the details; skip over the crucial ‘values’ step and therefore can’t draw evaluative conclusions; uncritically accept stated objectives as the only evaluative criteria; focus only on the average effect; fail to adequately triangulate and transparently weave sources of evidence; toss causation into the ‘too hard basket’ while still claiming to have documented ‘outcomes’; and fail to clearly communicate findings. This session will help managers and commissioners of evaluations see the possibilities so they can become informed and demanding consumers of real, genuine, actionable evaluation. You will get important pointers for writing RFPs and managing selection processes with a better chance of attracting a shorter list of high quality proposals; and for effectively managing the evaluation so as to maximise the chances of getting clear, authentic, actionable answers to high priority questions. Evaluators will also be interested in hearing about how their work can better meet the needs of clients and other stakeholders.
In May 2010, Nan Wehipeihana and I did a presentation for the anzea Regional Symposium in Wellington. Here are the slides from the session (see link at the bottom), for those who may be interested.
Policy-relevant, strategic evaluation asks (and answers) questions that go beyond the evaluation of a single initiative. It addresses such questions as: (1) What is the value of a particular policy initiative as a contributor to strategic policy outcomes? (2) How well does each policy initiative fit with and complement the other initiatives that make up the strategic policy mix? Are there any unnecessary overlaps? (3) What is the collective value of the suite of initiatives that have been deployed to achieve a particular strategic outcome? (4) Have we got the right mix to deliver on the key outcomes? (5) Which approaches to achieving key long-term outcomes are working most cost-effectively for whom, under what conditions, and why? This session uses a current case (evaluation of Ka Hikitia, the Ministry of Education’s Māori education strategy, to outline some specific strategies to help ensure evaluation delivers high quality, truly useful information to calibrate macro-level policy – and to incisively inform Ministers on policy effectiveness.
The following is a list of selected publications, some of which are accessible online. In addition, many more recent short pieces are available on Jane Davidson and Patricia Rogers’ joint blog, http://GenuineEvaluation.com, as well as on this site.
Davidson, E. J., & Martineau, J. W. (2006). Strategic uses of evaluation. In J. W. Martineau, L. Merritt, & K. Hannum (Eds.), Leadership development evaluation handbook (pp. 433-463). San Francisco, CA: Jossey-Bass.
Davidson, E. J. (2003). Linking organizational learning to the bottom line: Methodological issues, challenges, and suggestions. The Psychologist-Manager Journal, 6(1), 54-67.
Rose, D. S., & Davidson, E. J. (2003). Introduction to program evaluation. In J. E. Edwards, J. C. Scott, & N. S. Raju (Eds.), The human resources program evaluation handbook (pp. 3-26). Sage.
Sanders, J. R., & Davidson, E. J. (2003). A model for school evaluation. In D. L. Stufflebeam & T. E. Kellaghan (Eds.), International handbook of educational evaluation (pp. 807-826). Dordrecht, Netherlands: Kluwer.
Davidson, E. J. (2003). The program theory evangelists vs. the “theoro-skepticsâ€: Who’s right, when and why—and where to next? Mechanisms. [Newsletter of the Program Theory and Theory-Driven Evaluation TIG, American Evaluation Association.]
Davidson, E. J. (2000). Ascertaining causality in theory-based evaluation. New Directions for Evaluation, 87, 17-26.
Sathe, V., & Davidson, E. J. (2000). Toward a new conceptualization of culture change. In N. M. Ashkanasy, C. P. M. Wilderom, & M. F. Peterson (Eds.), Handbook of organizational culture and climate (pp. 279-296). Thousand Oaks, CA: Sage.
The following are copies of selected past conference presentations in PDF format. All items are copyright; please cite the presentation and this website if they are referenced.
NOTE: If clicking on them directly doesn’t work, please right-click and choose “save as”.
