As part of this week’s National Professional Learning and Development Facilitator Training (held in Auckland and sponsored by the Ministry of Education), Nan Wehipeihana and I presented a workshop for PLD providers:
This is an introduction to some of the rubrics developed for the Measurable Gains Framework, designed to evaluate the effectiveness of Ka Hikitia, the Ministry’s strategy for Maori learners.
Roger Maaka, an academic who has studied and advised on indigenous issues internationally.
Donna Mertens is past president of the American Evaluation Association and editor of the Journal of Mixed Methods Research, as well as a professor at Gallaudet University in Washington, DC.
Thomas Schwandt is a professor at the University of Illinois at Urbana-Champaign and a member of a standing committee of the National Research Council of the US National Academies of Sciences.
Elliot Stern is Professor of Evaluation Research at Lancaster University and the current editor of Evaluation: the international journal of theory, research and practice.
Jane Davidson, formerly the Associate Director of the Evaluation Center at Western Michigan University, Jane now heads an evaluation consulting firm in Auckland, New Zealand.
Policy-relevant, strategic evaluation asks (and answers) questions that go beyond the evaluation of a single initiative. It addresses such questions as: (1) What is the value of a particular policy initiative as a contributor to strategic policy outcomes? (2) How well does each policy initiative fit with and complement the other initiatives that make up the strategic policy mix? Are there any unnecessary overlaps? (3) What is the collective value of the suite of initiatives that have been deployed to achieve a particular strategic outcome? (4) Have we got the right mix to deliver on the key outcomes? (5) Which approaches to achieving key long-term outcomes are working most cost-effectively for whom, under what conditions, and why? This session will outline five key elements needed to position and effectively engage with the evaluation function so that it delivers high quality, truly useful information to calibrate macro-level policy – and to incisively inform Ministers on policy effectiveness.
Want us to keep you posted?
Please subscribe to the site feed to receive updates about workshops, presentations, publications and keynotes – plus Real Evaluation ideas and techniques. See top right corner for subscription options (email updates highly recommended).
Let’s face it. Every year there are evaluations that bitterly disappoint client organisations by failing to produce actionable answers to important evaluative questions. Weak evaluations frequently lack incisive evaluation questions; employ the wrong methodology for the questions; get lost in the details; skip over the crucial ‘values’ step and therefore can’t draw evaluative conclusions; uncritically accept stated objectives as the only evaluative criteria; focus only on the average effect; fail to adequately triangulate and transparently weave sources of evidence; toss causation into the ‘too hard basket’ while still claiming to have documented ‘outcomes’; and fail to clearly communicate findings. This session will help managers and commissioners of evaluations see the possibilities so they can become informed and demanding consumers of real, genuine, actionable evaluation. You will get important pointers for writing RFPs and managing selection processes with a better chance of attracting a shorter list of high quality proposals; and for effectively managing the evaluation so as to maximise the chances of getting clear, authentic, actionable answers to high priority questions. Evaluators will also be interested in hearing about how their work can better meet the needs of clients and other stakeholders.
Want us to keep you posted?
Please subscribe to the site feed to receive updates about workshops, presentations, publications and keynotes – plus Real Evaluation ideas and techniques. See top right corner for subscription options (email updates highly recommended).
If a reported “outcome†is not caused by a programme, it is not an outcome at all; it’s a coincidence. Simply measuring variables that may or may not be causally related to a programme (i.e. could just be coincidences – who knows?) doesn’t tell you anything about the quality or value of the programme, therefore it can’t be referred to as outcome evaluation – it’s just measurement.
Isn’t causal attribution heinously expensive, almost never feasible, and doesn’t it require some form of experimental design? Not necessarily. In this interactive seminar, Jane will use case examples to illustrate eight strategies for inferring (or ruling out) causal links between programmes and suspected outcomes: (1) Ask those who have observed or experienced the causal effect, (2) Check if the content of the intervention matches the nature of the outcome; (3) Look for distinctive effect patterns (modus operandi method), (4) Check whether the timing of outcomes makes sense, (5) Examine the relationship between program “dose†and “responseâ€, (6) Use a comparison or control, (7) Control statistically for extraneous variables, and (8) Identify and check the causal mechanisms. These strategies are outlined in Jane’s (2004) book, “Evaluation Methodology Basics: The nuts and bolts of sound evaluation†(Sage).
Want us to keep you posted?
Please subscribe to the site feed to receive updates about workshops, presentations, publications and keynotes – plus Real Evaluation ideas and techniques. See top right corner for subscription options (email updates highly recommended).
