It was a reflection on part of the theme for the 2010 American Evaluation Association conference – Evaluation Quality – with a focus on Ernie House’s (1980) concepts of truth, beaty and justice in evaluation (click the link to access PDFs of the key chapters in House’s book, from the AEA e-library).
Although House wrote that truth trumps beauty and justice trumps them all, Jane argued for the critical importance of ‘beauty’ (the coherent, persuasive, compelling story or argument) – with the ‘values’ and the reasoning that guide conclusions about quality and value made clearly visible, transparent and understandable – as a hugely important path to both truth and justice.
In May 2010, Nan Wehipeihana and I did a presentation for the anzea Regional Symposium in Wellington. Here are the slides from the session (see link at the bottom), for those who may be interested.
Policy-relevant, strategic evaluation asks (and answers) questions that go beyond the evaluation of a single initiative. It addresses such questions as: (1) What is the value of a particular policy initiative as a contributor to strategic policy outcomes? (2) How well does each policy initiative fit with and complement the other initiatives that make up the strategic policy mix? Are there any unnecessary overlaps? (3) What is the collective value of the suite of initiatives that have been deployed to achieve a particular strategic outcome? (4) Have we got the right mix to deliver on the key outcomes? (5) Which approaches to achieving key long-term outcomes are working most cost-effectively for whom, under what conditions, and why? This session uses a current case (evaluation of Ka Hikitia, the Ministry of Education’s Māori education strategy, to outline some specific strategies to help ensure evaluation delivers high quality, truly useful information to calibrate macro-level policy – and to incisively inform Ministers on policy effectiveness.
Some key things to note about the competencies are:
This is not a checklist of competencies that every evaluator should have ALL of in order to be ‘competent’; this is a list of competencies that all evaluators will have some of, but no one person will have all of.
Each evaluator and each evaluation consulting business needs to consider what strengths profile it has (and is aiming to develop). In other words, how do you position yourself as distinct from other evaluators or evaluation consulting firms in this country (or, internationally, if that’s the space you compete in)?
Each evaluation team (and each manager commissioning an evaluation) needs to think through what profile of competencies is required for a particular project and whether their (or, a proposing) evaluation team collectively has those strengths.
Although this list will be of some interest internationally, there are clearly some elements that are very specific to the Aotearoa context.
The key things that are very different between this and the many other evaluator competency lists that have been developed around the world are, in my view: (1) the ‘valuing’ piece, i.e. defining what outcomes should be considered ‘valuable’ and ‘important’ and how ‘quality’ is defined for a particular evaluation and (2) putting cultural values and worldviews right at the centre of not only the valuing piece, but also in defining what constitutes [culturally] appropriate approaches, methodologies, analysis, reporting, etc.
Policy-relevant, strategic evaluation asks (and answers) questions that go beyond the evaluation of a single initiative. It addresses such questions as: (1) What is the value of a particular policy initiative as a contributor to strategic policy outcomes? (2) How well does each policy initiative fit with and complement the other initiatives that make up the strategic policy mix? Are there any unnecessary overlaps? (3) What is the collective value of the suite of initiatives that have been deployed to achieve a particular strategic outcome? (4) Have we got the right mix to deliver on the key outcomes? (5) Which approaches to achieving key long-term outcomes are working most cost-effectively for whom, under what conditions, and why? This session will outline five key elements needed to position and effectively engage with the evaluation function so that it delivers high quality, truly useful information to calibrate macro-level policy – and to incisively inform Ministers on policy effectiveness.
Want us to keep you posted?
Please subscribe to the site feed to receive updates about workshops, presentations, publications and keynotes – plus Real Evaluation ideas and techniques. See top right corner for subscription options (email updates highly recommended).
Let’s face it. Every year there are evaluations that bitterly disappoint client organisations by failing to produce actionable answers to important evaluative questions. Weak evaluations frequently lack incisive evaluation questions; employ the wrong methodology for the questions; get lost in the details; skip over the crucial ‘values’ step and therefore can’t draw evaluative conclusions; uncritically accept stated objectives as the only evaluative criteria; focus only on the average effect; fail to adequately triangulate and transparently weave sources of evidence; toss causation into the ‘too hard basket’ while still claiming to have documented ‘outcomes’; and fail to clearly communicate findings. This session will help managers and commissioners of evaluations see the possibilities so they can become informed and demanding consumers of real, genuine, actionable evaluation. You will get important pointers for writing RFPs and managing selection processes with a better chance of attracting a shorter list of high quality proposals; and for effectively managing the evaluation so as to maximise the chances of getting clear, authentic, actionable answers to high priority questions. Evaluators will also be interested in hearing about how their work can better meet the needs of clients and other stakeholders.
Want us to keep you posted?
Please subscribe to the site feed to receive updates about workshops, presentations, publications and keynotes – plus Real Evaluation ideas and techniques. See top right corner for subscription options (email updates highly recommended).
If a reported “outcome†is not caused by a programme, it is not an outcome at all; it’s a coincidence. Simply measuring variables that may or may not be causally related to a programme (i.e. could just be coincidences – who knows?) doesn’t tell you anything about the quality or value of the programme, therefore it can’t be referred to as outcome evaluation – it’s just measurement.
Isn’t causal attribution heinously expensive, almost never feasible, and doesn’t it require some form of experimental design? Not necessarily. In this interactive seminar, Jane will use case examples to illustrate eight strategies for inferring (or ruling out) causal links between programmes and suspected outcomes: (1) Ask those who have observed or experienced the causal effect, (2) Check if the content of the intervention matches the nature of the outcome; (3) Look for distinctive effect patterns (modus operandi method), (4) Check whether the timing of outcomes makes sense, (5) Examine the relationship between program “dose†and “responseâ€, (6) Use a comparison or control, (7) Control statistically for extraneous variables, and (8) Identify and check the causal mechanisms. These strategies are outlined in Jane’s (2004) book, “Evaluation Methodology Basics: The nuts and bolts of sound evaluation†(Sage).
Want us to keep you posted?
Please subscribe to the site feed to receive updates about workshops, presentations, publications and keynotes – plus Real Evaluation ideas and techniques. See top right corner for subscription options (email updates highly recommended).
