What are some useful ideas and tools for schools working to accelerate student achievement in literacy (and/or other areas), particularly for Maori and Pasifika students? This presentation was delivered to school leaders, literacy leaders and their literacy facilitators (external PLD providers) participating in the Secondary Literacy Project (with overall design and national coordination from the University of Auckland’s Woolf Fisher Research Centre).
In it, I share some thinking about:
interpretation of progress data from e-asTTle, using …
a range of comparisons that can be made depending on the evaluation question one asks
multiple perspectives on effect sizes and their interpretation
the use of evaluative rubrics for interpreting accelerated student progress and achievement, and
for inquiring into key drivers of student progress such as effective teaching, educational leadership and school culture, case management and implementation, and many more
Click to download a PDF of the presentation (including several slides that were omitted from the actual session).
As part of this week’s National Professional Learning and Development Facilitator Training (held in Auckland and sponsored by the Ministry of Education), Nan Wehipeihana and I presented a workshop for PLD providers:
This is an introduction to some of the rubrics developed for the Measurable Gains Framework, designed to evaluate the effectiveness of Ka Hikitia, the Ministry’s strategy for Maori learners.
It was a reflection on part of the theme for the 2010 American Evaluation Association conference – Evaluation Quality – with a focus on Ernie House’s (1980) concepts of truth, beaty and justice in evaluation (click the link to access PDFs of the key chapters in House’s book, from the AEA e-library).
Although House wrote that truth trumps beauty and justice trumps them all, Jane argued for the critical importance of ‘beauty’ (the coherent, persuasive, compelling story or argument) – with the ‘values’ and the reasoning that guide conclusions about quality and value made clearly visible, transparent and understandable – as a hugely important path to both truth and justice.
On December 2nd 2010, Jane delivered a keynote address for the Self Assessment for Quality conference, hosted by the Otago Polytechnic in Dunedin, New Zealand.
The audience were representatives from tertiary (=higher) education organisations from throughout Aotearoa New Zealand working to implement a new evaluative approach to quality assurance, where they ask and answer questions about the quality of their offerings and services and the value of their outcomes for learners and other key stakeholders (such as employers, communities and iwi).
The theme of the conference was Self-Assessment for Quality: How do you know good when you see it?
Obviously, some of the slides need further explanation for those who weren’t there. I am planning to flesh out some of the key points on the Genuine Evaluation blog over the next few weeks.
Conference fees for the AES conference in Wellington are pretty hefty for many evaluators, so anyone eligible may want to check out the following offer:
The Australasian Evaluation Society is pleased to invite emerging Indigenous evaluators from Australia and New Zealand and emerging evaluators from South East Asia and the Pacific to apply for financial assistance to support their attendance at the 2010 AES International Conference. The AES will provide support with applications on request. Requests should be received no later than 21 May 2010
Roger Maaka, an academic who has studied and advised on indigenous issues internationally.
Donna Mertens is past president of the American Evaluation Association and editor of the Journal of Mixed Methods Research, as well as a professor at Gallaudet University in Washington, DC.
Thomas Schwandt is a professor at the University of Illinois at Urbana-Champaign and a member of a standing committee of the National Research Council of the US National Academies of Sciences.
Elliot Stern is Professor of Evaluation Research at Lancaster University and the current editor of Evaluation: the international journal of theory, research and practice.
Jane Davidson, formerly the Associate Director of the Evaluation Center at Western Michigan University, Jane now heads an evaluation consulting firm in Auckland, New Zealand.
Some key things to note about the competencies are:
This is not a checklist of competencies that every evaluator should have ALL of in order to be ‘competent’; this is a list of competencies that all evaluators will have some of, but no one person will have all of.
Each evaluator and each evaluation consulting business needs to consider what strengths profile it has (and is aiming to develop). In other words, how do you position yourself as distinct from other evaluators or evaluation consulting firms in this country (or, internationally, if that’s the space you compete in)?
Each evaluation team (and each manager commissioning an evaluation) needs to think through what profile of competencies is required for a particular project and whether their (or, a proposing) evaluation team collectively has those strengths.
Although this list will be of some interest internationally, there are clearly some elements that are very specific to the Aotearoa context.
The key things that are very different between this and the many other evaluator competency lists that have been developed around the world are, in my view: (1) the ‘valuing’ piece, i.e. defining what outcomes should be considered ‘valuable’ and ‘important’ and how ‘quality’ is defined for a particular evaluation and (2) putting cultural values and worldviews right at the centre of not only the valuing piece, but also in defining what constitutes [culturally] appropriate approaches, methodologies, analysis, reporting, etc.
Policy-relevant, strategic evaluation asks (and answers) questions that go beyond the evaluation of a single initiative. It addresses such questions as: (1) What is the value of a particular policy initiative as a contributor to strategic policy outcomes? (2) How well does each policy initiative fit with and complement the other initiatives that make up the strategic policy mix? Are there any unnecessary overlaps? (3) What is the collective value of the suite of initiatives that have been deployed to achieve a particular strategic outcome? (4) Have we got the right mix to deliver on the key outcomes? (5) Which approaches to achieving key long-term outcomes are working most cost-effectively for whom, under what conditions, and why? This session will outline five key elements needed to position and effectively engage with the evaluation function so that it delivers high quality, truly useful information to calibrate macro-level policy – and to incisively inform Ministers on policy effectiveness.
Want us to keep you posted?
Please subscribe to the site feed to receive updates about workshops, presentations, publications and keynotes – plus Real Evaluation ideas and techniques. See top right corner for subscription options (email updates highly recommended).
Let’s face it. Every year there are evaluations that bitterly disappoint client organisations by failing to produce actionable answers to important evaluative questions. Weak evaluations frequently lack incisive evaluation questions; employ the wrong methodology for the questions; get lost in the details; skip over the crucial ‘values’ step and therefore can’t draw evaluative conclusions; uncritically accept stated objectives as the only evaluative criteria; focus only on the average effect; fail to adequately triangulate and transparently weave sources of evidence; toss causation into the ‘too hard basket’ while still claiming to have documented ‘outcomes’; and fail to clearly communicate findings. This session will help managers and commissioners of evaluations see the possibilities so they can become informed and demanding consumers of real, genuine, actionable evaluation. You will get important pointers for writing RFPs and managing selection processes with a better chance of attracting a shorter list of high quality proposals; and for effectively managing the evaluation so as to maximise the chances of getting clear, authentic, actionable answers to high priority questions. Evaluators will also be interested in hearing about how their work can better meet the needs of clients and other stakeholders.
Want us to keep you posted?
Please subscribe to the site feed to receive updates about workshops, presentations, publications and keynotes – plus Real Evaluation ideas and techniques. See top right corner for subscription options (email updates highly recommended).
an overview summary table showing the main evaluator competency documents that have been developed around the world
short analyses of each of the competency documents reviewed, including a separate and quite substantial analysis on cultural competencies for evaluators
a discussion of some of the main issues to consider in how competencies are constructed/written
summaries of discussions about the Aotearoa context and what competencies are going to be particularly applicable and important here
This document is being released as background reading while we wait for the draft evaluator competencies from anzea to come out for wider consultation in April.
If a reported “outcome†is not caused by a programme, it is not an outcome at all; it’s a coincidence. Simply measuring variables that may or may not be causally related to a programme (i.e. could just be coincidences – who knows?) doesn’t tell you anything about the quality or value of the programme, therefore it can’t be referred to as outcome evaluation – it’s just measurement.
Isn’t causal attribution heinously expensive, almost never feasible, and doesn’t it require some form of experimental design? Not necessarily. In this interactive seminar, Jane will use case examples to illustrate eight strategies for inferring (or ruling out) causal links between programmes and suspected outcomes: (1) Ask those who have observed or experienced the causal effect, (2) Check if the content of the intervention matches the nature of the outcome; (3) Look for distinctive effect patterns (modus operandi method), (4) Check whether the timing of outcomes makes sense, (5) Examine the relationship between program “dose†and “responseâ€, (6) Use a comparison or control, (7) Control statistically for extraneous variables, and (8) Identify and check the causal mechanisms. These strategies are outlined in Jane’s (2004) book, “Evaluation Methodology Basics: The nuts and bolts of sound evaluation†(Sage).
Want us to keep you posted?
Please subscribe to the site feed to receive updates about workshops, presentations, publications and keynotes – plus Real Evaluation ideas and techniques. See top right corner for subscription options (email updates highly recommended).
