Impact is often a key evaluation need.  It aims to understand the longer term outcomes that are generated through interventions, projects or programmes, that often have particular policy objectives and funding requirements.  Impacts may be positive or negative, intended or unintended, direct or indirect.

This article is designed to give an overview of things to consider and also links to other resources to assist in planning an approach.

“There is not one right way to conduct an impact evaluation. What is needed is a combination of methods and designs that suit the particular situation.” – Rogers et al, 2015

Impact evaluation usually has one or more of the following intents

Accountability – effective risk management and an audit of:  Was what was aimed at being carried out, carried out

Analysis – what is working and not working to inform continuous improvement, does it need to scale up or scale down, is the initiative on track

Allocation – helping inform how to continue to allocate funding and resources and is the resource allocation having the intended impact

Advocacy – a demonstration of the value of the investment

When selecting evaluation methods and designs, three issues must be considered:

1. The nature of what is being evaluated
2. The intended use of the evaluation
3. The available resources (and any constraints)

“Impacts are broader than stated goals. For the purposes of accountability, learning, value-for-money and ethical conduct, it is important that the term ‘impact’ also includes unintended impacts, positive or negative” – Rogers et al, 2015

Impacts do not only refer to what has happened, in some cases, the impact is in terms of preventing negative changes: ‘Impact also includes the reduction, avoidance or prevention of harm, risk, cost or other negative effects’, (Warwick & Nolan 2014).

Other types of evaluation to impact evaluation

Values clarification (Needs analysis) – What is needed? What would success look like?

Monitoring – How is it going? (regular reporting of metrics)

Process evaluation – How is it going (periodic investigations)?  Is it being implemented according to plan or according to data-informed revisions to that plan?  What has been done in an innovative program?

Economic evaluation – What has been the value of the intervention? Has the intervention been cost-effective (compared to alternatives)?  What has been the ratio of costs to benefits?

Questions for Impact Evaluation

Descriptive Questions – To determine how things are or were and what has happened and has there been any changes.

Causal Questions – What changes (positive or negative) have taken place and can be contributed to the investment.  What has contributed to any impact.

Evaluative Questions – Those that indicate the value or otherwise or the merit or worth of the changes that have been brought about.

Economic evaluation – this is often added to assess the cost benefit or cost effectiveness of the approach.

Economists will often be asked to conduct a cost-benefit analysis to determine which programme delivers the best value for money.  Programmes whose outcomes are not readily measurable will suffer.  Understanding the context in which the results were achieved (the why) is also important.  This points towards a serious issue for Government funded industry programmes:

“The danger is twofold. The first danger is that only relatively simple programmes are developed. There is a risk that programmes become less ambitious when they place a greater emphasis on measuring outcomes. The second danger is that promising programmes are not valued simply because their results cannot be measured, while relatively ineffective programmes are valued because aspects of them can be more easily measured.” – Rogers et al, 2015

Processes to Use

Programme Logic – “Programme logic can help to identify gaps in logic or evidence which should then be a priority for the evaluation. It identifies intermediate outcomes which could be evident in a shorter time frame and provide early indications of whether or not the programme is on track to achieve the intended impacts and whether the risks of negative impacts are being effectively managed. Programme logic also provides a structure for constructing an evidence-based narrative about the value and impact of a programme – Rogers et al, 2015.

For Further Information

Roger, P., Hawkins, A., McDonald, B., Macfarlan, A., Milne, C. (2015) Choosing appropriate designs and methods for impact evaluation.  Office of the Chief Economist – Australian Government

Warwick, K. and A. Nolan (2014), “Evaluation of Industrial Policy: Methodological Issues and Policy Lessons”, OECD Science, Technology and Industry Policy Papers, No. 16, OECD Publishing, Paris

BetterEvaluation 2015, http://betterevaluation.org/

CSIRO 2014, Impact Evaluation Guide (unpublished).