A new framework for assessing the impact of ACIAR-supported biosecurity programs is being developed. It will measure not only tangible economic and environmental impacts, but also benefits to society such as improved livelihoods and capacity building, which may not have previously been systematically assessed.
The framework is being designed by The University of Melbourne’s Centre of Excellence for Biosecurity Risk Analysis (CEBRA). It presents a range of modelling options to evaluate impacts across a wide scope of biosecurity-related projects supported by ACIAR.
CEBRA chief investigator Professor Tom Kompas leads the framework development for ACIAR, which he said is due for completion early next year. The framework establishes a template for a transparent, rigorous investigation of the contribution that particular sets of projects make to biosecurity and societal outcomes.
Although economic benefit remains a critical measure, the framework contains other features that provide a more far-reaching and nuanced assessment of impact, he said. ‘It’s not just dollars and cents. It’s community impact, social impact and environmental impact.’
Measurement of the broader impacts around societal change are guided by employing a ‘Theory of Change’ (ToC), developed at the framework’s core. A ToC distils the thinking behind how a project is expected to contribute to longer-term changes. For ACIAR-supported biosecurity projects, this may include increased policy capability to manage pests and diseases, better health, more secure and profitable livelihoods, and even improved gender equity.
The impact assessment framework developed a series of generic ToCs for different types of biosecurity research projects – for example, biological control, vaccination and integrated pest management projects. These generic ToCs represent the hypotheses about how these projects were expected to work. The team will then collect data and develop models to test the hypotheses in the real world and show the scale of changes achieved in these different domains.
CEBRA ecologist Dr Edith Arndt, who developed the framework’s ToC models, said significant societal benefits from biosecurity projects have traditionally flown under the radar.
‘This framework builds on the good evaluation culture ACIAR has, and provides a more robust structure around evaluating the impact on issues, such as food security and gender equity, by using metrics and not just qualitative information based on anecdotes and observations,’ said Dr Arndt.
Other features of the framework include bioeconomic modelling options for evaluation, such as a sophisticated cost–benefit analysis that can measure intangible benefits of suppressing pests or disease, and a toolbox of spread models that simulate the growing distribution of target pests.
CEBRA economist Ms Christine Li, who identified and developed the framework’s economic and spread analysis models, said the resource aligns with ACIAR objectives by enabling the collection of quantifiable data on economic, social and environmental impacts.
‘Having these sorts of evaluation guidelines means that people can collect data necessary for evaluation right from the start,’ said Ms Li. ‘Because you can’t manage what you don’t measure.’
Benefits and improvement
Over the past 40 years, ACIAR has invested in more than 90 biosecurity projects focused on pests and diseases affecting food security and livelihoods for smallholder farmers in diverse agrifood production systems in partner countries.
‘Most of these projects also have a significant value to Australia as it helps us to understand, prepare for and prevent cross-border spread of pests and diseases more effectively,’ said Ms Bethany Davies, ACIAR Research Program Manager, Portfolio Planning and Impact Evaluation.
The need to more accurately capture the regional benefits of strong biosecurity systems, as well as the value of benefits for Australia in terms of forward defence against biosecurity threats, underpins the new framework being developed.
Ms Davies said that although ACIAR can clearly demonstrate that its use of the Official Development Assistance budget has made a substantial difference to the specific places where it works, the value of co-benefits to Australian farmers has not been as clear.
Our work in biosecurity is an example of an area of work that simultaneously delivers huge benefits to both partner countries and it is important to share these stories as well.
Ms Bethany Davies,
ACIAR Research Program Manager, Portfolio Planning and Impact Evaluation
‘The evaluations also help us learn more about how to conduct agricultural research for development more effectively in the future – by reflecting on what worked and what didn’t and how we can achieve better development results,’ said Ms Davies.
She expects the new framework will help to more systematically document how biosecurity projects can contribute to multiple benefits and support the development of more sophisticated and nuanced research projects in the future.
ACIAR PROJECT: ‘Valuing the contribution of ACIAR to biosecurity in Australia and overseas’ (IE/2021/168)