Enhancing project impact and science capability through ongoing evaluation

• Ministry of Agriculture and Rural Development, Vietnam

The Vietnamese Government has increasingly directed budgetary resources to agricultural research and development. There has been a corresponding increase in community demand for accountability and expectations of seeing real outcomes from R&D. However a strong evaluation framework to demonstrate and help deliver outcomes is lacking, and there is a need to strengthen both the role and frameworks for evaluation to address this problem.
The ability to evaluate a range of R&D projects and their outcomes, including such grey areas as scientific capacity and capability, begins with project planning. But many Vietnamese organisations have little experience in organisational learning and supporting evaluation systems, and as a result their planning to attain efficient and effective R&D outcomes is often limited or absent. This is particularly the case with 'fuzzy' or poorly defined concepts and projects. This creates a separation between producing project outcomes and the expectation of meeting public demand to show outcomes and link them to new projects.
In Australia the Department of Primary Industries, Victoria (DPIV) has addressed a similar set of issues, driven by State Government policy priorities. Part of this approach has focused on evaluation tools for emerging impacts of R&D and how these can be applied to guide projects to better outcomes. Mapping outcomes of science capability and capacity has also been addressed through a story approach utilising performance indicators and concept mapping. The DPIV approach was trialled and tested for its applicability in the Vietnamese context at the Ministry of Agriculture and Rural Development (MARD).

The project aimed to enhance the impact of agricultural research by MARD and DPIV by developing an evaluation procedure for research and development projects suited to the MARD environment, and by developing a procedure for evaluating science capability (initially for DPIV, later for MARD).

The project team undertook an analysis of MARD's current evaluation approaches and developed draft evaluation guidelines for MARD. Initially they conducted a trial and modification of DPIV and MARD evaluation guidelines on two MARD/ACIAR projects, then later in the project applied the work to a further selection of MARD projects. Other activities included training and professional development for two MARD evaluation development officers selected to implement the trials, and periodic evaluation of the trials and guidelines.
To develop a procedure for evaluating science capability (initially for DPIV, then later for MARD), the methods included:
concept mapping to define areas of interest in science capability, around which performance indicators and stories were collected
design and implementation of a performance indicator system
design and implementation of the story approach (also known as the Most Significant Change approach) for evaluating science capability
periodic evaluation of the trials undertaken.

This project helped to ensure that monitoring and evaluation has a bright future in The Department of Science and Technology (DST) of the Ministry of Agriculture and Rural Development (MARD). Some of the solid outcomes achieved include Monitoring and Evaluation (M&E) Guidelines, and a Science Capability Framework (SCF). DST now has a network of trained evaluators and is fully committed to utilising these resources (including committing of funds) to ensure this is achieved.
The development of M&E Guidelines in both Vietnamese and English for use in DST has been a large component of the project and required extensive work by both partner countries. The bringing together of cultural knowledge from the staff at DST along with the reinforcement of current evaluation thinking and practice from the Department of Primary Industries Victoria (DPIV) has established a framework upon which further evaluation capability can now be built. Support from senior staff needs to be ongoing within DST if M&E momentum is to continue becoming an entrenched part of the organisational culture.
The M&E Guidelines developed are an excellent resource for ongoing training. Staff members have gained practical experience in evaluation and training and have trialled the guidelines as a training tool; further experience and skill development will enhance the skill transfer process.
DST has established a network of over 100 evaluators and has designed a three-tier capability structure that incorporates managers, practitioners and experts to ensure that M&E is part of the normal way of doing business. DST is committing 1.5-2% of the national budget for agricultural research to M&E, including funds to continue training within the organisation.
The success of this project is recognised at Ministerial level by both the Victorian and Vietnamese governments. Staff who trained through the project are being sought by other parts of the Vietnamese government, NGOs and throughout DST - a compliment for the project but an ongoing challenge for DST management, as unmet demand will restrict the growth of M&E within the organisation.
To assist in the process of embedding M&E within the organisation and ensuring momentum is not lost, DST has developed a set of M&E Regulations that provides the necessary institutional controls.
Objective 2 of the project related to the development of a tool to evaluate science capability. This was undertaken first in DPIV and then trialled in MARD. The MARD experience highlighted some points of difference relating to the prioritisation of categories, and this led to further work to ensure the tool fitted MARD's specific situation.