The value of collaborative learning: building the OutNav Community

We know that for an organisation wrestling with challenges such as how to streamline data collection or make sense of their social media analytics, nothing beats the chance to learn from others who have been there before.

It has always been a priority for us to establish a strong community of users around our approach. Therefore, we were delighted to welcome a pioneering group of OutNav users to our second OutNav Community get together in Codebase on the 30th August 2018.

It is just over six months since we gave our first group of beta testers from Starcatchers, Healthcare Improvement Scotland and the Thistle Foundation access to OutNav. Since then, we have been joined by more beta testers and together we have made significant progress on our journey to create the first software product that enables non-expert researchers to plan and implement robust and meaningful outcome evaluation using a theory based approach.

It has been inspiring to witness the innovative work our clients have taken forward to embed meaningful outcome evaluation practices within their day to day work.

Our clients have overcome numerous practical, political and conceptual challenges, like reconciling the outcomes of their work with those specified by funders and working out mechanisms for staff to systematically reflect on practice. In so doing these organisations are cementing their place at the forefront of best practice internationally in meaningful outcome evaluation.

The focus of this meeting of the OutNav Community was on sharing practice.

Pioneering OutNav users from three organisations, presented their work to date. Fraser Stone from Starcatchers presented his report from the Creative Kin project evaluation he ran from start to finish in OutNav. Tabitha Casey from the University of Edinburgh demonstrated her use of OutNav to complete the final analysis and report on the Partnership Research to Tackle Violence Against Children in Zimbabwe project. Ashley Spalding from Stirling Council talked about her plans to use OutNav for the evaluation of the Neighbourhood Care project. The community enjoyed engaging with, and reflecting on the issues and learning from these projects.

One of the most overwhelming issues emerging from the discussions was the time that it took to become confident in outcome evaluation.

Everyone in the room had stories to tell of early mistakes, be that in planning, analysing or engaging colleagues.

People also had masses of learning from the early months of their work in this way. Some of the key messages and issues are explored here:

The value of meaningful outcome stories

Whilst talk of outcomes is everywhere across UK public services and beyond, the reality is that very few projects or organisations meaningfully report on outcomes. Common approaches to outcome reporting focus solely on the outcomes ‘achieved’ and in doing so mask the complexity and value of the work undertaken. Whilst most people at the meeting had been working with the approach and software for some time, this approach to reporting is new and they all found seeing other peoples’ outcome reports created through OutNav inspiring. As one person said, “it is really helpful to see the story”. It was particularly valuable to see different approaches others had taken to incorporate photos and graphics into their reports, really bringing the story alive. Other participants shared positive feedback they had already received on the coherence of their outcome-based reporting.

Getting people on board

OutNav is a tool that enables organisations to embed evaluation into everyday work. Whilst being able to draw on external or specialist research is often very helpful, we believe that making evaluation everyone’s job is critical to ensuring that the learning from any process of evaluation is continually translated into improvement. A key issue discussed at this OutNav Community meeting was how to get people on board with evaluation early in the process. Effective strategies discussed included:

• engaging people from across the organisation or project in the process of outcome mapping and evaluation planning,
• spending time with front line workers to understand from them what is realistic in terms of data collection as part of their role,
• engaging funders, commissioners and board members early when implementing the approach so they know the kinds of reports to expect, and
• using OutNav in team meetings so that everyone has a chance to participate in assessing progress to outcomes and also discussing what can be done to improve the projects as well as the evidence base.

Understanding context

Participants highlighted the complexity of their projects and the multiple factors that influence whether or not their projects and programmes will make an effective contribution to improving outcomes. For many of these projects, small improvements in knowledge, confidence, skills or behaviours for some of the people they work with, constitutes a big success. Other projects are very vulnerable to factors out with their control undoing all of their good work, for example cuts to mainstream services and or benefits. Participants highlighted the importance of making these challenges to projects explicit early on. This is important to ensure that:

• funders and commissioners have reasonable expectations about what can be achieved,
• risks can be understood and mitigated, and
• risks and assumptions for the evaluation are identified and data collected to understand how they are affecting the work.

To help organisations achieve this we often use the ‘ISM Behaviour Change Framework’ a toolkit developed by the Scottish Government to promote environmental behaviour change, but that can be used for any change project to understand how factors operating in the individual, social and material contexts are likely to influence success. Participants were particularly interested in this tool and in hearing how others had used it alongside the outcome maps to analyse context.

Find a champion (or two)

A key reflection from the morning was the importance of having one or more people champion outcome evaluation within each organisation. Participants highlighted the many different time pressures everyone faces and how easy it is for evaluation of any kind to slip down the agenda. They also reflected the need to engage with the process regularly to keep up confidence and skills in driving forward the evaluative work.

Investing time in OutNav is time well spent

Using OutNav creates new opportunities for outcome evaluation, for example the ability to share good evaluative practice between projects. Participants reflected that the more time they spent in OutNav the more they benefitted from its accessible and structured approach and its collaborative and practical functionality.

For us here at Matter of Focus this second OutNav Community meeting felt like a really important milestone. We have spent the past year and a half planning for and building a tool that reflects the complexity of public service evaluation and supports non specialist researchers to carry out great outcome evaluation in a way that embraces, rather than reduces, that complexity. Along the way we have had moments of uncertainty and questions about whether we have bitten off more than we can chew. Through all of this the commitment and inspiration of our amazing pioneering clients has kept us focused and on track. We are very excited about what we can achieve together over the coming months and years!