How to bring different voices into your evaluation

When used well, quotations have the power to give an immediate and unique insight into the perspective of the person who has offered it. This brings many benefits.

The benefits of using quotations

Using quotations well:

• forms an important part of the evidence base to support your impact/outcome claims.
• brings reporting to life, providing vital insights into the world and personal experience of the speaker.
• provides space for people to speak for themselves, going some way to addressing power imbalances around any human service.
• emphasises the views of people using services, communities and front-line workers, which can help highlight and learn from their experiences to change the way we think about an issue or programme.
• communicates what matters by giving any audience a feel for the work by preserving the language that people themselves use to describe their experiences.
• can be memorable, compelling and accessible, engaging your audience in a way that other evidence cannot.

Your organisation may have good reflective practices already in place, and it’s likely that your relationships with the people that use your services offer rich opportunities for gathering voice. But how do you use these assets to their fullest potential to learn, improve and tell the story of how your activities make a difference?

Building skills and confidence in qualitative analysis, and implementing consistent recording practices and processes are important first steps that will help you draw together this kind of information easily when it is needed.

Building the story of your contribution

Your organisation may already use voice powerfully and effectively to communicate your work or to advocate for and with the people you serve, for example in case studies or funding bids. At Matter of Focus we advocate for building on this practice as part of a structured evidence base, which, in combination with other forms of evidence and over time, creates a clear “contribution story”.

A contribution story is an evidence-informed narrative that connects an organisation or iniative’s activities (“what we do”) over a series of steps to the outcomes or impacts they are seeking to influence (“what difference this makes”).

When we work with initiatives to help them understand and build their contribution story we use a process called outcome mapping that takes a plain language approach to understanding change, using our headings. For more on this see our post: A simple framework to help you understand change.

How to use quotations well

Using voice well to support your evaluation involves a series of steps:

1. Working out who is your focus
2. Identifying opportunities for gathering quotations
3. Reflecting carefully on ethical responsibilities
4. Ensuring quality and robustness in your interpretations
5. Presenting your work in a clear, attractive and engaging way

1. Who is the Focus?

If you have already created an outcome map and pathways for your initiative, these will help guide you in working out whose voices you need to include and provide a focus for data gathering.

It is important to think about who will notice the differences or impacts you are seeking, and how they will know. (What will they see? What will they hear?) This could range from a family member who notices a change in a person using your service to a senior decision-maker who notes a policy or system change. Often, people realise the importance of collecting the experience of people who benefit from initiatives, but neglect the insight and reflections of colleagues or other people who might be important to understanding change.

When gathering data in our work at Matter of Focus, we often match the questions we ask people to our six outcome mapping headings and to the particular project, programme or organisational outcomes that sit under each of these:

• What you do
• Who with
• How they feel
• What they learn and gain
• What they do differently
• What difference does this make

This helps to structure reflections and pull data together once collected.

2. What opportunities do you have for data gathering?

There are many opportunities for recording voice as part of everyday practice. You may find that you are doing some of these already or are able to adapt to them very easily.

Quick wins:

• If you’re holding practice reflection in your team online, why not record the conversation? (You can do this in Zoom and Teams.) You can then pull-out key data and quotes
• With permission, recording phrases given as informal/casual feedbac
• With permission, recording phrases as part of your routine contact recording – considering how you will be able to bring all of these together
• Gathering comments from your social media platforms
• Implementing a simple feedback form.

Consider:

• Interviews including exit interviews or periodic focus group discussion
• Including open questions in your feedback survey
• Feedback cards using prompts such as unfinished sentences or questions
• Creative work such as artwork or modelling that allows conversation to unfold alongside it
• Gathering comments offered when completing standard measures or using standard tools that support your work
• Gathering reflections from colleagues involved in delivering your initiative using reflective impact logs.

---

---

3. What are your ethical responsibilities?

Collecting data from or about people falls within the rules, regulations and ethics of the setting where it is collected. If you are gathering someone’s thoughts or experiences for one purpose, for example for a support plan, or when discussing an individual case with a partner agency, it may not be appropriate to use these publicly. It is good to be very clear with people about how and why you want to use their data, and to get their permission for this when you collect it. Whether or not you need permission will also depend on your organisation and data standards – consider anonymity, context and who will see the data.

A spirit of openness and respect is key to ensuring a good process for everyone involved.

4. How will you interpret and analyse your data?

Before reporting or presenting quotes you need a process of critical analysis and interpretation. This involves reviewing the data and selecting quotations that illustrate what you think is important. It is particularly useful to go through the analysis and interpretation process with other people to reach collective understanding. OutNav provides a clear and versatile structure to help with this process – see our post on how this works: Track your progress and impact in OutNav.

A process of analysis usually involves highlighting common themes whilst also being aware that including minority views can be important – if one person’s experience is very different from the rest what can you learn from this? Your analysis will be stronger if you can connect the dots between evidence that comes from different sources, for example if there is alignment between what people say and what you or someone else has observed. As part of your critical thinking and sense-making, it’s important to reflect on what might have influenced the voices you have heard, for example do the people you have spoken to dislike being interviewed, and are people agreeing because they want to please you or don’t want to lose the service?

Here are some questions to help you when you’re analysing your quotations

• What factors have shaped the voices you have captured?
• Have you truly given a voice to all of the different participants and stakeholders, attending to and trying to understand different voices and views?’
• How do different accounts fit together? Can you see patterns across groups or between groups?
• What conclusions can be drawn and where do you need to be more cautious?
• What is being said or not said?
• Why do you think they might have said that?
• Does anything surprise you?

5. How should you present quotations

Presenting your work well – in reports, case studies or other presentations, makes a huge difference to how voice and your contribution story will be heard. Based on our experience, we offer the following top tips.

Tips for presenting ‘voice’:

• Be selective – voice should illustrate your deeper understanding of the patterns in what you have collected. You don’t have to include everything and in fact it’s better if you don’t.
• If a quotation is short it should normally sit in-line with quotation marks and if it is longer it should feature as indented text (or in a box). The same formatting should be used throughout so it is clear when you are using voice and when it is the organisation’s point of view/interpretation.
• While studies of language and communication use detailed transcription, it’s not necessary to include every ‘um’ and ‘you know’ in evaluation reporting – focus on readability and getting the person’s meaning across.
• Try showing evidence of different kinds together to strengthen the point you are making, e.g. a visual image or quantitative data alongside voice.

---

“For a recent evaluation, I used a SmartSurvey to find out how young people experienced receiving individual grants. I included questions that combined a multiple-choice response with an open-ended follow-up asking ‘why?’ I presented the quantitative and qualitative data side by side in a simple table so the reader can see how many people responded in a particular way, alongside sample quotations to give a picture of the meanings behind this. In choosing the quotations, I was mindful to pick a small selection illustrating the main kinds of response – in this case more general statements of appreciation and very particular impacts on education and employment the funding had brought”

---

• Icons or colour-coding can be used to differentiate types of participants or the theme of the quotations selected.

• Word clouds and word lists (showing frequencies) are good for showing frequently used key words.

More on how to make good use of qualitative data

---

---