How do you know if your project is effective?

The Citadel Youth Centre

The Citadel Youth Centre is a community-based youth service in Leith, Edinburgh, that has been working for decades with disadvantaged young people and their families.

Matter of Focus worked with Citadel to help evaluate their Big Lottery funded Families Project.

Workers collected data from children and parents about what they think of the service and what they have learned and gained, but Citadel staff asked Matter of Focus to approach stakeholders as they felt a neutral outsider would receive more honest opinions.

It was agreed that we would devise and run a stakeholder survey online, and communicate with stakeholders to ensure neutrality.

Using an outcome map to shape the survey

Following our approach, we used the Citadel stakeholder pathway in their outcome map to shape the survey questions.

We drafted survey questions that reflected the desired outcomes across the outcomes map. Initially questions focused on the ‘what we do’ column – asking stakeholders to comment on Citadel Families Project activities. 

The survey then included reflections across the outcome map – looking at engagement, learning and capacity building, behaviour change and higher-level outcomes. We included some questions about the risks and assumptions associated with the outcome map – particularly around working with vulnerable children.

It was possible to ask questions directly relating to outcomes to these stakeholders as they were other professionals who have contact with the same families. They have a degree of professional judgement that can be useful to call on in this kind of evaluation.

The survey asked whether stakeholders agreed that:

1. Children who attend Citadel are at less risk due to Citadel’s involvement
2. Families who attend Citadel strengthen their links with the community
3. Citadel’s engagement with parents and children you know has led to:

• Improved parenting skills
• Improved relationships within families
• Improved attachment between parents/carers and children
• Increased engagement with other services
• Other (if other, please expand)

All but one of the stakeholders who responded felt able to comment on all these outcomes.

Only one of the high-level outcomes was included in a direct question, which was whether stakeholders thought that fewer children will be ‘looked after’ at home, in kinship care or accommodated.

A taste of what we found out

The majority of stakeholders who took part in the survey either strongly agreed (59%) or agreed (30%) that children who attend Citadel are at less risk due to Citadel’s involvement. Two respondents neither agreed nor disagreed as shown below:

In order to further understand how Citadel has improved outcomes for children, when stakeholders were asked what was gained overall by working with Citadel, respondents stated;

”[Citadel] really enhances the support we can give to our children, building their confidence, resilience and improving their relationships with others”.

“Citadel are a valuable support to families and other involved organisations. The staff are very skilled at engaging children and their families and supporting them towards change.”

“Children have more self-esteem and confidence. The work from Citadel is invaluable and the support is hugely appreciated.”

Citadel staff are pleased to have gathered these views as further evidence of the effectiveness of their programme.

“These results look great. And I feel they have a validity to them coming from an external evaluator. We will be able to incorporate the results in our final project evaluation and refer to them in future funding applications. So, thanks to Matter of Focus for doing this.”

Andy Thomas the Citadel Families Project Manager

You can read more about the Citadel Families Project here.

Try this for yourself

If you have a well-defined outcome map with risks and assumptions, it can be quite simple to create a survey, interview or focus-group questions from it:

1. Go through each item (stepping stone) in the map and see if you can turn it into a question.
2. Remember that you can include direct questions, Likert scales (that ask ‘to what extent’ and have a choice of answers on a simple scale), or open questions that allow people to respond how they want through free text.
3. Include some of the key risks and assumptions that the people who will be asked are best placed to comment on (in the example above stakeholders had good insight into how families used other services, and to what extent children were at risk – both key risks).
4. Allow at least one place where people can give their views freely.