Evaluation Report Family Support – The Transition and Resilience Project: Feasibility and pilot study report

Executive summary
The project
The Transition and Resilience (T&R) programme provided support to families, aiming to strengthen the home environment, empower families to make positive changes and improve children’s behaviour. Schools referred 10-to 14-year-olds who were at risk of exclusion and showed evidence of antisocial behaviour (ASB), poorattainment or early childhood trauma to the London Borough of Hammersmith and Fulham (LBHF) FamilySupport service, which ran the intervention. Families were typically offered up to six months of support, usuallyfor around one hour per week, delivered to the children directly in schools or at home. There were up to sixcomponents offered, including intensive support from an inclusion practitioner; group work and peermentoring; ‘Brain in Hand’ (personalised support accessed via a phone or iPad that draws on cognitivebehavioural therapy (CBT) to support children); counselling sessions delivered by Teens in Crisis Counselling(TIC+); mentoring delivered by Active Successful Engagement (ASE); and a virtual reality (VR) intervention thataims to improve children’s decision-making delivered by Round Midnight. The components that were offeredwere tailored to support young people’s specific needs and aspirations.
YEF funded a feasibility and pilot evaluation of the T&R programme. The feasibility study aimed to ascertain what factors supported or interfered with the successful delivery of the programme, whether the intervention’s recruitment, retention and reach were feasible and what service users felt about it. Forty-six families participated in the intervention during the feasibility phase, and 15 participants (including eight professionals, four parents/carers and three children) took part in focus groups and interviews for the feasibility evaluation. The pilot study then aimed to describe the referral and screening process, assess family retention, ascertain the readiness for a larger scale evaluation, evaluate the implementation process and assess the direction and magnitude of changes in child behaviour and family-functioning outcomes over time. Seventeen participants (including five young people, five adults with parental responsibility, and seven professionals) were interviewed for the pilot, while quantitative data for a total of 89 families involved were analysed. Quantitative measures used related to the delivery of the programme (including data on referrals, screening and assessment processes), demographic data and some core measures [including the Strengths and Difficulties Questionnaire (SDQ), a measure of behaviour, and the SCORE 15 Index of Family Functioning and Change (SCORE 15)]. The evaluation was undertaken from February 2020 to May 2022. Both the feasibility and pilot studies took place during the COVID-19 pandemic, requiring both the delivery and evaluation teams to adapt to challenging circumstances.
Key conclusions
During the feasibility study, the COVID-19 pandemic and consequent social distancing measures had a profound impact on the project. The first lockdown prevented delivery and necessitated amendments to the original model of delivery. Recruitment was still effective. Family Support developed strong relationships with schools to facilitate referrals, and the referrals made by school partners were generally considered appropriate. It was clear that demand for the programme was considerably higher than the supply of places.
In the feasibility study, the small number of parents and children interviewed identified a range of benefits and improvements that they attributed to the programme These included supporting children’s transition to secondary school and improving behaviour. Overall, these interviewees had a positive experience. Areas for improvement suggested by participants included extending the length of the intervention and more communication from the intervention team.
The pilot found that programme retention was satisfactory, with the programme reporting that 60% of families completed the intervention. However, measure completion rates were low, with only 49% of families providing data six months into the programme. Poor data reporting meant that it was also not possible to fully assess the referral process.
Despite changes caused by COVID-19, the programme was broadly implemented as intended and in alignment with the logic model. One change made during the feasibility phase was to remove the Brain in Hand, CBT-informed, personalised phone and iPad support, as this was not being used.
The evaluator judges that the T&R programme was not ready for a randomised controlled trial. Several issues would need to be addressed before proceeding to further evaluation, most notably improving data collection.
Interpretation
During the feasibility study, the COVID-19 pandemic, reorganisation and redundancies in the delivery team and social distancing measures had a profound impact on the T&R programme, preventing the delivery as originally planned. This was despite the intervention and its partners demonstrating considerable flexibility and adaptability in their approach, an important strength in its implementation. One of the main challenges identified in relation to engaging children during COVID-19 was their (typically limited) access to technology, devices and data. However, recruitment was still effective. Family Support targeted schools according to levels of deprivation and exclusion and developed strong relationships with them to facilitate referrals that were generally considered appropriate. It was clear that demand for the programme was considerably higher than could be met within the commissioned level of places.
In the feasibility study, the small number of parents and children interviewed identified benefits and improvements that they attributed to the programme. These included supporting the transition to secondary school; improving behaviour; children becoming aware of topics linked to bullying, gangs, knife crime, self-esteem and careers; and becoming more confident in expressing themselves. Overall, interviewees were positive about their experiences with the intervention. Evaluation participants’ suggestions for improvement showed that they had faith in the intervention. However, they wanted it to last longer, and parents wanted to feel more informed about the activities being run with their children.
The pilot study suggested that programme retention was satisfactory, with the programme reporting that 60% of families completed the intervention. However, data completion rates were low, with less than half of families (49%) providing data six months into the programme. Despite changes that had to be made for COVID-19, the programme was broadly implemented as initially intended and in close alignment with the logic model. The evaluator also judged that families found practitioners to be approachable, accessible and trustworthy. Indeed, when interviewed, the few parents and children described having straightforward and easy access to practitioners, building relationships and bonds with practitioners and they emphasised the merit of a whole-family approach. However, it was clearly difficult for intervention staff to gain participant engagement in completing the core evaluation measures, particularly from parents/carers.
One change that was made during the feasibility phase was to remove the Brain in Hand, CBT-informed, personalised phone and iPad support from the programme. This specific element of the project was not widely used, particularly by younger children who did not generally have access to mobile devices. Similarly, schools would not permit the use of mobile phones during school hours. In terms of the core outcome measures, for children, the reported changes over time across various measures of behaviour and family functioning were positive. Very few parents completed the core measures, as outcome measurement collection was severely hampered by the COVID-19 pandemic. This evaluation was designed without a counterfactual group, so we cannot be confident that improvements reported by children were solely a result of the T&R programme.
The evaluators judged that the T&R programme was not ready for a randomised controlled trial. Several issues would need to have been addressed before proceeding to further evaluation, most notably whether data collection could be improved. Although the pandemic impacted the quality of data collection, even in these circumstances, the level of measure completion in this evaluation was low. YEF is, therefore, not currently proceeding with any further evaluation of the T&R programme.