Annex D : Matched counterfactual analysis
Technical description
1 . The matched counterfactual approach involves comparing two equally sized groups which are forced to have an identical mix of certain characteristics . In this case , we compare all learners from Uni Connect areas with another equally sized group of learners from non-Uni Connect areas . This second group of learners from non-Uni Connect areas is carefully chosen to match the original group on a pre-defined set of characteristics . This matched group is then known as the ‘ matched counterfactual ’, because it represents a hypothetical situation where learners from Uni Connect areas had instead come from non-Uni Connect areas .
2 . This matched counterfactual group was created by randomly sampling ( with replacement ) from the population of learners from non-Uni Connect areas . This was done such that each learner from a Uni Connect area matched one other learner from a non-Uni Connect area in the same cohort on the following characteristics : their number of GCSEs at grade A * to C ( or 9 to 4 ), whether they achieved a standard pass in GCSE English , a standard pass in GCSE Maths , their sex , their ethnicity and their free school meal status . Matching in this way meant there would always be the same number of learners from Uni Connect areas and non-Uni Connect areas within each combination of the characteristics listed above . In other words , both groups were guaranteed to have the same mix of these characteristics .
3 . The key difference is that one group was living in Uni Connect areas in Key Stage 4 , while the other was not . This should allow for a fairer comparison of outcomes between these two groups over time , which can begin to shed light on the impact , if any , of the Uni Connect programme . Of course , there will remain other differences that are not possible to account for , such as the amount of support each learner received from their school or family . If these unobserved differences in characteristics between the two groups change over time , this will distort our understanding of the impact of Uni Connect programme . There will also be within the categories of matched characteristics , such as the exact GCSE grades achieved by each learner beyond the number of ‘ standard passes ’.
4 . The choice of the two groups was determined as follows . Because there are far more learners from non-Uni Connect areas , 99.9 per cent of learners from Uni Connect areas had a combination of characteristics which could be exactly matched with at least one learner from a non-Uni Connect area , meaning only 0.1 per cent of learners from Uni Connect areas had to be discarded for this reason . A further 0.6 per cent of learners from Uni Connect areas were discarded because , although there was at least one learner from a non-Uni Connect area with the same mix of characteristics , there were enough to match each learner one-to-one . We would otherwise have been forced to sample some learners from non-Uni Connect areas more than once , which would have artificially reduced the sampling variation and resulting estimates of statistical uncertainty . This means we created a single unique group of learners living in Uni Connect areas for each cohort of school leavers , for which there were at least as many learners in non-Uni Connect areas with each unique combination of the matching characteristics .
5 . Similarly , 0.8 per cent of learners from non-Uni Connect areas were also discarded , since they held a combination of matching characteristics which was not held by at least one learner living
62