in a Uni Connect area . However , unlike the group of learners from Uni Connect areas , the matched counterfactual group is not unique , because it is selected by random sampling ( with replacement ) from the much larger population of learners from non-Uni Connect areas ( see Table B2 ). As a result , if only one random sample is taken , there is a risk that it happens to be an ‘ unusual ’ group of learners , who are not typical of the population as a whole .
6 . To mitigate against this , we took 1,000 random samples of learners from non-Uni Connect areas , such that selected learner living in a non-Uni Connect area matched with one other unique learner living in a Uni Connect area in each sample , based on the characteristics described previously . 44
7 . Our analysis was then conducted 1,000 times , by separately comparing the application outcomes of each matched counterfactual group with the same unique group of learners from Uni Connect areas every time . The full range of results from all 1,000 analyses are presented in this report . This approach gives us confidence that the results we are seeing are not simply by random chance .
8 . In practice , within each of the 1,000 random samples , roughly 20 per cent of unique learners from non-Uni Connect areas are randomly selected and matched with learners from Uni Connect areas in each cohort . Around 15 per cent of those selected appear more than once in each matched counterfactual group . The datafile associated with this report contains details of the sampling rates for each of the 1,000 matched counterfactual groups . 45
Minimising spillover effects
9 . Learners from the same school or college as those targeted by the programme , but who are not living in a Uni Connect area , may also benefit from outreach activity . In the matched counterfactual analysis , we have sought to minimise these spillover effects . We did this by excluding learners who were living outside Uni Connect areas but were known to have attended a school or college which was engaged by the programme in at least one of the years these learners attended . These schools and colleges were identified in one of the following ways :
a . They were listed in the December 2017 partnership monitoring return to the OfS .
b . They were listed in the winter 2018 partnership monitoring return to the OfS , as schools or colleges who ‘ are or will be in receipt of activity ’.
44
Sampling was done with replacement because there were only a limited number of learners from non-Uni Connect areas who were eligible for matching . Sampling without replacement could therefore have resulted in selecting the same handful of eligible leaners from non-Uni Connect areas in each of the 1,000 repeated samples , which would ultimately understate the variation in the distribution of estimates . Sampling with replacement also had the benefit of allowing us to use an established method for estimating statistical uncertainty , namely ‘ bootstrapping ’. Although sampling with replacement will sometimes mean the same individual is selected more than once within a given sample ( roughly 15 per cent of learners in each matched counterfactual group ), sufficient variation should be achieved if enough resamples are taken .
45
Available at www . officeforstudents . org . uk / publications / uni-connect-national-evaluation-updated-analysis /.
63