Limitations of data
Please note: this section of the toolkit was developed in connection with commitments in the Access & Participation Plan (2020-25), which included commitments to address awarding gaps affecting Black British students and students with declared mental health conditions.
Course / Tripos data
Given the current focus on the awarding gaps by ethnicity and disability groups, it is understandable that individual courses might want to examine their performance in those areas and assess their contribution to the overall University goal of reducing the gaps.
However, unlike the course-level work on gender gaps, where the datasets are much larger, there are constraints on how ‘granular’ we can get with quantitative analyses of examination data for either Black students or students with declared disabilities, particularly mental health conditions. The key limitation at present is student numbers
While it may be possible to observe gaps for Black and disabled students in a particular course on the Tableau awarding gap dashboards, the small numbers of students in many courses means that they are not always suitable for statistical analysis.
Fluctuations
Additionally, as a result of the small sample of students identified as Black or with a declared mental health condition for most historic academic years in the Tableau dashboards, some degree of year-on-year fluctuation and occasional outlier results are to be expected. Just one student getting a different outcome might result in that group’s proportion changing by over 2%.
Next steps
It should be noted that limitations of the available quantitative data do not mean it is unimportant for courses to pay attention to the available data about patterns of differential attainment for their students. Nor should they conclude that they do not have a responsibility to consider potential awarding gaps because there is no dataset for their course about a protected characteristic due to currently small student numbers.
This aligns with the conclusion of a Higher Education Funding Council for England report, Causes of Difference in Student Outcomes (Mountford-Zimdars et al 2015), in which some frustrations were acknowledged over the limitations in the data capturing underlying individual characteristics, or in identifying causes for awarding gaps, which commonly results in questioning of the validity of the evidence or conclusions on causation. However, others in the study noted that they did not need more and increasingly sophisticated multivariate statistical data, pointing out the the simple observation of differentials in progression itself was a sufficient 'cause' or mandate for requiring action and initiatives (p24).
We know that we have persistent awarding gaps that need to be addressed by the collegiate University. While more data is going to be valuable for a deeper understanding and monitoring of patterns of student attainment across the University, we can start work now to develop action plans based on recommendations from across the sector and from our qualitative research findings.