Blogs

GCSEs & A levels: Reliable to one grade either way?

GCSEs and A Levels Examinations & Qualifications
Ofqual has said that GCSE and A level grades are ‘reliable to one grade either way’. What does this actually mean, and to what extent can we have confidence in examination grades this summer? Dennis Sherwood explains
Image: Adobe Stock

We all know how much every young child cherishes Father Christmas. Sooner or later, though, they must learn the truth.

It is a dilemma for every parent – when to disclose the reality? It is a duty we cannot shirk, for while it may shatter a child’s belief, it would be worse if they heard it from someone else, in the school yard perhaps, and came home with the inevitable questions…

The truth is important but sometimes difficult. It is often easier not to reveal the truth. And if that failure to disclose can be masked by claiming that “it’s in the best interests of the believer not to have that belief threatened”, then that gives the person in authority an excuse…

 

Do you believe in exam grades

What am I talking about? Well, most people believe in their GCSE, AS level and A level examination grades. They believe them to be fully reliable and trustworthy.

But the truth is that they are “reliable to one grade either way” – these being the words of Ofqual’s former chief regulator, Dame Glenys Stacey, at a hearing of the House of Commons Education Select Committee held on September 2, 2020.

Dame Glenys was replying to a question about the reliability of grades (you can see for yourself here – at about 12:19:50). That truth is important, and I would argue we need to discuss the implications openly in education.

Because this truth implies that a certificate showing, say, “A level Economics: Grade B” means that the candidate perhaps merited a B but might have merited a grade C or even a grade A. Perhaps more importantly, it implies that no-one knows for sure.

And the truth is based on the results of Ofqual’s own extensive and thorough research, as presented in their two reports, Marking consistency metrics (2016) and Marking consistency metrics: An update (2018). Below is a graph (figure 13) contained in the 2018 report:

 

As you will see, the caption in the report defines this chart as representing, for each of 14 subjects, the “probability of being within plus or minus one grade of the ‘definitive’ grade, for all grades, for GCSE, AS and A level”.

Importantly, the term “definitive grade” is used by Ofqual to refer to the grade that a script would be awarded if that script were to be marked by a subject senior examiner.

Ofqual, understandably, is careful with its words. I don’t use “definitive” often, so on looking it up to check its meaning, I discovered synonyms such as “most authoritative”, “final”, “cannot be questioned”.

I therefore infer that the “definitive grade” is Ofqual’s benchmark of the “right” grade, the grade the student merits, as based on the performance of the student in that particular exam, sat on that particular day.

The chart shows that, according to Ofqual’s own research, for subjects from maths to psychology, the “probability of being within plus or minus one grade of the ‘definitive’ grade” is 1.00 – or 100%.

This means that it is virtually certain that the grade shown on a candidate’s certificate for these subjects is the “definitive” grade or one grade on either side – hence Dame Glenys’ statement to the Education Select Committee back in 2020 that grades “are reliable to one grade either way”.

For the other subjects, though, the “probability of being within plus or minus one grade of the ‘definitive’ grade” falls short of 1.00 (100%), dropping towards about 0.96 (96%) for English literature, for example. That means that about 4 in every 100 English literature grades, as awarded, are at least two grades adrift from the “definitive” grade.

It might not sound like much, so to make that statistic real let me note that in the summer 2023 exams, 630,261 students in England were awarded grades for GCSE, AS and A level English literature according to the official data from the Joint Council for Qualifications (JCQ). If we calculate 4% this implies that more than 25,000 grades were at least two grades off from the “definitive” grade.

Of course, about half of those “non-definitive” grades would have been higher and so those students will have gotten lucky. But the other half would have been awarded grades at least two grades too low.

This does untold damage since these wrong grades cannot be discovered or corrected by the appeals process which looks only for “marking errors”. I have discussed this conflict within the appeals system in a previous SecEd article on this issue (Sherwood, 2023).

The chart above is the evidence that, at best, grades are “reliable to one grade either way”.

This truth, however, is much denied. For example, an article in 2023, by Mary Curnock Cook, chair of Pearson Education, a trustee of the Higher Education Policy Institute, and former chief executive of UCAS, denounces Dame Glenys’ statement as “erroneous”.

Ms Curnock Cook’s article should be praised for acknowledging the issues with “definitive” grades and responding to the concerns that I and others are raising.

She argues that “definitive” mark is “a proxy ‘correct’ mark used for quality assurance processes only” to ensure that examiners are not marking too harshly or leniently. Instead she says we should focus on the “concept of a legitimate mark”.

She writes: “In many subjects there will be several marks either side of the definitive mark that are equally legitimate. They reflect the reality that even the most expert and experienced examiners in a subject will not always agree on the precise number of marks that an essay or longer answer is worth. But those different marks are not ‘wrong’.”

But if those “equally legitimate” marks lie on different sides of a grade boundary, the grade awarded is the result of the lottery of who happens to mark the script.

As a consequence, we have an exam system where one examiner could give a student a B whereas another might “legitimately” award that same student an A. The A and the B are both “equally legitimate”. But only one of those grades appears on the student’s certificate and the other doesn’t.

The question I ask quite simply is: Should we be happy about that? It seems strange that no-one wants to talk about this. In March, Ofqual’s current chief regulator Sir Ian Bauckham penned a blog entitled Students can look forward to exam results they can trust (2024).

He wrote: “As chief regulator of qualifications at Ofqual I’m committed to making sure that when students open their results in August, they receive qualifications that stand the test of time and fairly reflect their accomplishments.”

Anyone reading that blog will find no indication that grades are anything other than reliable – full stop.

Why are the authorities so reluctant to discuss this issue? Those writing about this are often accused of endangering public confidence in the exam system – which brings me back to Santa Claus and those parents who do not want to shatter their children’s cherished – though false – belief.

The truth is that exam grades are not reliable, full stop, but “reliable to one grade either way”.

 

Further information & resources

Related articles