In this series, I am attempting to elaborate and share what the recipe of test-enhanced learning (more commonly known as retrieval practice), spaced learning, interleaving, feedback, metacognition, and motivation might look like in and out of the classroom.
I am reviewing the research and cognitive science behind these concepts and the modulators underpinning the effective retention of knowledge.
In writing this series, nine clear but interlinked elements emerged. I am considering these elements across nine distinct but related articles:
- Memory (Published: March 30, 2022)
- Repeated retrieval practice or test-enhanced learning (Published: April 6, 2022)
- Spaced retrieval practice or spaced learning (Published: April 20, 2022)
- Interleaving (Published: April 27, 2022)
- Feedback and elaboration (Published: May 4, 2022)
- Successive relearning (this article)
- Metacognition (Published: May 18, 2022)
- Overcoming illusions of knowing or competence (Published: May 25, 2022)
- Testing, motivation and achievement (Published: June 8, 2022)
I would urge readers to also listen to a recent episode of the SecEd Podcast (SecEd, 2022) looking at retrieval practice, spaced learning, and interleaving and featuring a practical discussion between myself, teacher Helen Webb, and Dr Tom Perry, who led the Education Endowment Foundation’s Cognitive Science Approaches in the Classroom review (Perry et al, 2021).
This series, in reviewing the evidence-base, seeks to help you reflect on what will work for you, your classroom, and your pupils. This is article six and it focuses on successive relearning.
Successive relearning
“Practice tests and spaced study are both highly potent for enhancing learning and memory. Combining these two methods under the conditions in which they are most effective (practice tests that invoke successful retrieval from long-term memory and spacing study across days) yields a promising learning technique referred to as successive relearning.”
Rawson et al, 2013
"When you want to learn something that's going to remain with you, that you can retain for the long term, you know one exposure … one encounter with the information is probably going to be insufficient.”
Dr John Dunlosky, Tes Podcast, 2021
“One shot, one encounter with a set of materials isn't likely to promote long-term retention, so we know in order to retain something well it's probably the case that we need to re-encounter, revisit, review the information again and again."
Dr Sean Kang, 2016
Like most educators, I encountered retrieval practice first, before moving onto ideas of spacing and interleaving, before my investigations then led me to the work of Dr Kathleen Rawson and “successive relearning” – specifically the study, quoted above, by Dr Rawson, Dr Dunlosky and Dr Sharon Sciartelli (Rawson et al, 2013)
Conducted in an authentic introductory psychology course, college students learned 64 course concepts using a computerised flashcard program under the following conditions:
- Successive relearning: Students engaged in retrieval practice spaced throughout the semester and they had to retrieve information correctly three times.
- Spaced restudy: Students engaged in restudying spaced throughout the semester (but without retrieval/testing) – arguably the most common approach.
- Baseline: Students engaged in "business as usual" without using the flashcard program.
In two experiments, the successive relearning condition increased students' course exam performance by a letter grade (from C to B) compared to the spaced restudy and baseline conditions. This was consistent for both higher and lower performing students and more robust for three spaced retrieval sessions compared to only one retrieval session.
The terminology was familiar – retrieval, spacing – and the mode of retrieval practice was familiar (flashcards) and I was a teacher who simply wanted that same learning benefit for my own students. What I would later recognise was that this paper introduced three important pedagogical conditions or considerations:
- At what point is something considered learnt or mastered?
- The huge potential of repeated, spaced practice or “successive relearning” as the authors termed it.
- The role of self-regulation (or more to the point the potential pitfalls of students’ self-regulation of metacognitive monitoring – see article seven for more on metacognition).
At this point, I had not connected Dr Rawson's findings with Graham Nutthall's celebrated work, The Hidden Lives of Learners (2007): "Learning is rarely a one-shot affair. Single, isolated experiences seldom give birth to learning. What creates or shapes learning is a sequence of events or experiences, each building on the effects of the previous one.”
Arguably of greatest concern for teachers is that the absolute levels of long-term retention achieved in single-session studies are typically well below the mark to have real-world classroom relevance.
This was the first time that I had read a paper that discussed how we might achieve a level of mastery learning over multiple spaced retrieval sessions – much like Ebbinghaus (1885) reported more than 135 years ago when he discussed memory decay and introduced his “forgetting curve”.
This seemed to be a particularly promising technique.
Defining successive relearning
Successive relearning refers to multiple successful retrievals that are distributed across sessions (Bahrick, 1979).
In particular, successive relearning involves alternating between retrieval practice (with feedback) and study attempts until a certain level of retrieval success has been met, and then relearning that material (again alternating between retrieval and study) in subsequent sessions.
Successive relearning, therefore, combines two highly effective learning techniques that we have discussed previously in articles two and three of this series – retrieval practice and spaced practice – and could leverage the benefits of interleaving (article four) too “under conditions in which they are particularly effective” (Rawson et al, 2013).
To date, only a handful of published studies have investigated the effects of successive relearning on long-term retention (including Bahrick, 1979; Bahrick et al, 1993; Bahrick & Hall, 2005; Dunlosky et al, 2013; Higham et al, 2021; Janes et al, 2020; Rawson & Dunlosky, 2011 & 2012; Rawson et al, 2013 & 2018; Vaughn et al, 2016). For the most part all involve verbal learning tasks and employ the same basic methodology.
During an initial learning session, learners are presented with a list of to-be-learned items for cued recall practice trials followed by restudy as needed until each target is correctly recalled.
This procedure is then repeated in one or more subsequent relearning sessions such that each item is correctly recalled multiple times across sessions.
Long-term retention is typically assessed via a final cued recall test administered days or weeks after the last learning session. Given this methodology, it is very easy to conceive how this might look in a classroom…
Famous last words – research on successive relearning in authentic classroom contexts is limited. But not to be deterred, the laboratory studies demonstrate powerful, long-lasting, durable learning.
To illustrate the potential of successive relearning, consider one noteworthy study – Bahrick et al (1993). Over the course of nine years, they investigated the retention of 300 foreign language translations across several conditions (with different combinations of relearning sessions and spacing).
Each relearning session began with a paired-associate translation test (that is syllables, words, or other items in pairs, presented with one half of each pair to which students must respond with the matching half).
When an item had been correctly retrieved, the card was dropped from practice for that session. When an item had been incorrectly retrieved, they reviewed the item on the back of the card and set it aside for another test later in that session. A relearning session was complete once each of the translations had been correctly retrieved.
Final retention tests were administered one, two, three or five years – yes, years – following the last relearning session. Considering the length of these delays, levels of final retention were impressive. In one trial in which relearning sessions were spaced at a 56-day interval, more than 75% of the items were recalled after a one-year delay, and more than 60% of them were still retained after a five-year delay.
More recently, in two applied research experiments, successive relearning boosted students’ learning of course content by at least 10% (Janes et al, 2020). It was not hard to see the benefits of such durable learning, particularly over an examination period. Even more so when most research papers on retrieval practice favour “near learning”– where tests are taken relatively soon after the intervention.
Other investigations of successive relearning have yielded similarly impressive and “sizeable advantages” for retention after successive relearning as opposed to single-session learning.
Higham et al (2021), meanwhile, reported that recall of course material at the end of the semester was better for successive relearning compared to restudying. And perhaps more importantly, that increased recall during relearning sessions was associated with improved metacognition, increased self-reported sense of mastery, increased attentional control, and reduced anxiety – crucially, the students found successive relearning to be enjoyable and valuable.
Latimer et al (2021) state that retrieval practice and spacing are “best used combined with each other” and suggest that the number of exposures to learning content (initial exposures and training exposures) should influence our spacing schedules.
They concluded that the more learners are tested (exposed), the more beneficial an expanding schedule is compared with the uniform one (we explained and discussed expanding vs uniform spacing in article three on spaced learning).
So, how much successive relearning is enough?
Let’s turn to Rawson & Dunlosky (2011). Across three experiments, 533 students learned conceptual material via retrieval practice with restudy. The items to be remembered were practised until they could be correctly recalled by the students between one and four times during an initial learning session.
They were then practised until students could correctly recall them across one to five subsequent relearning sessions. This was some study – across the experiments, more than 100,000 short-answer recall responses were collected and hand-scored. A consistent qualitative pattern emerged.
The authors found that relearning had pronounced effects on long-term retention but with a relatively minimal cost in terms of the additional practice trials required.
On the basis of the overall patterns of durability and efficiency, Rawson and Dunlosky’s (2011) “prescriptive conclusion” – their words – is for students to practise recalling concepts in the initial learning session to three correct recalls and then to relearn them three times at widely spaced intervals (with a further additional relearning session increasing recall by only 2%).
And it is worth noting that there is one further clear benefit to this approach: “Successive relearning is an important technique not only because of its potent effects on student learning but also because it is a strategy that students can use on their own outside of class with minimal demands on instructors.” (Rawson et al, 2013).
Three – the magic number?
Rawson et al (2018) state that relearning potency is more than just a "dosage effect". Correctly recalling items one time in three sessions versus three times in one session yielded a 262% increase in retention test performance.
This is concomitant with Nuthall (2007) whose research emphasised: "The accumulation of at least three different sets of complete information about a concept makes the difference between a concept that is never quite learned and one that firmly connects to and integrates with previous knowledge, and hence is learned and remembered."
He also states: “First, a student has to make sense of new experiences by relating them to already known concepts and evaluating them against that information. Next, he or she has to hold the new experiences in working memory and to connect and integrate them with related successive experiences … a combination of experiences is needed to produce student learning.”
So, continuing my journey through the research, the need to “connect and integrate” knowledge and Nuthall’s use of the term "successive" now jumped off the page.
Nuthall concludes that a student “understands, learns and remembers a concept if they have encountered all the underlying information three times”.
He adds: “Provided a student is able to piece together in working memory, the equivalent of three complete definitions or descriptions of a concept, that new concept will be constructed as part of the student’s long-term memory.
“Students need time to process new concepts; not simple repetition but opportunities to come at material in different ways.”
And Nuthall’s research is remarkable in that only this “three times” rule has predictive value – and what value: he could predict with 85% certainty which student would correctly answer which question on a test and which student would not, based on their experiences across lessons.
Nuthall also finds that: “Students already know, on average, about 50% of what a teacher intends his or her students to learn” and that “different students will know different things, and all of them will know only about 15% of what the teacher wants them to know” (Nuthall, 2007).
And so: “There’s a need for constant monitoring of students’ understandings as the ways they understand and interact with information depends on their prior knowledge and understandings.”
The Hidden Lives of Learners was published posthumously in 2007. In 2001 Graham Nuthall said: “If we are to understand how teaching relates to learning, we have to begin at the closest point to that learning; and that is students’ experience.”
I have now read The Hidden Lives of Learners three times and I would recommend it to you. For a short overview of his work, see also Jan Tishauser’s 2019 ResearchED article or Tom Sherrington’s helpful blog (2020).
Successive relearning is super-efficient
Absolute levels of retention achieved by successive relearning range from 75% to 91% over intervals of two to seven days and 60% to 76% over intervals of three to four weeks (Rawson et al, 2018).
Our decision as educators then is to decide whether that is high enough? If not, fortunately, the rapid reacquisition or relearning that has been demonstrated (Vaughn et al, 2016; Rawson et al, 2018) provides some assurance that knowledge can be quickly and efficiently reactivated.
Previously, Vaughn et al (2016) demonstrated that successive relearning is extremely efficient with just seven minutes being required to relearn 70 pre-learned word pairs.
Rawson et al (2018) add: “Relearning had pronounced effects on long-term retention with a relatively minimal cost in terms of additional practice trials.”
This area of research is currently being explored by Yan et al, (2020) and Eglington and Pavlik (2020) – it is what is being referred to as the microeconomics of learning.
Dr Eglington told me: “Students have limited time to study, and thus the total time costs is a vital consideration. In fact, we found that practising efficiently had an even larger effect on memory than spacing usually does.”
And as De Jonge (2014) stated: “When time to learn is limited, it matters a great deal how one puts to use the available amount of time.”
So we are talking about the efficiencies of learning, in that:
- Easier items are recalled faster, permitting for more time for further testing (yes, this provides less learning per item than for challenging items).
- More practice leads to faster responding, – which leads to more time for further testing.
- Harder items may provide more learning, but they take longer (especially if the student fails to remember), thus reducing the time students have for further testing.
- Harder items are more likely to be answered incorrectly, which is frequently more time-consuming due to reviewing corrective feedback, thus resulting in less time for further testing.
Eglington and Pavlik state: “In short, determining the optimal practice schedule is a fundamentally economic problem: find the maximally efficient level of difficulty that provides the most gains in learning per unit of time.”
Their conclusions include that the “optimal difficulty” or “desirable difficulty” is “less than typically thought”. Success rates of 80% led to “40% more items recalled” in the trials than the next best conventional schedule.
Further, existing recommendations for ideal spacing schedules are insensitive to time costs. Simply, “practice difficulty” is highly relevant to optimising practice scheduling, even more so when time is limited – such as before an forthcoming exam.
Related to this, what Yang et al (2021) terms “repeated retrieval” presented learning gains that increased when items were quizzed once (0.44), twice (0.60), three times or more (0.64), and unlimited attempts (0.76).
This demonstrates that the benefits of repeated testing and the efficiency of successive relearning should be considered together, alongside the differing status of “retrievable” and “non-retrievable” knowledge and factors such as item difficulty and the role and time-cost of feedback.
Self-testing
Demonstrating meaningful improvements in course exam performance and on long-term retention tests, Rawson et al (2013) reported that these gains were attained in both supervised and unsupervised conditions.
The inference being that successive relearning could also be implemented outside the classroom as well. In fact, Vaughn et al (2021) reported that low-cost self-testing websites garner widespread use among college students and that this is associated with increased exam performance. Finally, and perhaps most importantly, self-testing could "potentially improve student learning without sacrificing class time”.
Armed with a knowledge of human cognitive architecture (memory – see article one), successive relearning makes sense – successive relearning is essential for long-term maintenance of knowledge and much more closely resembles authentic classroom teaching, learning and relearning.
As teachers we will all recognise the common sense summary appraisal offered by psychologist Harry Bahrick (1979) and I will end this article with his words: “Much of what is learned during a first exposure is forgotten during the interval between exposures and must be relearned after to become a part of semi-permanent knowledge. One can, therefore, conceive of the total acquisition process as a cycle of acquisition, loss, and reacquisition of information.”
Top tips for using successive relearning
So in practical terms, to embrace the use of successive relearning, we must alliterate: Retrieve – Reflect – Reveal – Repeat – Reorder – Refresh. By which, I mean we must ensure that:
- During initial learning, we repeatedly expose learners to retrieval. Reordering cues offers a minimal, additional difficulty.
- Moving to relearning, ensure learners have had time to forget what was initially taught.
- Learners experience multiple relearning opportunities, repeat or revisit the knowledge. Either reordering or refreshing the cues/flashcards depending on success rates.
- Retrieval is “successful” – or at least that there are low failure rates.
- We use “free recall” quizzes rather than “recognition recall” (see article two).
- Learners are encouraged to elaborate on or connect their thinking with what they already know.
- The quizzing activity is for learning and therefore “low stakes” (in that results are not used as part of a formal summative assessment). It may or may not include recording results if only for informative purposes.
- We use self-assessment to leverage the benefits of metacognitive monitoring (see article seven).
Takeaways
- Much of what is taught is quickly forgotten – and so needs to be relearned.
- A student “understands, learns and remembers a concept if they have encountered all the underlying information three times” (Nuthall, 2007).
- Successive relearning combines two highly effective learning techniques – retrieval practice and spaced practice.
- Successive relearning involves alternating between retrieval practice and study attempts until a certain level of success has achieved and then relearning that material in subsequent sessions.
- Practise recalling concepts in the initial learning session to three correct recalls and then relearn them three times at widely spaced intervals (Rawson & Dunlosky, 2011).
- Don’t forget the importance of feedback after retrieval attempts (see article five).
- Relearning is potent, durable and efficient: an excellent lesson or homework task (not just lesson starter tasks either).
- Relearning is equally effective in supervised (classroom) and unsupervised (homework?) conditions.
- Successive relearning is so much more than just increased recall. It also supports metacognition (see article seven) and motivation (article nine).
- And if you only have time to read one paper on this topic: The benefits of successive relearning on multiple learning outcomes (Higham et al, 2021): http://repository.essex.ac.uk/30280/
- And if you’re lucky enough to have the time to read two papers, then the Rawson et al (2013) study – The power of successive relearning: Improving performance on course exams and long-term retention – completely redirected my thinking when it came to retrieval practice: www.jstor.org/stable/43546826
- Kristian Still is deputy head academic at Boundary Oak School in Fareham. A school leader by day, together with his co-creator Alex Warren, a full-time senior software developer, he is also working with Leeds University and Dr Richard Allen on RememberMore, a project offering resources to teachers and pupils to support personalised spaced retrieval practice. Read his previous articles for SecEd via https://bit.ly/seced-kristianstill
References: For all research references relating to this article, go to https://bit.ly/3ytHqs8
Acknowledgement: This article would not have been possible without the cognitive challenge and support of Kristian Shanks, an assistant headteacher for teaching and learning at a large secondary comprehensive. I often find myself referencing our conversations and his curriculum insights and expertise have led to significant design considerations for RememberMore as well as resources now being accessed by teachers daily – thank you.
RememberMore: RememberMore delivers a free, personalised, and adaptive, spaced retrieval practice with feedback. For details, visit www.remembermore.app or try the app and resources via https://classroom.remembermore.app/