A Review to Settle Which Maths Interventions Help Was Never Finished

What the researchers planned

A Cochrane protocol is a public commitment made before any evidence is read. It fixes which studies will count, which outcomes matter and how the numbers will be combined, so that reviewers cannot later adopt whichever rule flatters the answer they happen to find. In 2016 one appeared for a question with real childhoods attached: which interventions improve mathematics for children who struggle with it.

The question was worth the machinery, and the protocol’s background section sets out why. Between four and fourteen percent of students are identified with a mathematical learning disability, depending on where the line is drawn, and roughly a fifth have weak numeracy, both figures quoted from the existing literature rather than established here. Between 1996 and 2005, the same section records, studies of reading difficulty outnumbered studies of mathematical difficulty five to one, an improvement on the decade before, when the ratio was sixteen to one.

Mairead Furlong and Sinead McGilloway of Maynooth University wrote the protocol with Fergal McLoughlin of Ireland’s Health Service Executive and David Geary of the University of Missouri, and published it in the Cochrane Database of Systematic Reviews. The rules they set were strict. Only randomised controlled trials and high-quality quasi-randomised trials would be admitted, each requiring a comparison group of children who received ordinary schooling, nothing at all, or a placebo. Single-group before-and-after studies were excluded, and so were head-to-head comparisons of two programmes with no untreated group to anchor them.

Every admitted trial was to be graded on whether the children, the staff delivering the programme and the person scoring the maths test knew who had received what. Effects were to be pooled separately for the first six months after a programme ended, and again at one year and at two, so that any benefit which faded would be seen to fade. Eligible children were anyone under eighteen scoring below the twenty-fifth percentile on a standardised mathematics test, which takes in both those with an identified mathematical disability and those at risk of one. Five families of treatment qualified: targeted maths teaching, behavioural and psychological programmes, non-invasive brain stimulation, medication, and combinations of these.

Then nothing. On 29 July 2022 Cochrane amended the record to state that the protocol would not be progressed to the review stage, because there had been no progress in over six years. No trial was ever screened against those rules, and no effect was ever pooled.

What it means for you and your child

Part of why the team wanted rules that strict is set out in the protocol itself. All but one of the earlier reviews they surveyed had combined results across very different kinds of study, including single-case designs, which are noted for producing particularly elevated effect sizes, and the reviews disagreed about whom to count. A field that cannot agree on its population or its designs will not agree on its numbers, and the protocol was written to end that.

That gives a parent something concrete to ask when a programme is described as evidence-based: what kind of study produced the number, and against whom were those children compared. A number drawn from children measured before and after, with nobody to compare them to, is a different object from one drawn from a randomised trial.

What the protocol does report, summarising the field as it stood, is that many randomised trials find targeted mathematics teaching more effective than the comparison for most children with these difficulties, while not working for all of them. The protocol also records, from the same literature, that younger children may benefit more than older ones. What the field had at that point was a tendency across reviews and no rule that would hold for one child.

The plan itself contains the most generous thing in the document. Furlong and colleagues did not intend to ask only whether these programmes work. They built in analyses of whom they work for, by subtype of difficulty, by severity at the outset and by age. Written into the design, before a single study had been read, was the expectation that children would respond differently from one another and that the variation was itself the finding.

This record cannot indicate which programme will suit your child, and it explains why nobody else can indicate it either. The trials exist. The work of sorting them was specified in 2016 and was never carried out.