DORE breakthrough published in, ummm ‘Leamington Courier’

DORE have chosen to distribute their news of a ‘Major Autism Breakthrough’ via the doubtless-estimable Leamington Courier.

Now, I’m all for the public understanding of science, and particularly advances in educational psychology, or I wouldn’t be writing this blog.

However, publishing science by press-release to local papers first is Not On, particularly when said science could influence desparate people to hand over a lot of money to you in the hope of a ‘cure’.

It’s not OK to say ‘The Dore Clinic has achieved massive successes while working with 1,000 patients suffering from the symptoms of high-functioning autism’, when the article goes on to say ‘In addition 56 people who had been formally diagnosed as suffering from autism have now completed the programme’.

So, how many people with an actual diagnosis of Autistic Spectrum Disorder (ASD), given by someone qualified to do so, were actually helped? There’s a lot of difference between the reliability of research on 1000 people, and on 56. If your child is diagnosed with an ASD, then the results of the 56 are a lot more relevant than 1000 people who may or may not have had some ASD traits, which are not uncommon within the general population and particularly amongst people with PDDs.

How were these people helped? We don’t actually know. The paper doesn’t seem to have been published in a journal, or made available on a DORE website, as far as I can tell (I’d love a copy when it does show up). What does ‘expressed emotion’ mean in practice, and how do you measure it improving?

‘Of this number 100 per cent showed improvement across a battery of cognitive, literacy and motor tests’ looks good but doesn’t mean much. If everyone scored higher on standing on wobble-boards & throwing beanbags post-DORE, that would count as an (unsurprising) improvement – but it would not necessarily help people much in their everyday lives. We also don’t have any demographic data – if most of the group are young, you would expect them to get better at things over the 18 months – 2 years DORE usually seems to take.

We have no information on drop-out rates. This is important, because people who don’t see an improvement will be more likely to drop out. If half of people who perform below average (so 50% of the sample) drop out, overall scores will improve – even if any ‘improvement’ or ‘getting worse’ was completely down to normal variation. It depends on how they’ve done the stats, but if you assume that the most severely affected to start of with will be more likely to drop out (& it’s not hard to see that someone with particularly problematic ASD symptoms might be less able to keep up regular daily exercises), and their scores are included in the initial but not the final groups, this makes matters even worse. This is why publishing your data & stats is important. (BTW, does anyone know drop-out rates for DORE? Unofficially I’ve heard 50%, but I’d appreciate info).
Again, this study does not seem to have a control group – so again you can’t tell what was just normal progression with time, what is non-specific effects which could be obtained by telling the family to pat their head & rub their tummy for 15 minutes twice a day as opposed to anything to do with DORE or the cerebellum specifically. ‘Doing something’, giving a child lots of attention, raised expectations, etc, will all have very real effects on a person & their family. We also don’t know what else they’ve done. I was interested to find that it’s not uncommon for DORE to suggest 1:1 more conventional tuition alongside the programme – so looking at studies of people in the ‘real world’ doing DORE then the question isn’t so much ‘How does DORE compare to doing nothing’ as ‘How does DORE & extra 1:1 help compare to doing nothing?’.

The point of peer-review is that experts in the field can look at a paper and correct mistakes before it goes out for public consumption. The point of publishing your full methodology & results is so that people can read them for themselves and see what you did, decide if you did it right, and maybe do it again to see if the same thing happens. Publishing unreviewed press releases with hyped claims is irresponsible. Not telling people exactly what you have done, what you found, and how you measured it, doesn’t allow them to weigh up the research for themselves.

Incidentally, some of the language used in the newspaper article is – rather outdated & patronising to say the least. I’m hoping that’s due to staff on a local paper not being used to covering disability-related stories, but ‘high-functioning autistic cases’ don’t exist – people with ASD do. They’ve also managed to mention ‘suffering’ seven times in a short article. I would strongly suggest a look at the NAS on ‘What (not) to say about autism’.

National Autistic Society’s rather noncommittal response to DORE .

I’ve heard of ‘publish or perish’, but this is getting daft. If the research was as good as it says, Nature Neuroscience at the very least would be falling over themselves to get their hands on it. It would be huge & deeply exciting news. The front pages of broadsheet papers would doubtless be devoting many inches. So why do DORE feel they need to publish this first in an obscure local newspaper, if the research is that good or the findings that exciting?