Brain zapping, or non-invasive brain stimulation, in all its forms, has hit the mainstream. You can even watch a YouTube video about how to build your very own not-approved-for-human-use transcranial magnetic stimulation machine! Light-heartedness aside, the therapeutic benefits of brain stimulation are regularly highlighted in the news, and the number of medical clinics that specialize in transcranial magnetic stimulation is increasing.
Around since the early 1980’s, transcranial magnetic stimulation (TMS) has become a popular way to investigate brain function and even to treat some brain disorders. Unfortunately, research supporting this ever increasing popularity is beset by studies with small sample sizes and true effects that are often small or nonexistent.
Over the years, various research groups have had difficulty reproducing published results involving transcranial magnetic stimulation. We therefore decided to survey scientists who use TMS to alter the behaviour of the brain’s motor cortex about their own experiences and research practices.
WHAT DID WE FIND?
Our survey of TMS researchers who published papers using the technique revealed that they could reproduce the originally reported effects only about half the time. This does not match up with the published scientific literature, where virtually all publications in this field report positive results. One explanation for this would be a publication bias towards ‘positive’ studies.
We also discovered a worrying prevalence of questionable research practices. These include stopping a study when the results seem ‘significant’, selecting subjects who ‘respond’ to a protocol, or cherry-picking what data to report – surprisingly even just on a ‘gut feeling’. And despite unanimity among respondents that these practices should always be reported in publications, we found that they were not actually mentioned.
SIGNIFICANCE AND IMPLICATIONS:
These results have many worrying implications. They highlight a possible waste of research funds, personnel and resources in a field where insufficient attention is given to ensuring scientific discoveries are robust and reproducible. The increased use of TMS as a therapeutic tool is also worrying because physicians involved in its prescription may be unaware of its limitations, and ultimately patients may not receive appropriate information about a treatment.
While a remedy for these endemic problems has yet to be found, we believe that increasing the sample sizes in TMS studies and eliminating – or at the very least reporting – questionable research practices would be a simple first step towards reproducible TMS research.
PUBLICATION:
Héroux ME, Taylor JL, Gandevia SC (2015). The Use and Abuse of Transcranial Magnetic Stimulation to Modulate Corticospinal Excitability in Humans. PLoS One 10: e0144151.
This rational approach to all science work, especially studies that influence ‘evidence based medicine’ is supported across the board. Unfortunately the logistical and political realities of research in medicine/science often weigh against the research practices recommended in the article. When careers and research funding are dependent on publication, and when publication is influenced by biases towards ‘positive’ studies we immediately have a problem. Convincing funders of the values of larger studies in a highly competitive environment, and addressing personal and professional reasons why researchers may take short-cuts will take time, and in the mean-time the quality of the work done will necessarily be limited.
Thank you for this comment. The dependence of career progression and research funding on ‘publications’ is well recognised. However, there is also now increasing recognition that too much published science is not reproducible, that ‘publication bias’ distorts our knowledge base, and experimental design and analysis can often be improved. We should all be aware of this: government funding bodies, the pharmaceutical industry, and many journal editors are realising the magnitude of the problem and they are doing things to improve what we do. As examples, NIH just announced changes to requirements for grant submissions and the American Physiological Society is working on changes for all its journals. We are in a time of change for the better I hope. Here is a short piece I did covering some of this territory: http://goo.gl/o7qF9O