A graphic that looks at the different factors that can contribute towards ‘bad’ science.
It was inspired by the research carried out by the author for one of his graphics (the aluminium chlorohydrate graphic), where many articles linked the compound to causing breast cancer, referencing scientific research which drew questionable conclusions from their results.
The vast majority of people will get their science news from online news site articles, and rarely delve into the research that the article is based on. This is why Andy thinks it is important that people are capable of spotting bad scientific methods, or realising when articles are being economical with the conclusions drawn from research, and that’s what this graphic aims to do. Note that the graphic is not a comprehensive overview, nor is it implied that the presence of one of the points noted automatically means that the research should be disregarded. The graphic is merely intended to provide a rough guide to things to be alert to when either reading science articles or evaluating research.
You can download the pdf version of the graph from here
Being able to evaluate the evidence behind a scientific claim is important. Being able to recognise bad science reporting, or faults in scientific studies, is equally important.
These 12 points will help you separate the science from the pseudoscience.
1. SENSATIONALISED HEADLINES - Article headlines are commonly designed to entice viewers into clicking on and reading the article. At times, they can over-simplify the findings of scientific research. At worst, they sensationalise and misrepresent them.
2. MISINTERPRETED RESULTS - News articles can distort or misinterpret the findings of research for the sake of a good story, whether intentionally or otherwise. If possible, try to read the original research, rather than relying on the article based on it for information.
3. CONFLICTS OF INTEREST - Many companies will employ scientists to carry out and publish research - whilst this doesn’t necessarily invalidate the research, it should be analysed with this in mind. Research can also be misrepresented for personal or financial gain.
4. CORRELATION & CAUSATION - Be wary of any confusion of correlation and causation. A correlation between variables doesn’t always mean one causes the other. Global warming increased since the 1800s, and pirate numbers decreased, but lack of pirates doesn’t cause global warming.
5. UNSUPPORTED CONCLUSIONS - Speculation can often help to drive science forward. However, studies should be clear on the facts their study proves, and which conclusions are as yet unsupported ones. A statement framed by speculative language may require further evidence to confirm.
6. PROBLEMS WITH SAMPLE SIZE - In trials, the smaller a sample size, the lower the confidence in the results from that sample. Conclusions drawn can still be valid, and in some cases small samples are unavoidable, but larger samples often give more representative results.
7. UNREPRESENTATIVE SAMPLES USED - In human trials, subjects are selected that are representative of a larger population. If the sample is different from the population as a whole, then the conclusions from the trial may be biased towards a particular outcome.
8. NO CONTROL GROUP USED - In clinical trials, results from test subjects should be compared to a ‘control group’ not given the substance being tested. Groups should also be allocated randomly. In general experiments, a control test should be used where all variables are controlled.
9. NO BLIND TESTING USED - To try and prevent bias, subjects should not know if they are in the test or the control group. In ‘double blind’ testing, even researchers don’t know which group subjects are in until after testing. Note, blind testing isn’t always feasible, or ethical.
10. SELECTIVE REPORTING OF DATA - Also known as ‘cherry picking’, this involves selecting data from results which supports the conclusion of the research, whilst ignoring those that do not. If a research paper draws conclusions from a selection of its results, not all, it may be guilty of this.
11. UNREPLICABLE RESULTS - Results should be replicable by independent research, and tested over a wide range of conditions (where possible) to ensure they are consistent. Extraordinary claims require extraordinary evidence - that is, much more than one independent study!
12. NON-PEER REVIEWED MATERIAL - Peer review is an important part of the scientific process. Other scientists appraise and critique studies, before publication in a journal. Research that has not gone through this process is not as reputable, and may be flawed.