Yesterday a new article was published in the prestigious Journal of the American Medical Association, JAMA. As a statistician, I find the fact that this article is published in such a prestigious journal in its current format somewhat surprising and definitely disturbing.
The key result that is noted in the article is the fact that male physicians taking multivitamins seem to develop less cancer than similar male physicians taking a placebo. In the multivitamin group, 1290 of 7317 (17.6%) subjects developed cancer, compared to 1379 of 7324 (18.8%) subjects in the placebo group. The difference is 1.2%, with a 95% confidence interval of [-0.05%, 2.45%] (note that my simple calculation of the difference in proportion is not statistically significant, compared with the survival analysis in the manuscript).
The issue I have with the paper is one of multiplicity. A simple comparison with the planned objectives and analyses of the study in clinicaltrials.gov indicate that there were 3 primary objectives of this study. There is additionally 4 secondary objectives stated, one of which is whether multivitamins reduces the risk of cancer. It seems fair to conclude that since the publication is addressing one of the secondary endpoints, then all seven outcomes were considered in the study. It is not entirely clear from the publication how many of the seven outcomes were statistically significant at the 5% level. The stated p-value is 0.04 for the main analysis in the publication. This must be interpreted in the light of all of the planned analyses in the study. A p-value of 0.04 is an indication that the probability of seeing this result if there is no actual difference between treatments is one in twenty-five. However, if there were 7 such analyses, then the probability of observing at least one result as impressive as that reported would be much less, possibly as low as one in four (i.e. quite likely).
To make matters worse, the authors have also reported adverse events based on statistical significance tests. I assume with such a large group of patients that there would have been a multitude of different types of adverse events reported (100s?). Were all of these different event types tested? In that case, it would be expected that you would find many statistically significant differences, just by chance (a significant result is only an indication that you would find such an extreme observation one in twenty times).
It is clear that this publication would not have demonstrated a clinically significant result if the fact that the study had so many objectives was taken into account. As such, the conclusions that the study makes should not be conclusive, and in my opinion, should not encourage anyone to take multivitamins to avoid development of cancer. I would however, highly recommend this paper to those in universities teaching statistics as a case study in what not to do.