Reporting and appraising research: a cautionary tale

Posted on by BMJ

Substituting various fats for carbohydrates or saturated fat: an uncertain recipe missing quantitative context and a cautionary example of reporting and appraising research

Guest Blog Post
Author: Martin Mayer, MS, PA-C
Institution: Department of Physician Assistant Studies, East Carolina University
Email: mayerm@ecu.edu

Broadly speaking, science is a way of thinking that involves asking answerable questions about phenomena and then systematically and impartially pursuing means to reduce uncertainty about the answer as much as possible. During the pursuit, findings must always be appropriately contextualized to avoid inaccurate, disproportionate, or otherwise mistaken interpretations, as such mistaken interpretations run contrary to the raison d’être of scientific inquiry. Unfortunately, confusion about and mistaken or overreaching interpretations of research abound.

A recently-published article investigating various patterns of fat intake on total and cause-specific mortality1 speaks to the above and will add tangibility to the above considerations; it therefore serves as an instructive example to be considered in some detail, but the concepts considered herein are certainly more broadly applicable.

NUTRITIONAL RESEARCH AND BASIC PRINCIPLES OF RESEARCH METHODOLOGY

Nutritional studies are often plagued by methodologic shortcomings that preclude strong knowledge statements and contribute to implausible results.2,3 Perhaps most bothersome is the lack of methodologic rigor required to start making causal inferences about dietary patterns or interventions, and better designs do seem feasible with proper design and sufficient infrastructural support (including, importantly, funding).2,3

There have been reproachful whispers of “methodolatry” with respect to appraisal of research, and some champion observational data as reflecting “real-world” data; nevertheless, well-designed, well-executed randomized controlled trials (RCTs) are undoubtedly the most reliable method to assess interventional effects or cause-and-effect relationships. Due to inherent methodological limitations, observational data are typically unable or less able to provide such insight, though Hill’s classic criteria offer foundational considerations for the degree to which observational data can begin to facilitate or permit causal inferences.4,5 For instance, there will never be an RCT of smoking and lung cancer, but observational data make this causal link abundantly clear; however, such instances of observational data clearly demonstrating a causal relationship are decidedly uncommon.

Still, good observational data do have value, and to the extent people blindly view the RCT as a sacred cow of epistemology (e.g., not applying the same degree of critical appraisal to RCTs as one would observational studies, failing to consider a given RCT within the broader context of what is known about the topic at hand [greatly simplified, this latter concept forms the basis for the Bayesian notion of priors]), the reproachful whispers of “methodolatry” have considerable credence, elevating them to appropriate admonitions.

WANG AND COLLEAGUES’ STUDY

Wang and colleagues recently published an investigation of intake of specific types of fat and possible associations with total and cause-specific mortality; specifically, they investigated quintiles of intake for specific types of fat and isocaloric substitution of specific types of fat for either carbohydrates or saturated fat at certain levels of energy intake.1 Theirs is among the most recent of many similar studies investigating dietary patterns and patient-relevant outcomes.6-10

Wang and colleagues’ data come from two large and well-known prospective cohort studies: the Nurses’ Health Study (NHS) and the Health Professionals Follow-up Study (HPFS). Follow-up for both cohorts via biennial postal questionnaires exceeds 90% of potential person-time. Wang and colleagues excluded those who did not report information on fat intake, those who reported what they considered to be implausible energy intakes (men, <800 or >4,200 kcal/day; women, <600 or >3,500 kcal/d), and those with a history of diabetes, cardiovascular disease, or cancer. The final sample for analysis had 83,349 women and 42,884 men and amounted to 3,439,954 person-years of follow-up. Dietary intake was assessed with a semiquantitative food frequency questionnaire (SFFQ); the SFFQ asks how often, on average, the respondent consumed a specified portion of food during the preceding year. For all but one survey used Wang and colleagues’ analysis, this was done for 116 to 150 foods. Wang and colleagues also collected detailed information on the type of fat or oil used when preparing food as well as the brand or type of margarines used. A total of nine SFFQ assessments from the NHS and seven SFFQ assessments from the HPFS were included in their analysis.

PROBLEMS WITH PRESENTATION AND INTERPRETATION – A CAUTIONARY EXAMPLE

The large sample and long and fairly complete follow-up are obvious strengths of Wang and colleagues’ study, but sample size and follow-up duration and completeness are not themselves sufficient qualities to establish the reliability or meaningfulness of research; indeed, their study still suffers from typical and important weaknesses inherent to cohort data and questionnaire-based nutritional studies. For instance, the observational design with use of SFFQs in populations that offer only questionable generalizability (e.g., exclusively health care professionals with noteworthy exclusion criteria) leaves much to be desired, and to the extent one might be inclined to point to the frequency with which food surveys or similar methods of dietary assessment are used in nutritional research, this ultimately does nothing to lessen the marked uncertainties and methodological weaknesses inherent in such a strategy. Prevalence is not and never will be a per se indicator of rigor or acceptability, and to argue otherwise is tantamount to argumentum ad populum (argument from popularity). While Wang and colleagues acknowledge weaknesses in their study to a certain extent in their discussion, they still make overly-assertive statements (e.g., “Our analyses provide strong evidence that using PUFAs [polyunsaturated fatty acids] and/or MUFAs [monounsaturated fatty acids] as the replacement nutrients for SFAs [saturated fatty acids] can confer substantial health benefits”1(p1142)) even though they later state “causality cannot be established” 1(p1143) by their study.

Although Wang and colleagues make efforts to provide evidential context for their findings via their discussion of related literature, another prominent weakness of their article is failure to provide appropriate quantitative context for their findings even if one theoretically accepts their findings as being likely reflective of an underlying truth (which must in reality be decided only after careful critical appraisal). This becomes even more problematic due to their repeated statements of “substantial” findings, sometimes also erroneously using causal phrasing (e.g., “can confer substantial health benefits”1(p1143)). Unfortunately, they only report associated relative metrics, which precludes a straightforward quantitative evaluation of their findings and even lends to an exaggerated sense of the findings.

ADDING QUANTITATIVE CONTEXT WHEN WHAT IS PROVIDED IS NOT SATISFACTORY

It is imperative to seek satisfactory appreciation of the quantitative implications of research findings, perhaps particularly when the research does not readily lend itself to such. When it is possible to construct or otherwise establish a reasonable quantitative framework, one can then use this framework as a thought experiment of sorts to help gauge the potential meaning of given findings under the (potentially strong) assumption that the research actually reflects an underlying truth. One can then subjectively levy any weaknesses in the methodology against this “best-case-scenario” framework in an attempt to form a judicious appreciation for the research findings.

Using the total number of deaths and person-years of follow-up in the individual cohorts and pooled dataset, one can derive baseline estimates for rate of death (supplemental file). With these baseline estimates, one can use the hazard ratios from the rightmost column of Tables 2 and 3 in Wang and colleagues’ study to estimate the associated risk of death with isocaloric substitution of a particular fat for total carbohydrates at a particular percentage of energy intake (supplemental file).11 One can use Figure 2 in Wang and colleagues’ study to do the same for substitution of a particular non-saturated fat for saturated fat (supplemental file). Finally, one can then derive absolute risk differences between baseline risk estimates and the dietary-substitution-adjusted risk estimates (supplemental file).

It is not clear why Wang and colleagues did not provide such estimates, and further data or analysis from the authors’ dataset might allow for better or additional estimates than those outlined above and in the supplemental file; in the absence of such, however, it remains important to consider what the reported data might mean on an individual level, and the above approach is certainly reasonable.

CONCLUSION

Wang and colleagues’ study ultimately leaves much to be desired, one should remember they conducted their study in an attempt to help clarify existing uncertainty surrounding this topic, and other research echoes the sentiment of uncertainty.7 So, although sometimes a reflexively-offered sentiment, further – and better – research does seem indicated. Additionally, relative metrics are most useful when appropriately applied to corresponding baseline absolute risks, but relative metrics in isolation are considerably less informative and can contribute to distorted appraisal of research findings. This can be readily appreciated via the supplemental file or by simply considering, for instance, the absolute versus relative difference between 0.5% and 0.25% (0.25% versus 50%, respectively). Relative metrics might also convey important information when pursuing a population-level appreciation of research findings. While this is certainly not irrelevant, clinicians and patients ultimately care most about applying research on an individual level. The additional quantification of Wang and colleagues’ data shows how this can be estimated (at least in the setting of hazard ratios) when estimates of absolute differences are not provided. With specific regard to Wang and colleagues’ study, the estimation of absolute risk differences suggests much less “substantial” findings than what Wang and colleagues’ article suggests even if one thought their findings were valid, and then when one considers the notable weaknesses in Wang and colleagues’ study, the absolute risk differences seem even less “substantial”.

The considerations herein, although important, are but a whisper amidst a roaring literature pertaining to the execution, translation, and application of medical research. Nevertheless, this writing hopefully makes clear the importance of researchers maintaining the utmost care when reporting research, always providing balanced and objective qualitative and quantitative context for their findings; similarly, readers must maintain an exquisitely judicious approach to the appraisal, synthesis, translation, and application of research.

References

  1. Wang DD, Li Y, Chiuve SE, et al. Association of specific dietary fats with total and cause-specific mortality. JAMA Intern Med. 2016;176(8):1134-1145. doi:10.1001/jamainternmed.2016.2417. Epub 2016 Jul 5.
  2. Nissen SE. U.S. dietary guidelines: An evidence-free zone. Ann Intern Med. 2016 Apr 19;164(8):558-559. doi: 10.7326/M16-0035. Epub 2016 Jan 19.
  3. Ioannidis JP. Implausible results in human nutrition research. BMJ. 2013 Nov 14;347:f6698. doi: 10.1136/bmj.f6698.
  4. Hill AB. The environment and disease: association or causation? Proc R Soc Med. 1965;58(5):295-300. PMCID: PMC1898525.
  5. Lucas RM, McMichael AJ. Association or causation: evaluating links between “environment and disease”. Bull World Health Organ. 2005 Oct; 83(10):792-795. PMID: 16283057. PMCID: PMC2626424.
  6. Chowdhury R, Warnakula S, Kunutsor S, et al. Association of dietary, circulating, and supplement fatty acids with coronary risk: a systematic review and meta-analysis. Ann Intern Med. 2014;160(6):398-406. doi: 10.7326/M13-1788.
  7. de Souza RJ, Mente A, Maroleanu A, et al. Intake of saturated and trans unsaturated fatty acids and risk of all cause mortality, cardiovascular disease, and type 2 diabetes: systematic review and meta-analysis of observational studies. BMJ. 2015;351:h3978. doi: 10.1136/bmj.h3978.
  8. Farvid MS, Ding M, Pan A, et al. Dietary linoleic acid and risk of coronary heart disease: a systematic review and meta-analysis of prospective cohort studies. Circulation. 2014;130(18):1568-1578. Epub 2014 Aug 26. doi: 10.1161/CIRCULATIONAHA.114.010236.
  9. Jakobsen MU, O’Reilly EJ, Heitmann BL, et al. Major types of dietary fat and risk of coronary heart disease: a pooled analysis of 11 cohort studies. Am J Clin Nutr. 2009;89(5):1425-1432. doi: 10.3945/ajcn.2008.27124.
  10. Mozaffarian D, Micha R, Wallace S. Effects on coronary heart disease of increasing polyunsaturated fat in place of saturated fat: a systematic review and meta-analysis of randomized controlled trials. PLoS Med. 2010;7(3):e1000252. doi: 10.1371/journal.pmed.1000252.
  11. Altman DG, Andersen PK. Calculating the number needed to treat for trials where the outcome is time to an event. BMJ. 1999 Dec 4;319(7223):1492-1495. PMID: 10582940. PMCID: PMC1117211.
(Visited 32 times, 1 visits today)