Some even most (depending on how you measure it) of what doctors do lacks strong evidence. Even when evidence exists it often doesn’t seem to be relevant to doctors—because their patients or their circumstances are so different from those in the trials that produce the evidence. This is especially true in low and middle income countries as most evidence is produced in high income countries. Understandably doctors grew wary of evidence from trials and make decisions on other grounds, particularly their experience. The result is that patients may suffer from not receiving effective treatment or receiving treatments that do more harm than good. Could this sad state of affairs be improved? A group of clinicians, researchers, evidence buffs, policy makers, and odds and sods (me) met in Bellagio in Italy last week to think about how. Most were from Africa.
How poor is evidence for practice?
Is the evidence behind healthcare really so poor when billions have been spent for decades on medical research? The weakness of the evidence became apparent when some 20 years ago people began to systematically examine and assemble evidence to answer clinical questions, produce guidelines for clinical practice, and assess the evidence behind health technologies. (An aside: when they see the word technology people automatically think about machines and computers, but health technologies include everything in healthcare, including policies, who does the work, how it is done, the training they receive as well as drugs, devices, operations, everything. It’s a very broad term.) Much of the evidence comprises studies that are uncontrolled, too small and short term, too biased, and too badly reported to be useful. Randomised controlled trials are essential for knowing the effectiveness of an intervention, but for many interventions they are either lacking or too poorly done to be useful to doctors treating patients.
So a study of 15 guidelines in cardiovascular medicine, which is probably the best researched part of clinical practice, found that 30-91% of the recommendations lacked good evidence and that in more than half of the guidelines over half of the recommendations lacked good evidence. A review of reviews of different areas of medicine found that overall only 19% of common clinical decisions were supported by randomised trial evidence, with a wide range from 2% to 53%. Most of these studies were from high income countries, but in a set of guidelines developed for use in primary care in underserved areas in low and middle income countries most of the 6000 decision points lacked good evidence.
The applicability of evidence for practice
The problem of evidence to support clinical practice in low and middle income countries is worse than in high income countries because some 90% of evidence has been produced in high income countries in conditions very different from those that prevail in low income countries. So even when the evidence is good there is a second question about whether it is applicable. The disease patterns, the patients, the availability of drugs and equipment, the capabilities of the clinicians, and many other factors may be so different that the high quality evidence seems useless to those treating patients. Or the evidence may be indirect in that it uses surrogate outcomes or comprises studies of Drug A versus placebo and Drug B against placebo when the doctor’s task is to choose between A and B.
The meeting heard of the applicability problem with evidence for treating children in Kenya with severe pneumonia. Three well done trials had found that oral penicillin was just as good as injected benzyl penicillin, but the trials were done mostly in Asia and none of the patients died which led Kenyan paediatricians to doubt the applicability of the evidence to their patients where mortality was high. Plus there was the worry that if sick children were being given oral rather than injected drugs the mothers would take their children home from the hospital. So the Kenyan paediatricians conducted a trial within routine practice in six Kenyan hospitals and found that oral amoxicillin did work as well as injected benzyl penicillin. As a result Kenyan guidelines were changed.
There is a feeling, however, that doctors may be over conservative when assessing applicability of research. Rather than ask “Is this evidence applicable to my patients?” it may be better to ask “Is there any compelling reason for thinking that my patients are so different from those in the trials that the intervention would work differently?”
One of the biggest worries around applicability has to do with most trials for new drugs being efficacy trials—that is, trials conducted in ideal conditions. These are the trials that drug companies have to do to get their drugs onto the market. They answer the question “Can this drug work?” not “Will the drug work in my patients, who may be much older than those in the trials and have many other conditions, in my far from ideal conditions?”
So those at the Bellagio meeting agreed that there is a problem. A starting point for some at the meeting involved in using evidence to produce guidelines and policies was that it didn’t make sense to keep producing these guidelines and policies when underlying evidence was so weak without trying to find a way to produce more high quality, applicable, direct primary evidence.
How to produce better evidence?
The current research system doesn’t provide an answer to the problem of producing more high quality, applicable trials. It is geared to produce long term, high quality, and expensive trials driven by researchers not clinicians. Such trials are needed, but they need to be supplemented by trials produced in routine care answering practical questions asked by clinicians—or, to put it another way, they must produce evidence that is applicable and will convince clinicians to base their practice on the research.
The need for more “pragmatic” trials
Everybody agreed that we need more “pragmatic” trials, but what is a pragmatic trial? It’s a term first used in 1967, and in essence it’s a trial designed to help with decision making—should we use this drug in this patient or do these guidelines improve patient care? Pragmatic trials are on a spectrum with “explanatory” trials at the other end, and explanatory trials are designed to show a causal relationship between an intervention and an outcome—does this drug work in reducing, for example, blood pressure in ideal, highly controlled conditions?
At the pragmatic end of the spectrum, the trial is conducted in routine practice, eligibility criteria are wide, the organisation is simple, as few extra data as possible are collected, and the outcome measures used matter to those who part in the trial, both clinicians and patients. The hope is that the clinicians or policy makers will accept the results of the trial and act on them.
Some seemed to wonder if these were dumbed down, low quality trials. But they aren’t. They can be designed to have high “internal validity” (the question is answered with high confidence) and “external validity” (the answer applies in a wide set of circumstances).
So how to achieve more pragmatic trials conducted in routine conditions, answering questions that matter to patients, clinicians, and policy makers that will be acted on? And how to have these trials done rapidly and at low cost?
The first answer for producing more pragmatic trials
The organisers of the meeting arrived with a clear idea of the problem, although it took others a while to grasp it, but they had much less idea of the answer. There was a half formed (half baked even) idea of some sort of organisation that would collect research questions, prioritise them, support getting them done (with advice, training, and even money), and try to ensure uptake of the results.
A better answer: “a learning health system”
But during the meeting a more practical, exciting, and scalable answer arose—creating learning health systems that would conduct research, including pragmatic trials.
A learning health system sounds like empty jargon until you begin to unpick what it is, and luckily these beasts already exist. A learning health system is one in which patients and providers work together to choose care based on best evidence. It drives discovery, but is a natural outgrowth of patient care. It ensures innovation, quality, and safety, and does this in real time.
The meeting heard about ImproveCare, a learning health system set up in the US. It began in 2007 with eight paediatric gastroenterology practices coming together to improve the care of children with inflammatory bowel disease. They defined an outcome of remission. Agreeing the outcome was not easy and took six months. Next they created a registry and collected routine data, a process made much easier by the appearance of electronic health records. Clinicians may be sceptical about research, but they love seeing data on their patients. The data unsurprisingly showed wide variation in remission rates varying from 40% to 60%.
ImproveCare sought to standardise care, engage patients and their families in coproducing care, and introduce a series of improvements. Over seven years the number of patients in remission each month improved from an average of 50% to 80%. This was without any new treatments, but simply through standardisation, feedback of data, and cycles of improvement.
After several years the network began research. One practical question was whether Infliximab, an expensive biological drug, would improve remission rates better than less expensive conventional treatments. Trials done with adults in ideal circumstances showed it would, but would these results apply to children? Using the database those in the network could produce a “pseudotrial” by identifying patients who began on the new drug at different times. The result, a 50% improvement in remission rates, was almost identical with those in the ideal trial. This study was done much more cheaply than randomized trials and reassured clinicians, patients, and families that the drug worked well for children in the real world and (a judgement) justified the high cost.
Based on the success of the ImproveCareNow, PEDSnet was formed to scale the learning health system model across the US. PEDSnet includes nearly 5 million children and is conducting several pragmatic trials. It is part of a wider network, PCORnet, that includes several other networks like PEDSnet. It provides care to 75 million Americans and is an unsung benefit of Obamacare.
Components of a learning health system
The “secret sauce” of a successful learning health system is the platform, but a successful system has six components. The first is “community,” which can be hard to define, but ideally will include clinicians, patients, researchers, improvement specialists, information technology specialists, managers, and policy makers. For success at least some from the individuals within these groups must become emotionally committed. Ultimately there must be value for all participants or they will drop out.
Secondly, there must be a focus on outcomes. The learning health system must produce better outcomes for patients. If it doesn’t it will—and should—fold.
The third requirement is a common dataset that is as simple as possible, and data should be entered only once. Extra data might be collected for particular studies, but in the spirit of pragmatic trials thus should not interfere with routine care.
Quality improvement is the fourth necessity, and it was the usual methods of quality improvement that allowed the large improvement in the care of the children with inflammatory bowel disease. Researchers and quality improvement practitioners are often suspicious of each other: the researchers suspecting quality improvement practitioners of evangelism and lack of rigour, and the quality improvement practitioners accusing researchers of going too slowly and being satisfied with small improvements. But they should complement each other: researchers showing what works, and quality improvement practitioners ensuring the innovations reach everybody.
Fifth comes pragmatic research, and the research is one of the components that moves a quality improvement network into a learning health system.
Lastly comes governance, and good governance will ensure a voice for all those in the system, particularly patients.
Is a learning health system possible in Africa?
But could a learning health system work in a low and middle income, particularly African, country? The conviction was that it could. All learning health systems start from different places and proceed at different rates in different ways. Nascent platforms were identified in Kenya, Malawi, Zambia, and South Africa, and interrogation of leaders of the nascent platforms in Kenya and South Africa made those at the meeting think it could work in those two countries at least.
This answer to the problem of too little evidence for clinical practice, particularly in low and middle income countries, has distinct advantages over the alternative half formed in the minds of the organisers. There’s a clear route forward. It builds on what already exists and is scalable. It concentrates on outcomes and includes all players, including clinicians and patients. The research will happen in a context that allows it to be quickly implemented, and there is every chance that research could be done quickly and cheaply in a highly pragmatic way.
The learning health system also avoids the nagging doubt of people like me that the best way forward to improve the health of the world’s people would be to implement what we already know for sure—like secondary prevention of cardiovascular disease, where less than 10% of people globally get evidence based treatments that have been available for decades. The learning health system might make them a priority.
So we emerged from three days entombed in a room with exquisite views of Lake Como and surrounding mountains with agreement on the problem and a clear vision of an answer. Now the challenge is to make it happen, and tasks have been allocated.
Competing interest: RS paid his own fare to Milan, but the Rockefeller Foundation paid for his transport to and from Milan Airport and his accommodation and food, including wine with every meal except breakfast. There was also free beer, and he had three bottles. He’s convinced (perhaps wrongly) that he remains uncorrupted.
Richard Smith was the editor of The BMJ until 2004. He is now chair of the board of trustees of icddr,b [formerly International Centre for Diarrhoeal Disease Research, Bangladesh], and chair of the board of Patients Know Best. He is also a trustee of C3 Collaborating for Health.