After 30 years of trying to teach clinicians, policymakers, journalists, and patients the basic concepts of deciding if claims about health interventions are valid, Andy Oxman, one of the originators of evidence based medicine, decided that it’s tough to teach adults new ways of thinking because of all the baggage in our heads. So he and his colleagues in Norway and East Africa have turned their attention to children. A group of 20 health and education researchers, teachers, clinicians, publishers, designers, and creators of games from eight countries met last week in Bellagio, Italy, to discuss progress with the work and future plans.
It is possible to teach young children the concepts of better decision making about health interventions
Oxman started by trying to teach his own children and found that they could grasp the concepts. He then went, armed with a bag of different coloured sweets that he used as a teaching aid, to his children’s school and found that the children could understand the importance of concepts like comparing like with like when testing interventions and of blinding. He was delighted when a teacher suggested that comparable groups could be selected by picking every other child and the children pointed out that that did not work and could easily explain to the teacher why.
The next step was to make an application for funding to test the idea of teaching children the basic concepts of assessing the validity of claims about health interventions, and the Norwegian team was successful with getting funding to develop learning resources and test them in a large trial among primary school children in Uganda.
Which concepts to teach?
But what concepts should be taught? Oxman and his colleagues, including Iain Chalmers, one of the founders of the Cochrane Collaboration, identified in the book Testing Treatments and other sources 32 concepts that are essential in evaluating claims about healthcare interventions. These concepts are available on Testing Treatments interactive and are divided into three broad categories: asking if claims are justified; asking if comparisons are fair; and making informed choices.
Each concept is accompanied by an explanation and a description of implications. For example, the concept that “Treatments may be harmful” has the explanation that “People often exaggerate the benefits of treatments and ignore or downplay potential harms. However, few treatments are 100% safe.” The implications are “Always consider the possibility that a treatment may have harmful effects.”
Chalmers says that the list of concepts is a “living document” and will be modified. Those at the Bellagio meeting were supportive of the concepts, although there was discussion over whether they might be grouped in some other way or whether a simpler version based on principles might be created.
For the trial in Uganda the 32 concepts were reduced with the help of teachers to 24 thought relevant and teachable to 10-12 year olds. The team then realised that it was possible to teach only 12 in the nine 80 minute lessons available. It’s hoped that the other 12 will be taught later.
Assessing ability to apply the concepts
A means of assessment was needed for the trial, and a systematic review found little that was useful. The team thus had to produce their own means, and they have developed a bank of multiple choice questions. They continue to develop and expand these, but here is an example:
Regina has an illness that makes it difficult for her to breathe. She hears on the radio about a medicine that has helped many people for their breathing problems.
Question: How sure can Regina be that the medicine does not have any harms?
Options:
- It is not possible to say. However, medicines are rarely harmful.
- Not very sure, because all medicines may harm people as well as help them
- Very sure, since the medicine has helped many people, it is unlikely that it also harms people
How to teach the concepts?
The next step was to think how best to teach the concepts. This proved to be a long and difficult process that took two years and involved many mistakes. The team recognised that the methods must be stimulating and fun. The team brainstormed ideas with researchers, designers, and teachers, sorted through the ideas that emerged, designed prototypes, tested them on the children, and modified them. Many ideas had to be abandoned and most prototypes went through many stages of development.
The team started trying to teach with games, which worked well with small numbers of children but became unmanageable in large classes. They tried sweets, paper aeroplanes, and many other ideas before they arrived at a carefully designed comic book. This is given to the children, which in itself is motivating: the children liked having their own book. The book, which is available online includes exercises and classroom activities that are like games. There is also a teacher’s guide and other learning resources.
At the meeting we divided into small groups and tried our hand at devising ways of teaching a particular concept. I’ve described my experience of the process in another blog. There was much discussion at the meeting of the possibility of designing games, including video and computer games, that might help assist the teaching and make it easier to scale up the programme. Those at the conference who create games were all interested in producing games that would have social benefit as well as be fun, but it became clear that producing a game that gets widely played is not easy. Nevertheless, it can be done and may be done.
The classes and the Uganda trial
The classes start with reading the comic book out loud, sometimes taking on the roles of different characters in the story. Teachers stop to ask the children questions to check their understanding as they read and stop to explain new words. This is followed by an interactive activity that the class does together and then by exercises that the children do on their own. The effectiveness of the teaching is being tested in a cluster randomised trial in 120 schools in Uganda with 15 000 10-12 year olds. The outcome measure is a test with the MCQs. This doesn’t favour those who have received the teaching because they will not have answered such MCQs during the course and all schoolchildren in Uganda answer MCQs in national exams. There will also be process evaluation using both qualitative and quantitative methods to explore other potential effects—both harmful and beneficial—and to learn more about how and why the intervention has whatever effects are found. The teaching and testing has now been completed, so results should be available before the end of the year.
What is already clear is that the children and teachers are excited by the programme and enjoy it. Some teachers were teaching the programme at 6.15 in the morning. The Ugandan ministry of education is impressed and hopes to continue and spread the programme. There is also interest in Kenya and Rwanda, where the learning resources have been pilot tested.
There was discussion at the meeting of possible adverse effects. Assuming that something of value was taught during the 12 sessions that had been replaced, then something will have been lost. Another adverse effect might be to teach students to question all authorities, including health professionals. Most of those at the meeting, including health professionals, thought this a good not a bad thing.
What exactly is the aim of the programme?
The most intense discussion at the meeting was over the aim of the teaching: was it to teach critical and scientific thinking or to get the children to make healthier choices? The outcome measure from the trial will be a test of understanding of the concepts not health choices.The primary aim of the team is, however, to enable children to make healthier choices, recognising that children aged 10 to 12 don’t make many health choices. Several of those at the meeting were sceptical that teaching the concepts would lead to healthier choices because so many other factors come into play when people make choices.
Everybody at the meeting favoured the teaching, with many saying they hoped their own children might have such teaching. Most thought that the teaching would encourage critical and scientific thinking, but we heard how there is huge experience of trying to teach critical thinking with many published studies. Much of this evidence was unknown to most of those at the meeting with a health background. One conclusion is that teaching critical thinking in the abstract doesn’t seem to work, although this conclusion is disputed by some. Critical thinking seems to be confined to particular domains: so students who have learnt to think critically within philosophy do not then think critically within science. Indeed, it may be that effective critical thinking within physics may be different to critical thinking within biology or health. Critical thinking and scientific thinking are separate skills with some overlap.
There continues to be, however, great enthusiasm for teaching critical thinking, and it was recently identified by the World Economic Forum as a 21st century competency along with creativity, communication, and collaboration. That critical thinking may well be different in different domains offers an opportunity for the team to teach critical thinking on health interventions. People also observed that health education is included in the school curriculum in many countries, and that perhaps the team’s programme might be included within health education, teaching critical thinking and scientific reasoning at the same time.
Spreading the programme
The role of teachers in the programme was much discussed, and, although the team had hoped to create a programme that didn’t depend on teachers’ prior understanding of the concepts, everyone at the meeting agreed that it would be crucial for teachers to understand the concepts and be enthusiastic about the programme. Without their engagement and commitment the programme will not spread.
The programme is already being taught in an international school in Oslo, where an earlier version was pilot tested. This shows that the school was impressed enough with the programme to incorporate it into its curriculum. Many of the schools that have been involved in piloting and testing the programme have also been enthusiastic about it. The team has applied for a grant to develop and evaluate learning resources for 16 year olds in Norway, students who are old enough to make health decisions, meaning that decision making could be an outcome measure. The same application covers teaching concepts that are relevant to policymaking using role play and digitised resources. The aim is to not only help them to make better personal health choices (as patients or future health professionals), but to enable them to make well-informed judgements about policies that affect health (as citizens and future policymakers).
The team is also making an application to the MacArthur Foundation’s call for $100m proposals to help solve a critical problem. The critical problem is the pain, suffering, premature death, and wasted resources that currently result from poorly informed decision-making on health interventions. The plan would be to implement the learning resources into health systems and schools in 12 countries by 2023 with plans to introduce them into 100 countries. In addition, there would be 10 000 evidence based summaries to inform health decisions in six languages. There was debate over whether the summaries should be part of the programme or whether the team would do better to concentrate on teaching the concepts.
The chances of winning such a proposal are small, but the team plans to go ahead anyway. Those at the meeting urged them to do a detailed market analysis, recognising that to flourish and spread the programme will need in crowded curriculums to replace existing programmes and attract funding that is currently going elsewhere. To succeed, those who decide on curriculums and allocate resources need to see the programme as a “must have” rather than a “nice to have.”
All those who attended the meeting were enthusiastic about the programme and would like to see it spread. But the central message to the team was to be clear about what exactly they are trying to achieve and how best they might achieve it within school curriculums.
Richard Smith was the editor of The BMJ until 2004.
Competing interest: RS had his fare paid to attend the meeting by the James Lind Initiative and was provided with food and board in the Bellagio Center for four days by the Rockefeller Foundation.