Should We Prove Or Improve Our Methods?

 Abstract
Randomised Controlled Trial (RCT) is the gold standard for proving hypotheses, but it does not answer all questions. Bayesian science assesses the relation between certain events in everyday practice. Its main advantage is that it addresses real-world questions like “will this medicine work for this patient?”. We must askourselves if CAM methods are scientifically valid enough according to up-to-date standards. If this is not the case,we can improve them by bayesian research. This will also improve performance in RCT. Bayesian assessment of 6 homeopathic symptoms indicates that 50% of the existing information might be incorrect. Bayesian research can easily be implemented in everyday practice. Participating doctors must be well aware of the fact that the method is designed to improve, not to prove anything.There are no competing interests. The research mentioned in this paper has been financially supported by Koninklijke Vereniging HomeopathieNederland (Royal Dutch Association for Homeopathy), by Artsenvereniging Homeopathie Nederland (VHAN) (Dutch homeopathic doctorsassociation), the Stichting Fonds Wetenschappelijk Onderzoek Homeopathie and by the Louise van Eeghen stichting.
Introduction
Proven efficacy of a medicine in a Randomised Controlled Trial (RCT)does not give us all the information we need. We know that it has a specific effect, but we don’t have certainty that the medicine will workin daily practice. The effect of a medicine depends on a number of variables, some of which remain unknown. The answer to the question if a medicine works in daily practice is a probability, not a yes or no statement like the outcome of RCT.
Working with probabilities is common practice in diagnostic research. [11] Diagnostic research investigates the relation between signs of illness (symptoms, tests, etc ) and diagnosis. A medical diagnosis will mostly depend on more than one sign; several symptoms and tests add up resulting in more certainty about the diagnosis. This process has a theoretical basis, formulated by Thomas Bayes (1702-1761).[2] Clinical decision making is a fundamentally bayesian process.[3] Some CAM methods state that they treat the person, not the complaint. As an example, the choice of a homeopathic medicine cannot be based on only one item like a diagnosis. The success of a medicine depends on a combination of facts (personal traits), like the process of a conventional diagnosis. Recently this view is also adopted bypharmacogenetics.
CAM methods are largely based on practice experience. Bayes’ theorem can be regarded as an algorithm that describes how we learn from experience. In medicine it’s most accepted application is diagnostics, but in many other fields it is used to improve success rates, like in expert systems and neural networks.
In homeopathy many data are retrospectively and not systematically collected. Applying bayesian methods in prospective research we can assess the prognostic value of our experience. In homeopathy symptoms indicate chances of success instead of diagnoses. A Dutch prospective assessment of the prognostic value of six homeopathic symptoms shows how bayesian research can be used to improve the method.
Bayes’ Theorem
In 1763 Bayes theorem was published, describing the way we learn from experience.[4] It showed how we make valid predictions about the future from past experiences. Bayesian reasoning has since invaded every field of science, because it produces useful results in everyday processes via expert systems and neural networks. The Bayesian principle in diagnostics is quite simple: a diagnostic test is useful if it is positive more frequently in people with the disease than in people without the disease. Likelihood ratio (LR) expresses the relation between the occurrence of the positive test in the population with the illness and in the population without the illness. A test with a higher LR is more indicative. The bayesian formula to predict an event is:
Posterior odds = LR x prior odds1
If a symptom occurs as frequently in the population with the disease as in the rest-population LR=1. Such a symptom gives no information at all.
If we were to assess the diagnostic properties of ultrasonography regarding appendicitis we should perform a prospective research on patients submitted with abdominal pain.
Each patient is submitted to ultrasonography and in the end the outcome of ultrasonography and histology (reference test) are compared with each other. Such a research could give (hypothetical) results as in Table 1.
According to Table 1 the occurrence of positive ultrasonography in the population with appendicitis (also called true positives) is 90%; the occurrence of the positive test in the population without appendicitis (false positives) is 9%. In this hypothetical example LR=10.
The diagnosis appendicitis could be constituted as follows: Suppose that a patient comes with abdominal pain, which becomes worse on motion. The prevalence of appendicitis is 2% in the general population with abdominal pain (=prior chance).[5] Suppose that pain worse on motion occurs 8 times more frequently in patients with appendicitis than in other patients, then for this symptom LR=8. The first symptom that the patient presents increases the odds that he has appendicitis 8 times. The next step is to perform a test, say ultrasonography. If the LR of ultrasonography for appendicitis is 10, the odds of having appendicitis increases again, now 10 times. With translation from chance to odds andvice versa this looks like Table 2.
In this example the chance of having appendicitis increases in two consecutive steps from 2% prior chance to 62% posterior chance. After the symptom ‘pain worse on motion’ the probability of the diagnosis appendicitis increases from 2% to 14%. The posterior chance after the first symptom/test is 14%, and this becomes the prior chance for the second symptom/test. The translation from chance to odds and vice versa seems awkward, but is easily performed by the computer.

 

RCT and Diagnostic Research
What is the difference between RCT and diagnostic research? If we look closer at the process between illness and result of treatment and the corresponding research we often see a two-step process, like in Figure 1.

First the diagnosis must be established, then the therapy based on that diagnosis must cure. Positive outcome of RCT means that a therapy works better than placebo for that diagnosis. This outcome is a yes-or-no statement based on experimental research with randomisation and blinding. The diagnostic outcome is different: We establish probabilities about different diagnoses. This research is generally not randomised.
It is possible that a therapy doesn’t work if diagnosis is not correct. But even if the diagnosis is correct the result of treatment does not merely depend on efficacy of treatment (=positive outcome of RCT). There are variables that are not related to diagnosis, but do influence the effect of treatment. With the new genetic revolution we discovered that there can be genetic susceptibility for certain medicines, see Figure 1.[6] Other factors influencing result of treatment are not included here for simplicity.
The genetic predisposition for responding to medicines reminds us of homeopathy. A homeopathic medicine works if it ‘fits’ the patient. Personal traits – not related to the conventional diagnosis – influence the effect of a medicine. For example: the homeopathic preparation of Arsenicum album can be effective against diarrhoea. We know from practice experience (see later) that many patients who are cured by Arsenicum album have a fear of death. Bayes’ theorem says that this experience can be used to predict chances of success with this medicine. If the diarrhoeic patient appears to have a fear of death, chances increase that this medicine will cure. We could regard this symptom ‘fear of death’ as a phenotypic expression of the genotypic constitution that makes the patient susceptible to the homeopathic medicine Arsenicum album. The increased chance of success could be calculated by Bayes’ formula.
Prognosis instead of diagnosis
In our example the combination of the diagnosis ‘enteritis’ with the genetic trait ‘fear of death’ constitutes a better chance of cure by homeopathic Arsenicum album than merely the diagnosis ‘enteritis’. So, by adding information we increase our certainty about result of treatment. This is virtually the same as the diagnostic process, but diagnosis is replaced by prognosis of illness.
Figure 2 shows how we can replace diagnosis by prognosis. We treat the diagnosis (enteritis) as one of the indications that Arsenicum album could cure. Fear of death is another indication that increases the probability that this medicine cures, and there may be other indications (symptoms/tests).

The relation between the diagnosis enteritis and result and the relation between the symptom ‘fear of death’ and result can both be assessed by bayesian research. We could perform prognostic research to assess the probability of result, similar to diagnostic research.
Instead of the relation between symptom/test and diagnosis, we assess the relation between the symptom/test and result (cure). In table results of a prospective assessment of the symptom ‘fear of death’ regarding the homeopathic medicine Arsenicum album are shown (see next paragraph).
Table 3
These results indicate eg that this symptom is present in about 24% of the population cured by the homeopathic medicine Arsenicum album and in about 4% of the rest population. The LR is about 6. Now
Bayes’ theorem can be used to calculate chances of cure by Arsenicum album in enteritis if the symptom ‘fear of death’ is present. Similar to the probability of the diagnosis appendicitis the probability of cure by Arsenicum album, given the diagnosis enteritis and the constitutional trait fear of death, increases stepwise. We assume that the prior chance that
Arsenicum album will cure, without knowledge about the patient, is 1%. We also assume that the LR for enteritis and Arsenicumalbum is 10.
Table 4
In homeopathy ‘cure’ is defined as amelioration of the presented complaint plus amelioration of general well being, probably due to the medicine. This is based on clinical assessment of all consecutive new patients. The ‘Arsenicum population’ is the population that responded well to Arsenicum album, irrespective of diagnosis. The rest-population is the population not cured by Arsenicum album, including patients cured by other medicines. The same procedure is known in conventional medicine as effect modification research.[7] The likelihood ratio for ‘fear of death’ andcure by Arsenicum album is 5.92.
Figure 1
Figure 2
Methods: Assessment of Homeopathic Symptoms
The example of ‘fear of death’ is part of an ongoing research concerning homeopathic symptoms in the Netherlands.[8] The Committee for Methods and Validation2 of the Dutch homeopathic doctors association started the first prospective assessment of six homeopathic symptoms in June 2004. The planned duration is three years.
After two years we collected 2266 evaluated prescriptions for 2506 patients. One of the assessed symptoms is ‘Fear of death’. This symptom and the medicines that are indicated by this symptom are mentioned in Kent’s repertory and the rubric ‘Fear of death’ contains over 100 medicines.[9] There were 103 patients with this symptom in the whole population, the prevalence was 4%.
Table 5
If the symptom is present in a new patient, the odds that Arsenicum album will cure increase 6 times. So, if the odds after the diagnosis but before the symptom were 10 to 99, they become about 60 to 99 (chance=37%) if the symptom is present.
Homeopathy already has an instrument that indicates something like LR for one century, Kent’s homeopathic repertory. The homeopathic repertory contains rubrics like ‘Fear of death’ with the medicines that could be indicated if this symptom is present. These medicine are presented in plain type, Italics or bold type, where bold type represents a strong indication. This system could be compared with low or high LR values.
Misguiding expert opinion
The clinical data composing the homeopathic repertory arise from expert opinion and this expert opinion can be biased and is influenced by chance. The outcome of our assessment differs from the present repertory rubric in 50% of the entries. The differences are as we expected[10]; the importance of medicines that are frequently used is over-estimated, infrequently used medicines are under-estimated. One medicine, Natrium muriaticum, is for certain incorrectly mentioned in the existing repertory-rubric. None of the 92 patients out of 10 practices who responded well on Natrium muriaticum had a fear of death.
The problem with expert opinion can also be explained by Bayes’ theorem. Our experience with consensus meetings validating homeopathic medicines over 10 years learns that homeopathic doctors can estimate prevalence of symptoms rather well.[11] During such a consensus meeting the prevalence of the symptom ‘fear of death’ in the whole was estimated to be 5%.
Suppose that the prevalence of ‘fear of death’ in the ‘Natrium muriaticum population’ is 1%. Many entries in the repertory are based on the opinion of one expert. If the editor of the repertory would ask the opinion of an experienced homeopath with 15 good Natrium muriaticum cases, exact binomial calculation of the chance that this homeopath has one patient with fear of death renders 14%. The doctor who has such a patient estimates that the prevalence of the symptom in the Natrium muriaticum population is 1 out of 15 (6.7%), more than his estimate of 1 out of 20 (5%) for his whole practice.
So in the experience of this doctor the symptom fear of death is a (weak) indication for Natrium muriaticum.
Only gathering results of 10 colleagues will learn that the real prevalence of this symptom in the
Natrium muriaticum population is 1%. This way many entries in the repertory are influenced by chance.
Discussion
RCT is not the only acknowledged scientific method in medicine. Bayesian research already exists in
diagnostic research. The same method, but applied as prognostic research or effect modification research, might be useful for CAM practice. This is illustrated by assessment of homeopathic symptoms. There are many flaws caused by expert opinion in the existing homeopathic instruments – like repertories – that can be repaired by Bayesian research. If the method is improved this way it will produce better results in RCT. Critics of bayesian philosophy state that it cannot be used to falsify any hypothesis.[12] But that is not the only reason that this kind of research should not be used to prove CAM methods. One of the most critical points in our assessment of the LR of homeopathic symptoms is the fact that the participating doctors judge the result of treatment themselves. It would be much more complicated and costly to assess result of therapy by independent observers. The participating doctors have to be convinced that this kind of research is not meant to prove anything. If the participants misjudge results of treatment, they hamper the quality of their own instruments. The participating doctors in this study attended consensus meetings about judging symptoms and results.[13], Results were regularly monitored and feedback was sent to the participants by e-mail newsletter.
Our ‘reference standard’ is ‘cure’.[14] The vagueness and subjectivity of this reference is another reason why this research cannot be used to prove any hypothesis like ‘homeopathy works’.
In fact we define our best cases and we investigate how to increase chances to obtain such results. Frei et al did something similar in their preparation of an RCT on ADHD.[15][16] They demonstrated that optimising our instruments (questions, repertory rubrics) improved results of RCT.
One of the advantages of LR research is that is has no negative impact on treatment (no placebo groups, no change in everyday practice). There is no risk for patients, they get the usual treatment. Furthermore, it is rewarding for patients and practitioners and practitioners are closer involved in the scientific development of their own method.
Our example is about homeopathy, but bayesian research can be applied in any field where you want to assess practice experience. We must develop this methodology further for each method to avoid bias, but the outcome is probably more reliable than the existing expert opinion.
Conclusion
Bayesian evaluation of everyday practice is an undervalued method in CAM. If a method largely depends on expert opinion it is liable to deficiencies caused by bias and chance. These deficiencies can be corrected by bayesian research.
Bayesian research stays close to daily practice, it therefore renders very useable results and does not influence daily practice or treatment. It must be clear that this kind of research is only suited for improving the method, not for proving it.
References
1. Knottnerus JA, editor. The evidence base of clinical diagnosis. 2002 BMJ Books London
2. Bianchi MT, Alexander BM. Evidence based diagnosis: does the language reflect the theory? BMJ 2006;333:442445
3. Gill CJ, Sabin L, Schmid CH. Why clinicians are natural bayesians. BMJ 2005;330:1080-3
4. Bayes T. An Essay Toward Solving a Problem in the Doctrine of Chances. Philosophical Transactions of the Royal Society of Londo.1763:53:370-418
5. Garcia-Aguayo FJ,.Gil P. Sonography in acute appendicitis: diagnostic utility and influence upon management and outcome. Eur.Radiol. 2000;10:1886-93.
6. Roses AD, Saunders AM, Huang Y, Strum J, Weisgraber KH, Mahley RW. Complex disease-associated pharmacogenetics: drug efficacy, drug safety, and confirmation of a pathogenetic hypothesis (Alzheimer’s disease). Pharmacogenetics J. 2006 Jun 13; [Epub ahead of print]. PMID: 16770341
7. Vineis P, Kriebel D. Causal models in epidemiology: Past inheritance and genetic future. Environ Health. 2006 Jul 21;5(1):21 [Epub ahead of print] 8. Rutten A.L.B., Stolper CF, Lugten RF, Barthels RJ. A Bayesian perspective on the reliability of homeopathic repertories. Homeopathy. 2006;95:88-93
9. Kent JT. Repertory of homeopathic materia  medica. New Delhi: B Jain Publisher; 1990
10. Stolper CF, Rutten ALB, Lugten RFG, Barthels RJWMM. Improving homeopathic prescribing by applying epidemiological techniques: the role of likelihood ratio. Homeopathy 2002;91:230-238
11. Stolper CF, Rutten ALB, Lugten RFG, Barthels RJWMM. Materia medica validation and meta-analysis: A postgraduate course combining learning and research. Homeopathic Links 2004;17(3):186-188
12. Mayo DG. Error and the growth of experimental Knowledge. TheUniversity of Chicago Press.
Chicago 1996
13. Rutten A.L.B., Stolper CF, Lugten RF, Barthels RJ. Assessing likelihood ratio of clinical symptoms: handling vagueness. Homeopathy. 2003; 92:182-186.
14. Rutten ALB, Stolper CF, Lugten RF, Barthels RJ. ‘Cure’ as the gold standard for likelihood ratio assessment: theoretical considerations. Homeopathy 2004; 93:78-83
15. Frei H, Everts R, Von Ammon K, et al, Homeopathic treatment of children with attention deficit hyperactivity disorder, a randomised, double blind placebo controlled crossover trial. Eur J Ped, 2005, 164/12; 758-767
16. Frei H, Ammon K von, Thurneysen A. Treatment of hyperactive children: Increased efficiency through modifications of homeopathic diagnostic procedure. Homeopathy 2006; 95:163-170
Source: HOMEOBUZZ, April 2008