How misunderstanding the principles and limitations of evidence-based medicine may discredit the concept

Full Text

How misunderstanding the principles and limitations of evidence-based medicine may discredit the concept¹

Doctors and several researchers in Russia have been rejecting the ideas of clinical epidemiology and evidence-based medicine (CE/EBM) for over 30 years. Additionally, there is almost widespread ignorance on this subject (in Russia, there is still no compulsory teaching of this subject, and familiarity with EBM comes from a selection of “scientific catering” [3]). Moreover, there are other underlying reasons for this situation.

A recently appeared German article sheds further light on this issue of political differences and trust in science. Its brief conclusion states that “trust in science... is polarized along political lines: conservatives (compared to liberals) tend to trust science and scientists less... a simplified explanation for this phenomenon consists in the fact that interaction of political stereotypes about scientists (for example, “scientists are liberals”) and their own political orientation are the main factors determining people’s trust in science. Across the political spectrum, people trust only those scientists whom they consider ideologically similar to themselves”² [4].

This seems to be the core of the problem of the relationship between doctors and researchers³. In Russia, as is often the case, it is manifested as an archetypal confrontation between “nationalists” and “Westerners.” N.S. Leskov wrote about it in 1893: “At the end of September 1893, at a meeting of the Society for the Promotion of Russian Industry and Trade, one speaker directly said that “Russia must separate itself, forget the existence of other Western European states, separate from them with a Chinese wall” [5]⁴.

Nothing has changed over the centuries: “We are doctors, not stupid appendages of evidence-based medicine...”; “fed up with your imposed standards of evidence-based medicine…. and this is all “evidence-based medicine” imposed from the USA and WHO” [6]⁵. Here is the opposite point of view: “Every word in American or British psychiatry manuals can be trusted, since their content is based on scientific facts and reflects clinical reality”..., “a scientific argument in our time can only be considered a fact obtained as a result of scientific research of adequate quality—primarily large randomized clinical trials (RCTs), as well as those conducted based on numerous controlled studies, systematic reviews, and meta-analyses (MA)” [7].

It is not obvious what discredits EBM more: the “attraction toward the acorn and the trough” or the obsequiously self-deprecating, rather irresponsible, “pro-Western” text⁶. It will be shown below that the ideas of EBM can be discredited, and incorrect results can be obtained by conducting a study that is demonstrative in form but erroneous in content and conclusions.

In the abovementioned articles [1, 2], the readers can familiarize themselves with a detailed analysis of the following provisions, which will be briefly repeated here⁷.

Attempts to evaluate the comparative efficiency of antidepressants (ADs) in MA resulted in paradoxical conclusions regarding their equivalence. One of the plausible explanations for this was the fact that researchers were engaged by sponsor-manufacturers, when everyone “praised their own talent,” and a new “performance leader” emerged in each study. This was shown, in particular, in the famous lecture by J. Ioannidis “Evidence-Based Medicine Has Been Hijacked” (a review of 185 MA on the efficiency of ADs over 7 years) [8]⁸.

In MA, the most effective drug could not be determined due to multidirectional data from individual RCTs, since in the next MA, it was already in a completely different place in the hierarchy. Only adverse events were clearly distinguished. Obviously, this, in particular, became the reason for Peter Goetsche to claim even earlier that, in general, all ADs do more harm than good [9]. Consequently, marketing switched to advertising ADs based on better indicators of side effects.

However, these facts have another explanation that systematic errors in assessing effectiveness are often due to the inadequacy of medical survey instruments. These instruments exhibit cultural specificity [10] that does not allow their use in another cultural space and, even after revalidation, does not make it possible to summarize their readings in transcultural MA.

In addition, the latent construct being “measured” (depression) is related to the numerical indicator in a nonlinear and probabilistic manner. The “assessment” of psychometric parameters reaches a point of fundamental impossibility of tracking the nature of the connection between latency (depression) and indicators (scale points), since the severity, and sometimes the very fact of their occurrence, is mediated by the individual meanings of mental experiences, which are of a semiotic nature [1].

Scales for certain narrow pragmatic purposes (e.g., assessing the dynamics of a condition in an individual patient) can be used after cultural adaptation. However, as the number of patients in clinical studies increases, that is, as their cultural and individual diversity increases, a random multidirectional scatter of indicators will begin beyond their boundaries. In my opinion, this could also lead to a false conclusion about the equivalence of the AD action, based on the results of MAs, where the indicators were “averaged.”

Surrogate (quantitative) outcomes in clinical trials (measures of depression severity) should be replaced by clinical indicators. As compared with a committed suicide, a prevented suicide is the most severe form here⁹.

Another source of systematic reviews in MA is the use of different instruments in RCTs, which are then combined into one MA. Here is an example of such a Cochrane MA, in which RCTs used six different instruments (abbreviations only), namely, HAM-D, MADRS, BDI, SF-36, HoNOS, and WHOQOL [12].

As can be seen, these are not only scales for assessing depression itself but also other tools. Additionally, “economic outcomes (e.g., days away from work/ability to return to work, number of primary care visits, number of referrals to secondary services, and use of complementary treatments) were added and, if reported, summarized in a narrative form.” Adverse events (e.g., committed suicides/attempted suicides), if reported, were summarized descriptively.

It is important to note that the listed economic indicators are also culturally dependent and differ across various healthcare systems. Moreover, they can vary individually based on the situational motivation driving the patient’s actions [13]. Due to these factors, the validity of their generalization is questionable.

Assessment tools are directly related to disease outcomes (endpoints)¹⁰. EBM has falsely or erroneously attributed a desire for almost exclusively quantitative indicators¹¹. It has even been asserted that EBM decided to use rating scales [14]. At the same time, it is “not noticed” that these instruments arose and were introduced into practice decades earlier (e.g., the Hamilton scale in 1960; Beck’s scale in 1978; and EBM in 1980–1990s). EBM inventions often include “Western” classifications of diseases (DSM¹², ICD¹³). However, neither they nor the assessment tools, as well as statistics, are related to the specifics of EBM.

Moreover, CE/EBM postulates the study of not quantitative (indirect, surrogate) but rather “clinical outcomes” (EBM term). The latter is formed at the point of transition from quantity to quality (calcium level—fractures, blood pressure level—strokes, heart attacks; severity of depression—suicide). For depression, the harshest clinical outcome is death. Therefore, once again, the primary outcome for evaluating the effects of AD treatment should be suicide prevention.

This has been pointed out by several researchers in the West¹⁴, such as John Geddes, professor of epidemiological psychiatry at Oxford University and a leading researcher in the field of depression. He emphasized the importance of using parameters of suicidal behavior as a primary outcome in depression research, stating that “suicidal behavior is the most important outcome for patients, their families, clinicians, and politicians” [15].

David Healy, a psychiatrist, psychopharmacologist, and a leading critic of the pharmaceutical industry, has argued that the use of surrogate measures, including PROM¹⁵, in depression research has led to an overreliance on ADs that may not actually reduce suicide risk [16]. Joanna Moncrieff, a psychiatrist and researcher, has argued that focusing on PROM as the primary outcome in depression studies reflects a limited perspective of the condition that ignores the social and cultural factors contributing to it, emphasizing that suicide should be the primary outcome in such trials [17]. Peter Kramer, a psychiatrist and author of the popular book Listening to Prozac, wrote that “the ultimate goal of treating depression is to prevent suicide” [18].

Nevertheless, most studies examining the effectiveness of interventions for depression focus on a surrogate primary outcome, namely, the severity of depression itself¹⁶. Moreover, individuals at risk of suicide were deliberately excluded from AD clinical trials, making the outcome of death “invisible” even as a side effect: “Mortality from suicide and suicide attempts decreased considerably in AD clinical trials after 2000 compared with the decade before 2000.” Basic patient demographic characteristics remained unchanged, and the effect of drug treatment on suicidality was not clear. These results may reflect expanded screening procedures and effective exclusion of suicidal patients from clinical trials of depression” [19].

Finally, it is unexplained why this fact was not noticed by the majority of EBM supporters, even by such outstanding figures as Peter Goetsche and J. Ioannidis.

Thus, a considerable number of clinical studies of ADs nowadays are an example of evidence-based design, where first “undesirable” outcomes are deliberately removed from the sample, inadequate outcomes are “measured” using unsuitable instruments, and then the non-generalizable is generalized into MA.¹⁷

Even if we begin to study not biological (medicinal) but psychotherapeutic effects, such results cannot be generalized in clinical studies. Quasi-measurements of the severity of depression or other existentially-dependent parameters (e.g., suicidal intentions rather than death itself) will “float” exactly in the same way, and we will again be forced to think about “hard” outcomes: “It is worth noting that several significant gaps exist in the literature. First, given the paucity of RCTs capable of detecting deaths due to suicide, it is unknown whether deaths due to suicide, rather than attempted (incomplete) suicide, can be prevented using psychotherapy. Moreover, it is also unclear whether psychotherapy aimed at reducing suicide attempts or suicidal ideation actually reduces deaths due to suicide” [20].

However, there is one remarkable obstacle to selecting suicide prevention as an outcome: “as suicide is a behavior with a low base rate, very large samples are required to conduct adequate studies” [20]. “The relatively low incidence of suicidal events, even in the population with depression, means that a reliable assessment of the effect of treatment on suicide risk would require... approximately two million participants” [21].

This will probably discourage future RCTs from producing convincing results on the efficacy of ADs. National registers could potentially offer a solution. However, their creation is a lengthy and expensive process, likely to be outpaced by the development of new ADs. Moreover, the content of the registers will partially depreciate. If we begin searching for a “hard” criterion of individual (or cultural) sense, answering the question in the epigraph of this article “Why do we live?” (i.e., we will determine why we need to get rid of depression), then even having found it, we will find ourselves again in an endless circle. Such an outcome cannot be generalized for a transcultural population, for example, in MA.

Thus, in matters of studying the efficiency of ADs, which would enable us to summarize them in universal clinical recommendations, we get into a methodological dead end. We must admit that we are on the verge of abandoning “universal human values” in psychiatry and will have to adopt individual and cultural values¹⁸.

INSTEAD OF A CONCLUSION

Who benefits from “neglecting” these errors? It is arguable that, just as with research funding [8], the pharmaceutical business is primarily interested in preserving this “pseudoscientific incorrectness.” In addition, there can also be personal interests of those who want to “scientize” (“mathematicize”) psychiatry and their own activities.

In addition, the mill of the market, interested in expanding the indications for the use of ADs through the medicalization of the ordinary situations, is accelerated by the idea promoted by it that social inequality is caused by certain biological imperfections that can be corrected by pharmacological drugs: “The theory of chemical imbalance assumes that there is a normal or ideal neurochemical state, against which any person can be measured. As the boundaries of disease expand¹⁹, large parts of the population are indoctrinated into dissatisfaction with themselves and the need to “correct” it by changing their brain chemistry. People are encouraged to stop being themselves, both in their emotional and material lives. Individual consumption (pharmaceuticals in this case) is presented as a means to achieve his end” [22].

Using the example of assessment scales “common to humanity” and the outcomes corresponding to them, it becomes clear that ideological issues (mentioned at the beginning of this article) always latently present in all spheres of human activity (in our case, an attempt to impose common meanings of existence on the whole world, ignoring cultural diversity) form scientific snobbery and defective research methodology. Figuratively speaking, nowadays research activities in medicine serve the interests of the white man²⁰.

This is also the choice of directions for clinical research on EBM, as highlighted by J. Ioannidis [8], as 25% of the global burden of disease accounts for 0.1% of work, such as clinical trials (i.e., EBM was engaged in what is interesting to the white population...); even the calibration of pulse oximeters was performed on white populations, and later, it was discovered that they gave incorrect results in people with darker skin. The same happened with the genomic project, which was performed almost exclusively on Europeans (96% of the studies were conducted on 12% of the Earth inhabitants) [23].

It is necessary to add one more, perhaps the most powerful source of, to one degree or another, conscious manipulation of medical data. The far-fetched reproaches of many doctors against CE/EBM (“prohibits creativity,” “cancels the clinical method,” etc.) are not caused by “methodological concerns²¹.” People in general and doctors in particular do not want anyone to control their activities²² and encroach on biopower, as M. Foucault²³ understood it [24, 25]. Nowadays, there is a confirmation of this. Long before the advent of EBM, the American doctor, aristocrat, Ernest A. Codman (1869–1940), who created the first disease registers, was accused of betraying class interests, since he encroached on “creativity”: “Striving to destroy the image of medicine as “art” depends on the wisdom of a selected group of supernaturally talented people; Codman also threatened to destroy the class-based reality underlying this public shell²⁴.” “The medical establishment on both sides of the Atlantic did their best to avoid data verification.”

And one last thing. “Data sets are not the “impersonal evidence” that Codman thought to be, …the data undoubtedly reproduces and concretizes the prejudices of society itself.” As the Massachusetts Institute of Technology computer scientist Marzyeh Ghassemi explains, the data offers “a glimmer of objectivity,” reproducing the ethnic, racial, gender, and age biases of institutionalized medicine. Thus, the tools, tests, and techniques based on these data are also not unbiased” [23].

ДОПОЛНИТЕЛЬНО

Финансирование. Исследование не имело спонсорской поддержки.

Конфликт интересов. Автор заявляет об отсутствии конфликта интересов.

Благодарности активному члену Общества специалистов доказательной медицины Павлу Викторовичу Желнову за предоставленные источники и дельные замечания.

Funding. This publication was not supported by any external sources of funding.

Conflict of interests. The author declare no conflicts of interests.

Thanks to Pavel Viktorovich Zhelnov, an active member of the Society of Evidence-Based Medicine Specialists, for providing sources and practical comments.

¹ This work represents a continuation of the discussion that arose after publications [1, 2].

² Hereafter, unless otherwise indicated, all emphasis and italics (quotations) are made by the author N.A. Zorin.

³ Centrism is not welcomed by the public. It requires the researcher to belong to one of the polar groups, and if someone disagrees with this, then they will either be assigned to where the public sees fit or will be ostracized from both sides. Serious exasperation here reaches a degree comparable with that in an officer’s servant described by F.M. Dostoevsky, who divided the world into two parts. He classified himself and his master as one part, and “the rest of the bastard” was classified as the other... [cit. according to 5].

⁴ “Almost simultaneously, two economic brochures were published, one of which is “On the beneficial medicinal effects of the bark and young shoots of the ash tree” and the other “On healing properties of glossy soot.”... While it was possible to obtain it (soot) only in Russian smoky huts, and nowhere else, since glossy soot was needed, which is only available in Russian huts, on the walls, rubbed with men’s sweat from interscapular region. Fluffy or shaggy soot had no healing properties. There is no such good in the West anymore, and the West will come to our shelter for our soot, and it will depend on us whether to give them our soot or not; and, of course, we can ask whatever price we want. We will have no competitors. This was stated seriously, and our soot was directly equated with rhubarb and tormentil root, with which it would compete and then kill them and become the glory of Russia throughout the world” [ibid.].

⁵ A collection of such statements from the site https://vrachirf.ru/ can be found at https://encyclopatia.ru/wiki/Критика_ДМ.

⁶ An analysis of this text is presented in [7]. This article, to a certain extent, is a continuation of the analysis on this subject.

⁷ I ask those who are going to criticize this text not to do it based on the theses, but look at the primary sources.

⁸ The FDA (Food and Drug Administration, the US Federal Service that controls the production, storage, and sale of food, drugs, and cosmetics) also stated that the effectiveness of all ADs is the same because otherwise has not been proven (2019) [11].

⁹ Studies must be comparative with another drug, since for ethical reasons randomized clinical trials with such an outcome and placebo control are unacceptable. Sensations and feelings are in principle not suitable as outcomes of clinical studies, since the criterion for their existence is the very fact of this existence. “Thus, for example, the statement that the rose which fragrance I inhale is material and exists objectively, as well as the statement that it exists only in the perceiving consciousness... are equally meaningless. Whether I consider a rose to be material or ideal will not affect the fact that I smell it, and this will not make it smell worse or better.” R. Carnap https://fil.wikireading.ru/12132.

¹⁰ If the outcome is death, then measuring instruments become unnecessary.

¹¹ Quantitative indicators imitate psychologically understandably the scientific nature of the work (“according to a latent attribute”: numbers mean “mathematics,” which means “science”); they are quickly obtained, in contrast to clinical outcomes, which, to one degree or another, have to be expected a long time (and here a dissertation has to be defended). This example shows the difference between classical science and research practice in medicine.

¹² DSM—Diagnostic and Statistical Manual of mental disorders.

¹³ ICD—International Classification of Diseases.

¹⁴ ChatGPT Mar 14 Version was used to collect information for this section. Free Research Preview. https://chat.openai.com/.

¹⁵ PROM—patient-reported outcomes measure.

¹⁶ I cannot help but repeat once again that this makes us participants in the joke: “Does your treatment help the patient?—“Oh yes! We have great improvement! The patient who considered himself Louis XIV now considers himself Louis XIII.”

¹⁷ This is not a criticism of CE/EBM! This is an example of how it is possible, unintentionally or deliberately, without observing the principles and postulates of EBM to obtain a false result.

¹⁸ I can already hear opinions about the invention of a “national multiplication table.” However, this comparison is inappropriate. Medicine, as I have repeatedly said, is not a science, but only one of the human practices, immersed in the market environment and, like any other production, using the technologies of fundamental disciplines.

¹⁹ Quantitative surrogate assessments become a tool for expanding the boundaries of the disease (medicalization of the everyday situation), since they enable to “set” arbitrarily the boundaries of acceptable expression of feelings, depending on marketing objectives. A person “with numbers in his hands” is shown that he is supposedly different from the rest, for example, by 10 points on some rating scale... (See the footnote above about Louis IV).

²⁰ I would clarify, in our case, “Anglo-Saxon man.”

²¹ Perhaps in innocence …

²² CE/EBM requires compliance with a certain set of medical technologies (recommendations and even standards) before proceeding to the creative application of this knowledge on a specific patient. Any accurate data (obtained by CE/EBM) can reveal poor quality work of doctors.

²³ The explicit or implicit ability of society and its power structures to normalize and regulate the biological functions of certain individuals (M. Foucault).

²⁴ Today, in an era where doctors who have lost their class affiliation are recruited from all social strata, it would be more correct to propose not class differences but an attack on biopower, the execution of which is delegated to doctors.

About the authors

Nikita A. Zorin

Author for correspondence.
Email: nzorin@inbox.ru
ORCID iD: 0000-0001-7245-216X
SPIN-code: 6352-6095
Scopus Author ID: 56763767500

M.D., Cand. Sci. (Med.)

References

Supplementary files