Origins in public health practice

The link between public health practice and EBHC is firmly established. Authors in the area often seek to establish its lineage by tracing its roots to the pioneering studies including the epidemiological work of Semmelweiss and Oliver Wendel Holmes, both of whom identified the role of health care practitioners in causing puerperal fever through lack of basic hygiene (Rangachari 1997). Systematic observation allowed Holmes to discern that incidence of puerperal fever seemed to be in clusters that were associated with particular physicians and concluded that these physicians were transmitting the disease-causing agent from one woman to another. Semmelweiss’ comparison of infection rates in women cared for by doctors (who did not wash their hands when attending to women after post-mortem examinations) and midwives (who were not contaminated in this way) is held up as an early prototype for today’s randomised controlled trials (RCTs).

If Holmes and Semmelweiss are claimed as its grandfathers, Archie Cochrane, an epidemiologist who worked on the earliest RCTs conducted by the Medical Research Council in the UK, could be said to be the father of today’s EBHC movement. After a lifetime spent working on studies of effectiveness he complained, ‘It is surely a great criticism of our profession that we have not organised a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomised controlled trials’ (Chalmers 1997). For Cochrane, an unbiased synthesis of evidence from good-quality studies is the true endpoint of health research. He campaigned for effective health care to be free in the National Health Service (NHS), but equally argued that interventions with no evidence of effectiveness should not be offered unless as part of a well-designed research programme.

Use of evidence

At the heart of the EBHC movement lie two simple premises. Many interventions lack evidence of effectiveness and much existing evidence of effectiveness is not put into practice. For practitioners and policy makers alike, the most significant aspect of this problem will be the identification and utilisation of existing evidence, since action can rarely be delayed until new research is commissioned to answer a question. Even where the main action is to undertake or commission research or implement pilot projects to be evaluated, it is to be hoped that such action would be based on an appraisal of the existing evidence-base (or lack thereof).

Muir Gray (2001) reduces the problem of identification and utilisation of evidence of effectiveness to a simple algebraic formulation (which I suspect he would not wish us to take too literally). In his formulation the decision-maker’s performance as an ‘evidence-based decision-maker’ is proportional to motivation (to utilise evidence) and competence (in identifying, appraising and interpreting evidence). It is inversely related to barriers which, in Muir Gray’s account, are lack of resources for accessing evidence (Figure 8.1).

Figure 8.1 Algebraic account of research utilisation (after Muir Gray 2001)

Muir Gray’s term ‘motivation’ covers a complex set of cognitive processes, which are often touched on in studies of research utilisation. In the extreme case it has been frequently argued that practitioners will simply disbelieve or dis-regard evidence if it does not accord with pre-existing beliefs (Hunt 1996, Jones 1997, Rodgers 1994). Within the paradigm of EBHC, these complex factors are essentially irrational, in that they will tend to lead the individual toward making an incorrect decision (or to be sufficiently unmotivated as to make no decision at all).

The experiences of the early pioneers Semmelweiss and Wendel Holmes illustrate that the power of rational argument alone is not enough to ensure that evidence is implemented. Semmelweiss was eventually dismissed from his job in Vienna and forced to return to his native Hungary, after failing to find further work. Wendel Homes was the subject of considerable scorn from eminent practitioners for many years, before his findings were finally accepted. One detects a whiff of anti-semitism in the treatment of Semelweiss and it is clear that the perceptions of the messenger have considerable impact upon people’s willingness to listen to the message. This is a complex topic, and one that is well outside the remit of this chapter, although readers might like to consult two excellent reviews of implementing evidence that have been commissioned by the NHS in recent years (Greenhalgh et al 2004, NHS CRD 1999).

There is, however, one issue of substantive import that must be addressed prior to moving on. I would not wish to ally myself to an extreme relativist pos-ition on knowledge. Absolute relativism has little place in a practice discipline where all parties must accept (if only on pragmatic grounds) that individuals do not completely construct their own realities. Events such as survival and death are of fundamental significance and subject to external verification. However, this does not necessarily mean that the evidence available for rational scrutiny is not subject to interpretation. What one party may perceive as useful evidence may not necessarily be seen as such by another. The questions asked about health care practice are selective and the answers that are published are also filtered and selected according to particular world views. It is not that practitioners are free to construct any version of reality that they choose, but equally the available evidence cannot be said to simply purvey objective ‘truth’ in any absolute sense either (Griffiths 2005).

Much of the research on research utilisation implies that disbelieving the evidence of research is always an essentially irrational phenomenon. Certainly, experience suggests that a blanket refusal to believe research evidence based on a sophisticated rational appraisal of its limited utility is rare. However, the distinction between effectiveness and efficacy identified earlier should serve as a warning that the implications of a particular piece of evidence for practice in a particular setting are often far from clear. Thus, lack of motivation might, in part, be a product of the lack of utility practitioners find in evidence offered to them for making decisions about effectiveness. We will return to this later when considering the formulation of evaluations. For now, we will consider the model decision maker who is motivated to find evidence for effectiveness and is willing (at least in principle) to implement it. What follows is based squarely within the EBHC paradigm. The reasons for this are twofold.

First, the literature on evidence-based practice clearly describes the basis on which many health care policy and practice decisions can be, and are, made. Indeed, although by no means ubiquitous (see Sheldon et al 1996), the EBHC model is probably now the dominant one in current health services research and research policy. Certainly, the majority of initiatives that resulted from the UK’s NHS research and development and information strategies in the early 1990s (e.g. the Centre for Reviews and Dissemination; the National Institute for Health and Clinical Excellence, which now incorporates the former Health Development Agency; the Health Technology Assessment programme; Health Scotland, which combines the former Health Education Board for Scotland and Public Health Institute for Scotland and the National Library for Health; see Box 8.1) seem to adhere broadly to the tenets of the model. The second reason is that the model’s critique of many aspects of current (‘evidence less’) practice is compelling, as are some of the solutions offered. The following sections will describe the nature of the evidence that the evidence-based decision-maker is urged to use in order to ensure patients and clients receive the most effective health care.

Hierarchy of evidence

Imagine we wish to implement a programme to control head-lice in children. In the first instance the question appears to be a simple one. What is the most effective treatment of head-lice? Clearly, in order to determine the relative merits of any one treatment we would wish to compare it with the alternatives (including non-pharmacological approaches). The treatment that led to the highest number of children being free of head-lice would be the most effective. So a carefully controlled comparison is what we require.

In evidence-based practice, the ideal single study design for answering this form of question is the (well-conducted) RCT. There is a clear and explicit hier-archy of evidence. There is some discussion of the relative merits at the lower end of the hierarchy, but for a single study there is no serious debate as to the superiority of the RCT. Best of all is a systematic review of good-quality RCTs encompassing all of the valid evidence (see below).

A physicist is able to create a vacuum in order to control extraneous factors, such as air resistance, to determine whether the acceleration of a pebble subjected to gravity is the same as that of a feather (it is!); no such parallel vacuum exists in health care interventions. Since in health care we cannot remove extraneous variables (differences in the characteristics of people), the RCT allows variation arising from these factors to be dealt with by distributing groups of people randomly, to different treatment groups. The probability of any differences between groups prior to the intervention is precisely modelled by probability theory, as are individual variations in response to treatments. Statistical-significance testing allows chance variation to be quantified and potentially discounted (certainly over repeated replications). Thus we can account for the play of chance and attribute any differences between groups at the end of the study that are in excess of what might be seen purely by chance to differences in the effects of the treatments people received.

The problem of using non-random groups to compare the effectiveness of interventions is clear when considering a hypothetical comparison of two head-lice treatments. Imagine treatment X is the most expensive pediculocide on the market. If we simply compared those children whose parent bought and used treatment X with those who received the extremely cheap treatment Y it is difficult to determine what caused any observed differences in louse infestation. Treatment X may appear more effective because it is more diligently applied by parents who have invested heavily in treatments. Alternatively, treatment Y might appear more effective because it tended to be bought by parents who thought they ought to treat ‘just in case’, but had in fact seen no lice. The permutations are endless, but just about any pattern of outcomes could be accounted for by explanations other than differential effectiveness in treatments.

Only when treatment X is compared to treatment Y in a well-conducted RCT can differences between the two groups be confidently attributed to the treatment itself. In the paradigm of EBHC, the ideal form of research evidence for answering a question such as this is thus the RCT. This, then, is the type of evidence that decision-makers should seek in order to answer their effectiveness questions.

However, evidence from RCTs is not always available. In some cases random-isation to groups may be practically or technically impossible. In these cases, other comparative designs, such as cohort studies (where outcomes of comparable groups with and without exposure to the intervention are compared), case control studies (where known cases, e.g. those exposed to an intervention or experience, are compared to matched controls) or ‘n of 1’ trials in which an individual (or group) is exposed to one or more interventions intermittently (in random order), may be considered. The natural experiment on head-lice treatment comparing those who simply chose to use different treatments would constitute a (rather poor) cohort study. Studies of disease causation must generally be cohort studies, since randomisation to the causal agents of disease is frequently impossible and almost never ethical. The individual case report, or series of case reports, where the effectiveness of the intervention is illustrated by a change in status before and after exposure to an intervention or agent, with no comparison whatsoever, is seen as evidence, which is so weak as to be negligible in most cases.

Many public health interventions are delivered at a level of service organisation where randomisation of individuals to receive ‘treatments’ would not be possible. For example, if one wished to investigate the impact of an initiative aimed at promoting self-care within a community using multiple sources of dissemination, including distribution of a self-help/self-assessment guide, it would be highly problematic to randomise individuals. The aim of the intervention is to disseminate information within a community. An ‘ideal’ design here would probably be one in which the unit of randomisation was at the level of the community itself (for example, a whole town) and effectiveness was measured in terms of population outcomes rather than at the individual level. However, widespread policy initiatives are often implemented without extensive piloting. In such cases, and in the UK NHS Direct is an example of a national initiative that has many of these features, it may be that the only available comparison is with a historical cohort prior to the implementation of change.

In cases where non-randomised studies are conducted, a judgement on the quality of the evidence produced is generally made upon the extent to which the study has managed to control bias. A study on the effectiveness of primary care trusts (PCTs) might attempt to adjust (in statistical terms, ‘control’) for differences in the age and socio-economic status of individuals in the lists of practices involved in the study.

Complex techniques of design and analysis are deployed to improve the validity of results from such non-randomised (quasi-) experiments (Cook & Campbell 1979). All are essentially designed to control for the effect of extraneous causes that might lead to differences between groups, and thus to increase the confidence with which cause can be attributed to the intervention under investigation. In essence, the endeavour is to make the quasi-experiment as much like the RCT as possible. Thus, alternative study designs represent no shift in the paradigm. Rather they represent best fixes for technical problems that mitigate against the use of an RCT.

Nonetheless, two of the leading authors in the field offer the following observations: ‘With the data usually available for such studies, there is simply no logical or statistical procedure that can be counted on to make proper allowances for uncontrolled pre-existing differences between groups’ (Lord 1967, p. 305 cited in Cook & Campbell 1979). In other words, nothing really beats a good RCT, a point emphasised by Muir Gray (2001): ‘The main abuse of a cohort study is to assess the effectiveness of a particular intervention when a more appropriate method would be an RCT’ (p. 150).

Yet, as we have seen, in some cases the only available design to measure effect iveness is in fact a quasi-experiment. The criticism of the quasi-experiment and associated analysis techniques is offered here largely to establish their inherent imperfection and associated uncertainty. Unlike the true RCT, such uncertainty cannot be precisely quantified and thus the influence of other factors cannot be discounted simply by replication, and reduction of the probability of ‘random’ error accounting for the findings.