Questions about the effects of interventions: examples of appraisals from different health professions

CHAPTER 5 Questions about the effects of interventions


examples of appraisals from different health professions



Tammy Hoffmann, John W. Bennett, Mark R. Elkins, Craig Lockwood, Angela Morgan, Sharon Sanders, Alan Spencer, Luis Vitetta, Caroline Wright


This chapter is an accompaniment to the previous chapter (Chapter 4) in which you learnt how to critically appraise evidence about the effects of interventions. In order to further illustrate the key points from Chapter 4, this chapter contains a number of worked examples of questions about the effects of interventions. As we mentioned in the preface of the book, we believe that it can be easier to learn the process of critical appraisal when you see some worked examples of how it is done, and it is even better when the examples are from your own health profession. Therefore, this chapter (and Chapters 7, 9 and 11) contains examples from a range of health professions. Some of the clinical examples are relevant to more than one health profession. Each example is formatted in a similar manner and contains the following elements:


• a clinical scenario that explains the origins of the clinical question

• the clinical question

• the search terms and databases used to find evidence to answer the clinical question

• a brief description of the article chosen and the reason for its selection

• a structured abstract of the chosen article

• an appraisal of the risk of bias of the evidence

• a summary of the main results of the article that are relevant to the clinical question

• a brief discussion about how the evidence can be used to inform practice.

You will notice that in most of the examples the type of article that has been chosen to be appraised is a randomised controlled trial. You may wonder why this is the case when Chapters 2 and 4 explained that systematic reviews of randomised controlled trials should be the first choice of study design to answer questions about the effects of intervention. There is a good reason behind this. The authors of the examples that are contained in this chapter were asked to not choose (or indeed, specifically search for) systematic reviews if they were available to answer their question. Why? Because it is easier to learn how to appraise a systematic review of randomised controlled trials if you have first learnt how to appraise a randomised controlled trial. This chapter and Chapter 4 are designed to help you learn how to appraise a randomised controlled trial. Once you know how to do this, Chapter 12 will help you to learn how to appraise a systematic review. As you read through these examples, keep in mind that because the suggestions that the authors of these worked examples have provided in the ‘how might we use this evidence to inform practice’ section have been drawn from only one individual study, in reality, additional studies would need to be located and appraised prior to drawing clear conclusions about what should be done in clinical practice.


When appraising an article, you need to obtain and carefully read the full text of the article. We have not included the full text of the articles that are appraised in the example. However, for each of the examples in this chapter (and Chapters 7, 9 and 11), the authors of the examples have prepared a structured abstract that summarises the article. This has been done so that you have some basic information about each article. As we mentioned in Chapter 1, the more you practise doing the steps of evidence-based practice, the easier it will become. This is particularly true of the critical appraisal step. You may find it useful if you approach these worked examples as a self-assessment activity and try and obtain a copy of the article that is appraised in each of the examples (or just the ones that are relevant to your health profession if you feel more comfortable with that). You can then critically appraise the articles for yourself and check your answers with those that are presented in the worked examples.


One other thing to note about the examples in this chapter (and Chapters 7, 9 and 11) is that the appraisal of articles is not an exact science and sometimes there are no definite right or wrong answers. As with evidence-based practice in general, the health professional’s clinical experience has an important role to play, particularly in deciding about issues such as baseline similarity (as we saw in Chapter 4) and clinical significance (as we saw in Chapters 2 and 4). Some of the examples may contain statements that you do not completely agree with and that you, as a health professional, would interpret a little differently. Also, the examples are provided to give you an overall sense of the general process of evidence-based practice. The content that is presented in the examples is not exhaustive (particularly in the ‘how do we use this evidence to inform practice’ section) and there may be other factors or issues that you, as a health professional, would suggest or consider if you were in that situation. That is OK.



Occupational therapy example





Clinical scenario


You are an occupational therapist who works in a recently opened community adult health centre. In the last few weeks, you have received referrals for a number of clients who have been diagnosed with rheumatoid arthritis in the last one or two years. Most of these clients are female and middle-aged and their arthritis is causing them considerable pain (particularly in the hand) and affecting their ability to perform self-care activities. You are considering running an education group about joint protection techniques and wonder if this will be effective in reducing your clients’ pain and improving their function.



Clinical question


In adults with rheumatoid arthritis, is education about joint protection techniques effective in reducing hand pain and improving function?



Search terms and databases used to find the evidence


Database: OTseeker


Search terms: ‘rheumatoid arthritis’ AND ‘joint protection’


This search retrieved nine results. Only two of these articles specifically evaluated the effectiveness of a joint protection program, with the remainder evaluating the effectiveness of general patient education programs for people with arthritis. The two most relevant articles were both based on the same study, with one describing the 6- and 12-month results, and the other reporting the long-term (4-year) effects of the intervention. You initially choose to appraise the earlier article (the 12-month results) and, if the quality and results of this article are promising, you will then also look at the article that reports the long-term results.



Article chosen


Hammond A, Freeman K. One-year outcomes of a randomised controlled trial of an educational–behavioural joint protection programme for people with rheumatoid arthritis. Rheumatology 2001; 40:1044–1051.



Structured abstract


Study design: Randomised controlled trial.


Setting: Outpatients from the occupational therapy departments of two hospitals in the UK.


Participants: 127 patients with rheumatoid arthritis; aged between 18 and 65 years (mean age 50.5 years, 76% female); diagnosed with rheumatoid arthritis within the last 5 years, a history of wrist or metacarpophalangeal joint pain and inflammation, and experiencing hand pain during activity. Exclusion criteria included no other medical condition that affected hand function.


Intervention: Two small-group education interventions, both of 8 hours duration, were compared. One group attended a standard arthritis education program, which included 2.5 hours of joint protection education. Approximately 15–45 minutes was spent practising joint protection techniques. The other group attended a joint protection education program that used educational–behavioural teaching methods and aimed to enhance self-efficacy and motor learning. Participants in this program spent about two-thirds of their time practising and receiving feedback about hand joint protection methods.


Outcomes: Primary measures: hand pain experienced during a moderate activity (such as cooking or housework) within the last week (measured using a 100mm visual analogue scale); adherence with joint protection. Secondary measures: functional status (measured using the Arthritis Impact Measurement Scales [AIMS2] with a range of 0–10 where 0 indicated better function), indicators of disease severity, hand status and psychological status.


Follow-up period: 12 months. Assessments were performed at baseline, 6 months and 12 months.


Main results: Compared to participants in the standard group, participants in the joint protection group demonstrated significant improvements in terms of adherence to joint protection, hand pain, general pain, early morning stiffness, self-reported number of disease flare-ups, visits to the doctor for arthritis and the AIMS2 activities of daily living (ADL) scale. Both groups experienced an increase in hand deformity scores.


Conclusion: Among those who attended the joint protection program, significant improvements were found in adherence, pain, disease status and functional ability. It is suggested that joint protection can help to slow the progression of the effects of rheumatoid arthritis, over and above the effects of drug therapy, as the benefits became more apparent with time.



Is the evidence likely to be biased?



• Was the assignment of participants to groups randomised?

Yes. Participants were randomly allocated, using a four-block sequence.


• Was the allocation sequence concealed?

Yes. Allocation occurred using sealed envelopes that had been prepared in advance.


• Were the groups similar at the baseline or start of the trial?

Yes. The baseline characteristics were similar between the two study groups. There is a small difference between the two groups in terms of steroid use (6% of participants in the standard group and 20% of participants in the joint protection group) but this difference is probably not large enough to have affected the results. The baseline scores of the outcome measures were also similar between the two groups.


• Were participants blind to which study group they were in?

No. For this trial, it was not possible for participants to be blinded to group allocation.


• Were the health professionals who provided the intervention blind to participants’ study group?

No. For this trial, it was not possible for the health professionals who provided the intervention to be blinded to group allocation.


• Were the assessors blind to participants’ study group?

Yes, some for measures. Baseline, 6- and 12-month assessments were conducted by an independent assessor who was not informed of group allocation. The assessor was also asked to avoid discussing the education programs with the participants. However, for the outcomes that were measured by participant self-report, the assessment of these outcomes was not blind as participants were not blinded to group allocation.


• Were all participants who entered the trial properly accounted for at its conclusion and how complete was follow-up?

Yes. The follow-up rate was 95.3% at 6 months and 96.8% at 12 months. Reasons are given as to why some participants were not able to be followed up, as well as which group they were in.


• Was intention-to-treat analysis used?

Cannot tell. The authors do not explicitly state that an intention-to-treat analysis was conducted. Therefore, we cannot tell if this was done and so the results could be subject to bias because of this.


• Did the study have enough participants to minimise the play of chance?

Yes. Based on data from a previous study, the authors conducted a power analysis and determined that, with a power of 80% and a significance level of 0.05, 63 participants would be needed in each group.



What are the main results?


For hand pain at 12 months, the effect size is 12.98, which is a statistically significant result. Both the p-value (0.02) and confidence interval (CI), which does not include the no effect value of zero, show this. As hand pain was measured using a 100mm visual analogue scale, an effect size (between groups) of 13 is probably large enough to also be considered clinically significant. However, the confidence interval is very wide indicating some imprecision in the result. Thus, for some people the intervention effect may not be large enough to be considered clinically significant, whereas for other people the effect could be quite large and deemed clinically significant.


With respect to function, the ADL subscale of the AIMS2 was used to measure participants’ ability to perform self-care and household activities. The effect size for the ADL subscale at 12 months was 0.8, which is statistically significant (confirmed by both the p-value and the confidence interval). However, as the AIMS2 is scored on a scale of 0–10, an effect size of 0.8 is small and may not be considered by many to be a clinically significant result. However, if



TABLE 5.1 Hand pain and functional outcome for intervention and control groups at 12 months

image

we consider that the upper end of the confidence interval is 1.57, it is possible that the effect of this intervention could be clinically significant for some people. As explained in Chapter 4, the decision about clinical significance is a subjective one and depends on a number of factors such as the costs involved and preferences of the individual concerned.



How might we use this evidence to inform practice?


As you are reasonably confident about the validity of the study’s results and some of the results were of clinical importance, you proceed to assessing the applicability of this evidence to your clinical scenario. The clients with rheumatoid arthritis that you see are similar to the study participants in a number of ways such as age, gender (predominantly female) and disease duration. The majority of the participants in the study were assessed as having mild or moderate rheumatoid arthritis. You have yet to assess the disease severity in your recently referred clients. Once you do, you will check if their disease severity is comparable to the study participants’ before making a decision about whether to implement a joint protection program.


Before making the decision, you will also need to consider whether you and the health centre where you work have the resources available to offer a joint protection program such as the one that was evaluated in the study. You will also need to obtain further, more detailed, information about the content of the program and the teaching strategies that were used. The article mentions that further details may be obtained in another published article, so you will start by obtaining that article and, if you have questions after reading that, you will contact the authors of the article for more information.


Although the study found a statistically and clinically significant effect of the intervention on hand pain, the effect on the other outcome (self-care) that you were particularly interested in may not be conclusively clinically significant. Although it is unlikely that the program would cause any harm and it appears to offer some benefits to participants, there are considerable staffing resources associated with providing the program. You decide to obtain the full text of the article that reports the long-term effects of the joint protection program before making a decision about whether to implement the program at your workplace.



Physiotherapy example





Clinical scenario


As a physiotherapist on the cardiac surgery ward, you attend a conference and hear that some of your peers at another hospital are providing inspiratory muscle training to clients who are undergoing coronary artery bypass graft (CABG) surgery. The training is performed preoperatively in an attempt to reduce the likelihood of pulmonary complications occurring postoperatively. You are unaware of any research that has investigated the use of this intervention in clients undergoing cardiac surgery and decide to search for evidence to decide whether this is something you should institute at the hospital where you work.



Clinical question


Does preoperative inspiratory muscle training in addition to standard care prevent postoperative pulmonary complications in clients who are undergoing CABG surgery?



Search terms and databases used to find the evidence


Database: PEDro (using the ‘Advanced Search’ option)


Search terms: You decide that inspiratory muscle training is likely to be mentioned in the title of the article, although it may be described as respiratory muscle training. You therefore enter ∗spiratory muscle training in the ‘Title’ field. You consider that the patient population might be broadly defined in the title using a phrase like cardiac surgery, but you expect that it would be defined using coronary artery bypass graft or coronary artery bypass surgery in the abstract. Therefore you enter coronary artery bypass in the ‘Abstract & Title’ field. You want both of these terms to be present, so you select the option to ‘Match all search terms’ when searching.


The search returns six records, but this represents only two separate trials as each of these trials had related publications that were retrieved by the search. For the first trial by Hulzebos et al, there is the main publication which is accompanied by a report of pilot data, an English translation of the primary Dutch publication and an analysis of the feasibility of the intervention using data from the trial. For the second trial, there is the main publication and a Hebrew to English translation. Although both trials examine inspiratory muscle training for clients who are undergoing CABG surgery, several features of the main publication of the first trial show that it probably provides less-biased evidence to answer your question. Unlike the second trial, it used concealed allocation, blinded outcome assessment and intention-to-treat analysis, and fewer participants were lost to follow-up. In addition, the sample size of the trial was larger (N = 279 vs 84) and the trial was conducted more recently, so the surgical procedure and standard care in the postoperative period are more consistent with that offered at your hospital. Therefore you decide on the first article.



Article chosen


Hulzebos E, Helders P, Favie N et al. Preoperative intensive inspiratory muscle training to prevent postoperative pulmonary complications in high-risk patients undergoing CABG surgery: a randomised clinical trial. JAMA 2006; 296:1851–1857.



Structured abstract


Study design: Randomised controlled trial.


Setting: University medical centre in Utrecht, the Netherlands.


Participants: 279 clients (mean age 66.9 years; 77.9% male) undergoing CABG surgery at high risk of developing postoperative pulmonary complications, indicated by the presence of two or more of these criteria: age >70 years, productive cough, diabetes mellitus, smoking, chronic obstructive pulmonary disease and body mass index >27. Exclusion criteria included: surgery within 2 weeks of initial contact; a history of stroke; use of immunosuppressive medication for 30 days before surgery; and presence of a neuromuscular disorder, cardiovascular instability or an aneurysm.


Intervention: Participants were randomly assigned to receive either preoperative inspiratory muscle training (n = 140) or usual care (n = 139). The intervention group trained daily (20 minutes), seven times a week (six times without supervision), for at least 2 weeks before the surgery. Both groups received the same postoperative physical therapy and other standard care.


Outcomes: Primary outcome: the incidence of postoperative pulmonary complications, defined according to recognised criteria. Secondary outcome: duration of postoperative hospitalisation, in days.


Main results: Postoperative pulmonary complications occurred in 25 (18%) of the participants in the intervention group and 48 (35%) of the control group, odds ratio 0.52 (95% CI 0.30 to 0.92). Median duration of hospitalisation was 7 days (range 5–41) in the intervention group and 8 days (range 6–70) in the control group (p = 0.02).


Conclusion: Preoperative inspiratory muscle training reduced the incidence of pulmonary complications and the duration of postoperative hospitalisation in clients who were at high risk of developing a pulmonary complication after CABG surgery.



Is the evidence likely to be biased?



• Was the assignment of participants to groups randomised?

Yes, participants were appropriately randomised, with a computer-generated number list.


• Was the allocation sequence concealed?

Yes, the number list was sealed in envelopes which were held by an external investigator.


• Were the groups similar at the baseline or start of the trial?

Yes, the baseline characteristics were similar between the two study groups. Although in the table of baseline characteristics the authors of the article have reported a difference between the two groups in terms of median duration of mechanical ventilation (4h in the intervention group vs 5h in the control group), this is technically not a baseline characteristic as it was measured after the intervention had been provided. Additionally, it is unlikely that a difference of this size would be of clinical importance and have affected the results.


• Were participants blind to which study group they were in?

No. For this trial, it was not possible for participants to be blinded to group allocation.


• Were the health professionals who provided the intervention blind to participants’ study group?

No. For this trial, it was not possible for the therapists who provided the intervention to be blinded to group allocation.


• Were the assessors blind to participants’ study group?

Yes. The investigators who assessed outcomes were blinded to participants’ treatment group.


• Were all participants who entered the trial properly accounted for at its conclusion and how complete was follow-up?

Yes. Apart from three participants who died before surgery, all participants were followed up. This is a 98.9% follow-up rate.


• Was intention-to-treat analysis used?

Yes, it is stated that an intention-to-treat analysis was conducted.


• Did the study have enough participants to minimise the play of chance?

Yes. A power calculation was performed but the trial was terminated before this number was reached because safety monitoring determined that statistically and clinically significant results had been achieved at the interim analysis.



What are the main results?


Pulmonary complication: The risk of pulmonary complications is presented appropriately, using an odds ratio with a 95% confidence interval (refer to Table 5.2). The reduction in risk is both statistically and clinically significant. The 95% confidence interval includes only clinically worthwhile reductions in risk. Therefore, the results are sufficiently precise to make clinical recommendations.


This data for the first outcome, pulmonary complications, can be used to estimate two useful statistics. First, let us calculate the absolute risk reduction (ARR), which is simply the risk in the control group minus the risk in the intervention group: 35% – 18% = 17%. Using the formula (95% confidence interval ≈ difference in risk image) that was provided in Box 4.3 in Chapter 4, you also calculate the 95% confidence interval of the ARR to be 9% to 25%. The ARR statistic is useful in clinical practice because, if you decide to implement the intervention, you can use it to explain to clients the value that they can expect from undertaking the inspiratory muscle training regimen. After explaining what a pulmonary complication is and how it can delay recovery, many clients would consider the training program worthwhile to reduce their risk of such a complication from 35% to 18%.


An alternative statistic is the relative risk reduction (RRR). This is calculated by dividing the difference in risk between the two treatment groups, by the risk in the control group:



image



When an adverse outcome occurs fairly frequently in a study, the RRR is a useful statistic, but when the outcome is rare, a large RRR may still be found even though the ARR is very small, which can be misleading. Therefore you decide to use ARR instead of RRR.


You then use the absolute risk reduction to calculate the number of people that need to be treated in order to prevent one pulmonary complication.



image



The 95% confidence interval of this number needed to treat is 4 to 12. This was calculated by using the inverse of the numbers in the confidence interval of the absolute risk reduction that you calculated earlier (so 1/25 and 1/9). Therefore, for every six high-risk clients that we treat, one pulmonary complication will be prevented that would have otherwise occurred with usual care. However, this number needed to treat could be as low as four or as high as 12.


Duration of hospitalisation: The median duration of postoperative hospitalisation was 7 days (range 5 to 41) in the treatment group and 8 days (range 6 to 70) in the control group, and this difference was statistically significant.



How might we use this evidence to inform practice?


You decide that the trial is valid and that the results are important. Pulmonary complications are dangerous and uncomfortable for the client and they are expensive to treat. Therefore, the effect of this intervention on this outcome alone is clinically worthwhile, and the number needed to treat of six is useful in justifying the introduction of the service. Further justification of the clinical worth of this intervention comes from the other significant outcome, which was a reduction in the duration of hospitalisation. You plan to discuss the results of this study with your head of department as there are resource implications associated with introducing the service, particularly providing an intervention of the same intensity as was provided in the study. However, it is your recommendation that this intervention should be introduced.



Podiatry example





Clinical scenario


You are a podiatrist who works in a private practice. In recent days, you have seen several clients with heel pain which you believe is plantar fasciitis. Your management of this condition usually involves prescribing an orthotic device, which is made in an orthotic laboratory from a cast of the client’s foot. Because it takes several weeks for the orthotics to be made, you wonder whether prefabricated orthotics which can be bought over the counter and started immediately are as effective. You decide to search for the evidence.



Clinical question


In clients with plantar fasciitis, are prefabricated orthotic devices as effective as orthotic devices made from a cast of the client’s foot and using measurements specific to the client in reducing pain and improving function?



Search terms and databases used to find the evidence


Database: The Cochrane Database of Systematic Reviews, the Database of Abstracts of Reviews of Effects (DARE) and the Cochrane Central Register of Controlled Trials (CENTRAL) in The Cochrane Library. A systematic review of randomised controlled trials or a randomised controlled trial would be the ideal study design to answer this question. Ideally the randomised controlled trial would have 3 arms, allowing comparison of the two types of orthotic devices with each other and a control group.


Search terms: You start by checking for MeSH terms related to the population and intervention of interest and combine these with textword terms using the Boolean operator OR in the ‘Search History’ and ‘Advanced Search’ features of The Cochrane Library.


(Fasciitis, plantar (MeSH) OR (heel OR calcan∗ NEAR spur) OR (plantar OR heel OR calcan∗ NEAR pain) OR plantar fasciitis) AND (orthotic devices (MeSH) OR orthotic OR orthoses)


This search produced 23 hits, including 4 Cochrane Reviews, 18 trials and 1 economic evaluation. One of the Cochrane Reviews titled ‘Interventions for the treatment of plantar heel pain’ looks like it may address your question. Reading through the review you find it included only one trial evaluating the impact of orthoses and that this trial compared orthoses with stretching exercises. You also notice that the search for studies was only conducted up to 2002. You go back to your search to see if there are any trials published subsequent to the review that may answer your question. After looking over the titles and abstracts of the trials you are drawn to two articles that compare the effectiveness of different types of orthotic devices. They are both randomised trials conducted in patients with a diagnosis of plantar fasciitis. You select the trial titled ‘Effectiveness of foot orthoses to treat plantar fasciitis’ to read as it had a longer follow-up (12 months instead of 2 months) and looks at the effect of orthotic devices on both pain and function.



Article chosen


Landorf KB, Keenan A, Herbert RD. Effectiveness of foot orthoses to treat plantar fasciitis. Arch Intern Med 2006; 166:1305–1310.



Structured abstract


Study design: Randomised controlled trial.


Setting: A university podiatry clinic, Melbourne, Australia.


Participants: 136 participants, mean age 48 years, 67% female, with a clinical diagnosis of plantar fasciitis who had experienced symptoms for at least 4 weeks. People with a major orthopaedic or medical condition that may have influenced the condition were excluded.


Intervention: Participants were randomised to receive either: a) ‘sham’ orthoses which were made of soft foam moulded over an unmodified cast of the foot; b) prefabricated orthoses made from a thicker, firmer density foam moulded over the cast; or c) customised foot orthoses made from semi-rigid polpropylene moulded over neutral position plaster casts with a firm foam heel post.


Outcomes: The primary outcome of the trial was self-reported pain and function at 3 and 12 months, which were measured using the pain and function domains of the Foot Health Status Questionnaire (0–100 measurement scale).


Follow-up period: 12 months.


Main results: This trial found that pain and function improved in all 3 groups over time. Participants receiving prefabricated and customised orthoses showed greater improvement in function than the sham group at 3 months (mean difference of 8.4 points on the function domain between prefabricated and sham orthoses and a mean difference of 7.5 points between customised and sham orthoses). Prefabricated and customised orthoses also reduced pain compared with sham orthoses at 3 months (8.7 points and 7.4 points, respectively), though these differences were not statistically significant. At 12 months, there were no significant differences in pain and function between any of the 3 groups.


Conclusion: Both prefabricated and customised orthoses have similar small short-term benefits for people with plantar fasciitis and negligible long-term effects.



Is the evidence likely to be biased?



• Was the assignment of participants to groups randomised?

Yes. The randomisation sequence was generated using an appropriate method (computer-generated randomisation sequence).


• Was the allocation sequence concealed?

Yes. The allocation was concealed from participants and the investigator enrolling participants in the trial. The allocation sequence appears to be held off-site and, once a participant was enrolled and baseline assessments were completed, the allocation sequence was obtained by telephone or email.


• Were the groups similar at the baseline or start of the trial?

Related posts:

  1. Introduction to evidence-based practice
  2. Evidence about diagnosis
  3. Finding the evidence
  4. Talking with clients about evidence

Stay updated, free articles. Join our Telegram channel
Tags:
Mar 21, 2017 | Posted by in MEDICAL ASSISSTANT | Comments Off on Questions about the effects of interventions: examples of appraisals from different health professions

Full access? Get Clinical Tree
Get Clinical Tree app for offline access