59CHAPTER 4

General Design and Implementation Challenges in Outcome Assessment

Ann F. Minnick

1.  Ensure that the outcome assessment (OA) design can meet the project’s purpose

2.  Select outcomes

3.  Maximize the ability to link cause and effect

4.  Select a design that is amenable to resolving analytic quandaries

1.  What five steps would you take to ensure saliency of the outcome?

2.  What three steps will help achieve the necessary qualities of reality and “common currency” for your project?

3.  What seven challenges establish cause and effect in nonexperimental designs?

4.  Considering your outcome of interest, what analytic issues do you anticipate?

5.  What four implementation challenges exist within OA projects? Suggest a solution for each challenge.

The conduct of OA studies is expensive, especially in terms of human resources that might be applied to any number of other important activities. The results of OA projects are 60needed to determine public policies and institutional efforts. Both of these uses of OA results make it imperative that the studies be designed to avoid common design flaws and make parsimonious use of resources during their execution. Simply put, you need to avoid wasting your time and someone else’s money while producing valuable information.

This chapter is based on the assumption that few practitioners want to simply describe a single outcome but rather are trying to devise assessments that will help them improve care in multiple ways. This chapter discusses solutions to the four most common design problems and four implementation challenges to achieving this goal. Recognition of these basic problems and challenges will lead to the discovery of other issues that can be threats to the execution of OA studies. Although the list of potential solutions presented in this chapter is not exhaustive, it is designed to arm the person embarking on such projects with a basic set of effective responses.

  SOLVING DESIGN CHALLENGES

Four common design problems in OA are (a) ensuring that the design can meet the project’s purpose, (b) selecting outcomes, (c) maximizing the ability to link cause and effect, and (d) selecting a design that is amenable to resolving analytic quandaries.

  Linking Purpose and Design

Challenges

The first and perhaps most important step is to determine what question(s) the OA project seeks to answer. Many novices have found themselves implementing a design only to discover that they never determined the specific questions they sought to answer. This occurs most often when clinicians note that some naturally occurring event, such as a change in practice at one site, will result in what seems like experimental and control groups. They then begin to track outcomes, but, because specific questions were never posed, find that they neglected to collect data on some important variable or that the pre- or postintervention design they used cannot really capture the additional ongoing practice changes at the sites.

A second problem in linking purpose and design is the failure to plan a project that could have answered, with only a few design changes, many more questions that are of interest to the larger world of institutional and public policy making. Many practitioners can verbally explain the larger issues for which OAs are needed, but they design studies that do not help inform the important debates over outcomes and how best to improve them. At a minimum, any OA usually needs to include some exploration of patients’ physical and psychosocial outcomes as well as some elements of service costs and impact on the provider.

Solutions

Persons planning to embark on OAs can take the following steps to avoid these two problems:

1.  Write the questions your OA project seeks to answer. Next, answer the question: How will answering each of these questions lead to actions that will improve outcomes for patients, the practice and/or institution, and the public?

612.  For each project question, identify who cares about the answer and the level at which each person/agent functions in terms of making decisions that might influence changes your project might suggest. For example, is it a professional advanced practice registered nurse (APRN) group, the practice manager for your group, or a state agency? Could it be all of them if the design were changed? You will need resources for your study, even if your plans encompass only an assessment of outcomes within your own practice. These people/agencies could be sources of support. The first rule of sales (and gaining support for any type of project) is to meet the customer’s needs. Be sure your project does so.

3.  If in step 2 you could not identify more than one audience of interest, reconsider the questions. OA projects are too expensive to be one-trick ponies. If you identify someone who has resources but who you believe will not be supportive, consider how at least one question of interest to him or her can be included and be answered as part of the assessment. In providing an answer for what the individual or institution may think is the most important aspect of an OA, you will have the opportunity to bring these other questions (and findings) to his or her attention.

4.  Verify through literature review and consultation with persons at each of the specified levels that these are the most important questions. “Important” means those questions that arise because there are great gaps in understanding and for which solutions are most urgently needed.

5.  Seek consultation to ensure that it is possible to design an assessment that produces data that can answer the questions.

  Selecting Outcomes

Challenges

When the preceding five steps are taken, it becomes easier to address the problem of defining outcomes. Each outcome must have three attributes to make a project worth the investment: salience, objectivity, and common currency.

Salience is the quality of being related to the phenomenon of interest. By performing the five steps already mentioned, salience can be achieved.

Objectivity is the ability of an outcome to be measured without bias. For an outcome to be said to be based in reality, it must be one that has the quality of being true to life. One example of a bias problem that is a lack of objectivity is illustrated by a seemingly simple outcome: rehospitalization within 60 days after treatment. In one study, we had to grapple with the bias inherent in defining rehospitalization as having occurred only if it happened at the single hospital where most APRNs had privileges. There was the chance that some patients were being rehospitalized at several other hospitals at which a few of the APRNs also had privileges.

Another example of this problem involves physical restraint use as an outcome. Once the physical restraint is defined, it should be fairly easy to determine if someone is restrained. The issue arises in counting restrained persons. If patients are transported to a unit in restraints, should they be counted against the receiving unit in a project seeking to assess the outcomes of a restraint-reduction program? If not, how long should the unit be given to implement restraint alternatives before patients are counted as restrained? 62Should there be another outcome such as “duration of restraint use for patients admitted in restraint?” How much detail is necessary?

Reality can be defined as the extent to which the outcome definition has some fidelity to nature, that is, is true to life. Depression, quality of life, and spiritual health are examples of outcomes for which there are readily acknowledged problems in capturing the reality of the situation. Other outcomes, although seemingly immune to this problem because they are behaviorally based, are just as vulnerable. Consider the outcome “ambulation sufficient to accomplish five activities of daily living.” If the outcome is operationalized as the ability to do this in a setting assumed to be a one-story home, but many patients live in multistory dwellings, there is little that is true to life about the study because many people need to be able to not only just ambulate but also climb stairs. Resources to consult in the definition and measurement of common outcomes are listed at the end of this chapter. The books listed highlight the advantages, disadvantages, and design issues associated with each approach.

A final problem revolves around what outcome researchers often assume is “common currency” in defining outcomes. For example, if death is an outcome and the performance of numerous hospitals is being measured in the OA, the death rates will be very different in the hospital that includes its hospice unit in the report versus those that do not have such a unit. A hospital may include deaths in the emergency department and another may not. If a hospital is the public receiving facility for the pronouncement of death in police and fire cases, should these deaths be included in the operationalization of the definition of death? Responsible persons at each hospital often believe everyone at other hospitals uses the same definitions for outcomes when in fact there is no common currency.

Solutions

Solutions lie in rigorous definitions:

1.  Each time an outcome is mentioned in the project’s questions, underline it. Within the context of each question, define the outcome in terms that can be objectively applied within the context of the study. It is important that you do this with each question independently. You may find that the outcome you are referring to as “mobility” in question 1 may be very different by question 4.

2.  Discuss your definitions at sites where you plan to conduct the assessment to determine if data are currently amassed using your definition. Ask the responsible parties to describe any special situations they may have that could influence their outcomes, even when their definition is the one you propose. Be prepared to give examples of situations. Remember, most people do not believe their situation is the exception.

3.  Simultaneous with step 2, complete a review to determine what definitions were used in the most important outcome studies published to date. Although you may choose to define an outcome in a new way and may, in reality, be developing a new outcome of interest, an OA is strengthened if there can be some comparison with findings from previous studies. For example, in a study of physical restraints, we measured prevalence and incidence, although earlier studies had relied almost exclusively on the latter. We were thus able to ascertain that the lower usage we documented was in fact a 63decline based on comparisons with earlier reports, as well as demonstrate that there were very great differences between incidence and prevalence. Consult the Agency for Health Care Policy and Research (AHCPR) websites listed at the end of the chapter to learn how outcomes of interest to your project have been defined and measured.

  Tracing Cause and Effect

Challenges

As students of traditional research know, a well-executed, double-blind, randomized pretest–posttest design is effective when one seeks to establish that a particular intervention produces measurable effect(s). In OA projects, this approach is usually not an option because of real-life issues. For example, it is often not possible to randomize patients or blind providers to treatment. It is rare to find a project that seeks to measure only one outcome. The science of improvement drives the desire to identify variables associated with the outcome. Seven challenges to the ability to make conclusions about causation and to identify interventions that might result in outcome improvement are common. These seven challenges are:

1.  Patient autonomy. The patient may be following the recommended treatment on a continuum ranging from “entirely” to “not at all.” The patient may be following one aspect of the treatment entirely and another not at all. The patient may follow a treatment plan one day and not at all the next.

2.  Multiplicity of health problems in a single individual. Almost no patient presents with a single health challenge. Multiple system failures are common and the simultaneous presence of physical and mental disorders has been well-documented. This makes assessment of a single outcome related to a particular disorder difficult.

3.  Nonclinical characteristics. Income, education, insurance coverage, geographical location, exposure to violence, and many other variables can influence outcomes.

4.  Multiplicity of health providers. This includes known as well as unknown providers who, in turn, use many different types of treatments. Some of these treatments may have been obtained from ethnic healers. Some medications may have been obtained illegally. Other providers might be recognized in foreign countries and their advice obtained by the autonomous patient through the Internet. Even if the providers are known, their skill in providing a specific treatment may vary. Depending on the schedule, the patient may have received each treatment in a repetitive series from different providers.

5.  Unknown time delay between intervention and expected outcome. The classic example of this challenge is the difficulty in determining the outcomes of providers’ health-promotion activities because many years (and many intervening messages and experiences) will often pass before a condition manifests itself.

6.  Lack of baseline measurements. Patients often change providers, and accumulating good baseline measures of health status, quality of life, and other variables are expensive to collect de novo. Even if there is support for de novo measures, it is often impossible to collect a full record that captures the rich and complex changes in human life that may influence an outcome.

7.  The complexity of nonpatient, nonindividual provider variables. These variables include labor (overall staffing quantity and quality) and capital inputs (e.g., equipment), 64as well as conditions of employment and leadership. In studies of whether or not a particular activity influences outcomes, these types of variables rather than the activity itself may be paramount. For example, staff may have the same beliefs and knowledge about ways to avoid extubation accidents, but a shortage of supplies or staff may make the execution of these steps impossible. Merely assessing extubations by practice group or before and after an educational session with the staff will not assist in tracing why an outcome is occurring.

Solutions

All of the solutions depend on the outcome assessor having a broad knowledge of patients, providers, and system variables. Consultants for each of these areas during the design phase can be worthwhile. They may ensure that the potential effects of these variables are at least considered.

Through interviews with providers and patients as well as review of clinical documents, such as medical records, determine what the potential is that aspects 1 through 4 may influence the OA. During this process, attempt to determine if these aspects are evenly distributed across cases or if only select groups are influenced. For example, many patients at one clinic site may visit a traditional healer down the street, and patients at another site might not. As with the issue of defining outcomes, patients and providers will not necessarily think that their situations are unique. During the project-planning phase, you will need to ask questions that will provoke a wide-ranging discussion, such as “Tell me about some of the things you do for your arthritis besides coming to the clinic.”

Plan on multiple measurement over time. Multiple measures over time will help to ascertain any change/attenuation of effect on outcomes. This is especially important if the OA is part of an intervention effort. An outcome may at first seem to be favorably influenced, but there may be rapid attenuation. Conversely, it may take an unknown period of time for full effects to be realized.

After reviewing the availability of baseline data, recognize that significant OA resources may need to be assigned to build a database. The project budget needs to reflect this expense. Make it a priority to explore how to maintain the elements of these data after the assessment project is complete. Experience has shown that once providers and institutions have access to such a database, they are willing to devote the resources necessary for its maintenance because a well-designed database can be used for many OA projects, as well as to meet accreditation demands.

Use a framework such as the one in Figure 4.1 to ascertain that you have assessed the system variables that are most likely to influence outcomes. Many times the key to improving outcomes is to attempt to modify system—rather than individual provider or patient—variables. For example, in the past century, anesthesia outcomes were improved significantly when the tubing connection ends of various gases provided during surgery were made compatible only with the appropriate delivery device.

  Analytic Issues

Challenges

As can be deduced from the discussion of the many variables that need to be accounted for in an OA project, multivariate analysis becomes a necessity. Any outcome may be affected by attribute variables, contextual variables, and specific treatment effects. The problem in executing such an approach is that one usually does not know at the beginning of a project if the variables of interest are orthogonal to one another (an assumption of many statistical techniques). This problem is known as collinearity. A second problem is the definition and treatment of attribute and contextual variables. More analytic problems occur when an outcome is rare or infrequent. Finally, the third problem is that data of interest are drawn from different levels. For example, an outcome may be drawn from individual patient records, but variables such as staffing may be unit based with others drawn from an institutional level. Special techniques are needed for the analysis.

65

FIGURE 4.1 A framework of variables influencing patient outcomes.

Source: Developed in preparation of Minnick, Roberts, Young, Kleinpell, and Marcantonio (1997).

Solutions

Few practitioners are equipped to deal with these problems. The following steps are advisable for practitioners who do not have advanced statistical and design expertise:

1.  Recognize what one does not know and consult experts during the design phase. The timing is essential because many of the solutions to these problems are rooted in selecting the proper design and definitions. For example, in the case of rare outcomes, a case–control approach may be advised.

2.  The practitioner should, however, be knowledgeable enough to recognize the possibility that all three problems exist and to ask a statistician how the problems can be addressed. In asking the statistician for advice, the practitioner should inquire about the advantages and disadvantages of each proposed solution.

  ADDRESSING OA PROJECT IMPLEMENTATION CHALLENGES

  Challenges

Having considered the complexity of the design issues described previously, thoughtful practitioners may be tempted to run from the very idea of launching a systematic 66OA. This section is devoted to providing “doable” solutions for the four major implementation challenges: (a) assembling a competent and productive team, (b) securing the resources to complete the project, (c) obtaining institutional cooperation, and (d) enlisting the cooperation of providers and patients.

  Solutions

The solutions are based on the belief that the process of getting this type of project done is no different than the steps one would take in any type of project, from remodeling one’s home to opening a new clinic. You would neither attempt to do either of these projects alone nor would you attempt to go forward without adequate resources.

Stay updated, free articles. Join our Telegram channel

Join