Evaluation methods




  • Circle 1 represents the absence of health problem(s), treatment emphasis is condition prevention.
  • Circle 2 represents the presence of a health condition(s), treatment emphasis is condition management.
  • Circle 3 represents the Advance Illness stage, treatment emphasis may shift to palliative and/or end of life care.

Provider involvement can occur in more than one level, as represented by the overlap in the diagram’s health status level circles. These areas of overlap are the opportunities for care coordination and integration. The bottom of the diagram shows the patient/care recipient’s typical location or setting. These settings and the transitions between them add complexity to the involvement of providers and communication between them. The setting may also influence the choice or availability of providers. Changes in setting can be disruptive to continuity of care. The levels and types of care are affected by financing and regulation, the organization, and other environmental contexts. These effects, though present, are not shown.


To illustrate an application of the framework, consider an individual having no notable health problems. The “care” needed might involve a combination of prevention education, self-management, and primary health care. If the individual is diagnosed with a health condition, they progress to Level 2, where condition management becomes a goal. Primary care provision continues as previously, but specialists and episodic acute care may emerge. Within this level, we expect the involvement of two or more medical providers, and others who assist in treatment adherence and self-management. If physical function and/or mental or cognitive limitations are involved, then assistive and various non-medical services may be required to facilitate activities of daily living (ADL), instrumental activities of daily living (IADL), and social needs. For some individuals, communication and treatment adherence, and training and support may begin to involve informal care providers. Most of the Level 2 condition management population resides in community settings, although recipients may be living in congregate or group settings, or nursing homes. Transitioning between these settings and with hospital care is an especially critical time.


As an individual’s health status progresses to Level 3, care may shift from active management of the health conditions to one of palliation (for example, symptom management, patient comfort and pain management; and reductions in restorative treatment). Social and other supports to the recipient and the family may intensify. Advanced illness care may continue, but some or all care may be transferred to providers more oriented to supportive assistance. Locations of care might include one’s home, hospitals, assisted living, nursing homes, and inpatient hospice settings.


Transitions between levels of care, providers, and/or settings are especially critical occasions. One function of care coordination at these times is assuring communication among providers so as to minimize disruption in the continuity of care. Some information exchange may require physicians coordinating directly with each other, or nurses serving as a communications bridge and coordinator between various medical providers and family members or care recipients. Functions such as social support, accessing community resources, or alternative funding might be best served by social workers.


For any given patient/client, it is likely that multiple elements of care coordination operate concurrently. These may or may not be coordination with each other, and when not, can lead to duplication of effort, contradictory actions, and increase program cost. Another potential complication can occur when there is incentive for cost shifting between payers or organizations, such as when care provision involves transferring or sharing the cost of care. Examples of this include policies in which providers have constrained revenue from the same payer source (for example, hospitals with diagnosis-related or bundled rates for procedures; nursing homes with fixed daily rates; and home health agency maximum days of care). Such payments are at once an incentive for operational efficiency within the care coordinator’s organization and a motivation to move the patients to another provider and payer system. Cost shifts are problematic if they reduce the quality of care or if they transfer the cost to systems which do not have sufficient resources to afford this responsibility. Understanding the incentives influencing care coordinator actions is needed when evaluating coordination performance across the continuum of care.


Centers for medicare & medicaid services integration of care demonstration activities


Outcome and operational effectiveness research and its incorporation into policy, program guidelines, and financing is distributed among a number of federal and state government agencies, and other task forces and panels.3 In this section, discussion is limited to Medicare and Medicaid, programs administered by the Centers for Medicare & Medicaid Services (CMS).4 Many CMS research and demonstration efforts are mandated by legislation, and consequently offer a window into national policy and programs. Typically, budget neutrality is the CMS focus for new programs or benefits. This means that the innovations cannot cost more than usual care for the same type of recipients. These studies may also be concerned with issues like access to care, beneficiary outcomes, and other aspects of operational performance.


CMS programs finance health and long-term care for some of the most vulnerable populations in the United States (CMS 2005). For example, half of all Medicare spending is attributable to 6% of Medicare beneficiaries and Medicaid pays for more than half of all nursing home care. Given these disproportionate expenditures, identifying those most in need is potentially helpful in achieving budget neutrality. Care coordinators using screening assessments are a common approach to finding these individuals.


Consumer choice is a second perspective influencing CMS practices. This is reflected in such things as data systems that provide consumers with information about the quality of care in hospitals and nursing homes; beneficiary opportunities to change providers and health plans; and consumer directed care, particularly personal care services for persons with disabilities. Care coordination can come into conflict with consumer choice. For many families and individuals, care coordinators may need a balance between being a gatekeeper versus being a coach, educator, and facilitator.


A third characteristic of CMS efforts was noted earlier; they test the efficacy (including budget neutrality and recipient outcomes) of new interventions, and they may document practices and operational efficiencies. These research functions may be distributed between independent evaluators and governmental units that have program oversight.


Programs, such as Medicaid home and community-based service waivers, are generally approved and reauthorized for extended periods. This affords the state and provider the opportunity for incremental operational refinement. Such adjustments and developmental evolution are much rarer in national demonstration projects. We come back to this issue later.


Care coordination tasks and functions


The care coordination innovations and practices of most interest in this book are those attempting to obtain efficiency within levels of care (for example, hospitals, nursing homes, home care), and/or in the transitions between levels of care, providers, or between financing systems. Such work may have goals like improving access to the services offered at one level of care (including primary care, disease management, home care, and so on), minimizing the use of more intensive and/or expensive care (such as hospital care), and reducing practice variation and costs.


Program staffing and practices are basic to operational efficiency. We use this to illustrate delineation of tasks and functions into researchable elements. Each care coordination function has specific staffing considerations such as the number of care coordinators, their professional training, and the average time needed when working with individual clients. Clarity on these specifics and whether care coordinators have multiple functions that combine two or more roles is vital in cross program comparisons. However, many studies and program descriptions compress the issue into staff to client ratios –often without adjustment for full time equivalence in each role.


The complexity of unbundling roles is illustrated by a few of the basic care coordinator functions. We start with the role of gatekeeper into program or benefit eligibility. This function is common in assessing (and reassessing) home care services and nursing home needs. It also applies to disease management and other benefits where enrollment is targeted to specific levels of vulnerability. Assessment may be coupled with care planning, benefit authorization, and recipient training. These functions may also involve consultation with care teams, or other health care providers.


Reductions in cost and practice variation in care coordination have been another program evaluation target. Such efforts have contributed to the adoption of electronic assessment instruments, and algorithms that translate measured needs into care plans, benefit authorizations, and/or dollar allocations. How good are the studies that provided the basis of these procedures? This issue warrants further investigation. To the extent that standardized procedures have been adopted, have the expected efficiency and effectiveness been achieved? How has the saved care coordinator time been reallocated?


Another care coordination function is communication (for example, treatment plans, tests/laboratory results, medications) between the providers responsible for the interfacing levels of care, and between providers and the care recipients. Documentation of these activities is often in the form of recipient care record entries. Activities are likely to be variable based on the particular needs of the recipient and the circumstances relative to providers and other factors. Simple measures are frequently used rather than chart level specifics. These may record the presence or absence of care coordinators (and perhaps their case loads) in specific roles: hospitalists in acute care; teams involved with a patient’s transition from hospital to nursing home care; multidisciplinary teams involved with those in nursing homes, community settings, or in advanced illness care; and social service support to physicians in addressing vulnerable populations in community settings.


In these situations, the care coordinator(s) action may be treated as a “black box,” in which clients are either exposed to this service or not. This approach to examining the effective components that make up the black box is often sufficient for a first stage answer to the question of whether these programs achieve program goals such as reducing preventable hospital stays/days and emergency room use. Simple refinements to such questions may assess whether the outcomes were associated with the professional background or experience of the care coordinator, and whether the reimbursement methodology provided incentives influencing provider behavior.


A drill down into chart-based processes and intensity of care might be warranted to understand performance differences among individual staff or across programs. Such information might include the number, time taken, and nature of specific encounters; the information transferred and to whom; and whether the information was used. Process of care information is expensive, and usually more appropriate for continuous quality improvement than cross program comparisons.


Training and/or facilitating the patient or care recipients (including family members) in fulfilling their self-care or treatment adherence is another typical care coordinator function. Training might include how to modify personal behaviors (such as smoking, diet, exercise), manage chronic diseases (for example, medication use, symptom monitoring), or help the care recipient gain access to benefits provided by another organization or another unit within the organization.


As in the prior example, the provision of training and facilitation assistance is likely to be implicitly assumed (given a care coordinator’s professional training and experience), rather than measured directly. Patient/client outcomes assumed to be attributable to training can be directly measured. In-depth documentation of actions and content would usually be more appropriate for continuous quality improvement purposes, but the introduction of specific training might warrant direct study.


A last consideration in this outline of care coordination functions is the reminder that the effectiveness of care coordination is not isolated from the financial incentives affecting provider behavior. These take a variety of forms: risk-adjusted capitation reimbursement for health plans and providers; fixed payment for hospital stays and selected surgical procedures; other fee-for-service budget caps and bundled payment rates; and recipient co-payments and deductibles.


Capitation and prospective payment have usually been limited to specific levels or types of care, and a single source of payment such as Medicare or Medicaid. Examples include health maintenance organization or managed care plans where Medicare or Medicaid coverage (but not both together) is capitated. Long-term care has usually been excluded from the capitation rates. Among Medicare or Medicaid managed care members, those entering long-term care (especially nursing homes) may be required or encouraged to withdraw from the managed care plan.5


The partitioning of Medicare and Medicaid expenditures and service access seems to be changing. CMS, states, and foundations have begun to give increased attention to the expenditures of those dually eligible for Medicare and Medicaid. This includes both studies of those dually eligible and the databases compiling expenditures in each individual program and total expenditures. Among other things, these databases permit consideration of utilization and costs for those entering, and those within long-term care systems.


Looking ahead


The CMS research and demonstration programs highlighted above, and those described elsewhere in this book, are illustrative of state of the art approaches in the development and testing of care coordination policy and practices. These examples address many dimensions of the continuum of care. They also suggest some underlying perspectives likely to be reflected in program and policy design going forward. Key among these are support for consumer choice, the targeting of benefits and strategies to identified vulnerable populations, and an interest in using payment incentives (such as capitation payment) for utilization control and to influence provider behavior. Substantively, disease management, transitional care, and programs that begin to link Medicare and Medicaid financing across the continuum of care for those at risk/or receiving long-term care seem to be enduring or emerging priorities.


Care coordination practices have been operationally transformed and extended by states and provider organizations over the years. This has occurred in spite of a number of negative or ambiguous national evaluation findings that failed to demonstrate cost effectiveness (that is, budget neutrality). Care coordination’s face validity for facilitating communication between providers and across levels of care has trumped these studies. This is a good thing, as factors other than ineffective clinical operations have contributed to negative findings.


Some of these involve methodological issues, like failing to prevent the contamination of comparison group samples or provider behavior, or small sample sizes. There are also issues of program maturity, or more precisely, evaluating programs before they have attained steady state performance.


Program maturity


Program maturity has multiple elements. Understanding and addressing these elements is an important next step in attaining an effective and constructive care coordination evaluation program. One problem is illustrated by the lifecycle of national demonstrations in which providers and the program evaluators compete for these time-limited initiatives. Even though the organizations winning these awards have experience in the demonstration topics, they are still challenged to recruit recipients (and comparison groups), build out the intervention, launch it, and begin to immediately test program effects and efficiency. All of this allows very little time for the providers to refine their procedures and practices, even if the evaluation design permitted modification.


The time constraint also truncates the program exposure period for observing program effects. Outcomes measured after just a few months, 12 months, or at most a couple of years in the program, are typical. This limitation, combined with program immaturity, is double jeopardy against the likelihood of observing favorable program effects. A way needs to be found for incremental evaluations of innovations. This would allow for maturation, and protracted intervals for assessing outcomes.


State, community, health plan, and other on-going programs (including Medicaid waiver programs) are existing resources that address many of these limitations. Usually waiver programs do not have tight time constraints affecting operational start-up and performance monitoring. As a consequence, these programs have the opportunity for refinement and revision with experience. However, these opportunities may not be routinely taken. Moreover, these programs may not have a strong, if any evaluation component. Both these factors limit the documentation of the program effects.


Lessons from the field


The dynamic nature of care coordination practices, paradoxically, both complicates the collection of information about clinical and efficiency outcomes and argues for a systematic program to evaluate the operational procedures and tools as they are implemented. The experience of prior care coordination evaluations offers lessons for resolving this paradox and moving the practices forward.


One of these lessons is that evaluations with aims to refine clinical practice, reduce practice variation, and improve operational efficiency are of central importance. These have immediate value to basic operations and budgeting. Moreover, even if a program goal is testing efficacy, understanding how it was implemented is fundamental to determining the circumstances when the program is most effective.


Lesson two is that every operating program makes a determination about the resources committed to the ongoing analyses of program performance and cost effectiveness. These likely total vast sums nationally. Can these be more effectively used or even pooled? Perhaps it is time for a more systematic collaboration among programs. Current ad hoc efforts, such as the 2010 summit6 of recognized experts in this field convened by the Gerontological Society of America, is illustrative of the recognized value of information sharing. This one-day effort also illustrates the early developmental stage of these collaborative efforts. The proposal made here is to go beyond the “best practice” and informal communications among programs to formulate guidelines and standards relative to evaluation tools and cross-program reporting. Conferences may not be sufficient. An ongoing body or membership collaborative may be required to systematically pursue issues. Examples include:



  • Standards for measuring risk, resource use, determining costs, and other outcomes are needed. This will facilitate the understanding of evaluative information from one setting to another.
  • Cost-effectiveness analysis also depends on the selection of appropriate programmatic or practice alternatives to which innovations or programmatic changes are being compared. A clearinghouse on operating and conceptual alternatives, and secondary data sets tracking performance of these approaches, are possible resources. Work groups that identify and recommend appropriate comparisons are another possible resource.
  • Further, because procedures evolve in clinical practice, evaluations should be revisited periodically after the innovations have been implemented. This notion applies to both individual programs and program models that may be in place in multiple locations. Periodic studies and exchanges of operational procedures, outcomes, and cost effectiveness could provide a helpful feedback loop in refining practice. This work could occur serially within the same programs or as planned variations across programs. Each of these could be in different stages of maturity or operating in different circumstances.
  • A collaborative, planned variation approach also allows the experience of programs serving different high-cost, high-prevalence conditions to be compiled and shared in evaluating clinical performance and costs. Similarly, diverse populations of patients from a variety of practice settings can be incorporated into a shared database encompassing a broad range of health outcomes.

Lesson three is that evaluation designs, particularly those involving RCTs, need to be large (and consequently expensive) to have adequate statistical power. Policy makers, payers, and practitioners must decide whether freedom from bias is worth the extra cost. For most work, observational and cost-effectiveness studies that are quasi-experimental or collect pre-post data from the same recipients (or groups of similar recipients) will continue to be popular. Done well, this work can address the efficiency and effectiveness information needs of policy makers and practitioners. Increasing the volume of this work is achievable and could be built into program operations and conducted in collaboration with other states.


Lesson four arises from the organizational changes needed to implement practice innovations – topics frequently ignored in the up-or-down reporting of program outcomes. A key issue is organizational commitment. Organizations participating in a demonstration or any innovation must have a willingness to adapt their communication and service support infrastructure to the service model being implemented. Commitment is evident in a number of ways. Among them:



  • Allocation of strong, determined, and consistent administrative leadership for the program; the recruiting and training of critical staff.
  • Development and continual refinement of such essential tasks as health-risk screening instruments; electronic data systems that link risk identification, home assessment, program benefits, and care planning with primary care and other providers.
  • Continual refinement of communications processes that connect such things as hospital continuity of care, home health care, nursing home care, ambulatory care, and others.
  • Another element of commitment is that all critical infrastructure may not be in place at the outset of most innovations. This is especially true when major reengineering of clinical operations is necessary, such as if the organization is accepting financial risk for its operations and outcomes. Building that infrastructure is a major commitment that requires time and continual support throughout the organization. Limited term demonstrations produce little incentive for substantial infrastructure development. Similarly, once the infrastructure is in place, it is disheartening to abandon this with the closing of major demonstrations.

Lesson five is a corollary to the above. Organizational and practice changes do not happen instantaneously. Programs and approaches have to be initiated, phased in, appraised, and refined. This is complicated enough in a single-site program. When multisite programs are involved, time has to be allowed for phasing between sites and the work needed to attain comparable levels of practice. The net result is that it may take more than a year to establish internal operations – even longer among the multiple sites – and it may be very uneven among the network of providers with whom the program interacts.


This lesson loops back to the need to coordinate the innovation’s maturity with the evaluation design. One difficult situation is maintaining a comparison sample, especially when treatment and control groups are drawn from the same provider panels or organizations. The risk that sizable proportions of recipients of the comparison sample will move between providers and programs, and/or those providers may adopt some of the innovative practices increases with time. Such shifts contaminate the experiment and risk reducing observed difference with usual care. The failure to identify program effectiveness when it exists is very high in this situation.


These risks can be minimized if creation of comparison groups is deferred until the innovative program has reached an acceptable level of steady state operations. An alternative is to have repeated pre-post comparisons limited to those in the program before and after the introduction of the innovative changes. Another option for maintaining an uncontaminated comparison group across an extended period is to draw them in other communities or organizations. This may be possible only in well-financed evaluations.


Closing thoughts


The importance of having the government and private sectors stimulating experimentation and innovation in care for aged and disabled persons is an underlying assumption of this chapter. The innovations discussed in this book offer examples of the value, challenges, and limitations of the approaches taken to date. The emphasis on most program start-ups, understandably, is on infrastructure development. Fewer resources and much less time is given to incremental refinement in practices as programs gain experience. Public policy would be well served by working to refine, document, and disseminate these experiences and accomplishments. Decisions like this are more easily applied in programs (like Medicaid waivers) that are on-going, but possible with some time limited demonstrations.


Centers of excellence and collaborative networks are common in basic sciences and rehabilitation services. These function with an aim of incremental refinement and repeated testing. Is something analogous possible for care coordination practice?


Natural experiments for collaborations already exist. They include the many state HCBS waiver programs, and other Medicaid long-term care initiatives intended to prevent nursing home placement, or reduce hospital stays. Managed care programs (such as PACE programs, Medicaid and Medicare health maintenance organizations), and the Veterans Administration are other examples.


A third resource are organizations (including current and former care coordination demonstration programs) which are operating care coordination programs and are willing to participate in a collaborative center or consortium. The National Chronic Care Consortium (2001) is one example of this. This voluntary group was formed mainly among managed care providers to share experience and to consider standardization in measures and practice. They have given some attention to care coordination.


CMS (perhaps in collaboration with other federal agencies, private insurance, state governments, and foundations) could likely facilitate the formation of a consortium for care coordination collaboration. These might serve as more than centers for research and program operations dissemination. That function is already available through conferences. What is needed is a think-tank for the development of innovation models and practice refinement, instrument development and standardization, planned variations for program innovations testing, and establishing guidelines of practice. The consortium would likely involve voluntary participation and organizational commitment, but it could provide a resource to expand the evidence-based practice in the field of care coordination.


Notes


1. There is a vast library of readings describing approaches to evaluation research methods. Brinkerhoff (2003), Frechtling (2007), Levin & McElwan (2001), Patton (2001), and Shadish, Cook, & Campbell (2001) are among the authors who have informed our thinking about practical approaches.


2. This framework is adapted from Dobell & Newcomer (2008), p. 27. This work is a synthesis from multiple sources: best-practices in chronic care (Reuben 2002; Wagner et al. 2001), disease management (Villagra 2004), integrated care (Stone & Katz 1996), long-term care (Wiener & Stevenson 1998), and Medicare reform (Cassel, Besdine, Siegel 1999; Whitelaw & Warden 1999).


3. Among these are the U.S. Preventive Services Task Force (USPSTF), the American College of Physicians’ Clinical Efficacy Assessment Program (CEAP), UL National Center for Clinical Evidence (NICE), and AHRQ designated Evidence-Based Practice Centers (EPCs; Hefland 2005).


4. This synopsis was derived from the CMS Research Activities: The Active Projects Report. See the CMS web site http://www.hhs.gov/researchers/projects/apr/default/asp for a current listing of projects.


5. The Program of All-Inclusive Care (PACE) is an exception to both the separation of payment source and the exclusion of long-term care liability. This program is targeted to non-institutionalized persons age 55 and over, and operates under capitation payments from both Medicare and Medicaid reimbursement. The PACE provider assumes financial risk for all levels and duration of care across the continuum of care (Eng et al. 1997). EverCare is another exception. It is a managed care program for those in nursing homes (Kane et al. 2004).


6. Diffusing Care Coordination Models: Translating Research into Policy & Practice held on September 16, 2010, in Washington, DC.


References


Brinkerhoff, R.O. (2003) The Success Case Method: Find Out Quickly What’s Working and What’s Not. Berrett-Koehler Publishers, San Francisco, CA.


Cassel, C.K., Besdine, R.W., & Siegel, L.C. (1999) Restructuring Medicare for the next century: what will beneficiaries really need? Health Affairs, 18(1), 118–131.


Centers for Medicare & Medicaid Services. (2005) CMS Research Activities: The Active Projects Report. Available at: http://www.cms.hhs.gov/researchers/projects/apr/default.asp.


Dobell, G., & Newcomer, R. (2008) Integrated care: incentives, approaches, and future considerations. Social Work in Public Health, 23(4), 25–47.


Eng, C., Pedulla, J., Eleazer, P., et al. (1997) Program of All-Inclusive Care for the Elderly: an innovative model of integrated geriatric care and financing. Journal of the American Geriatrics Society, 45, 223–232.


Frechtling, J.A. (2007) Logic Modeling in Program Evaluation. Jossey-Bass Publications, San Francisco, CA.


Helfand, M. (2005) Using evidence reports: progress and challenges in evidence-based decision making. Health Affairs, 24(1), 123–132.


Kane, R.L., Flood, S., Bershadsky, B., et al. (2004) Effect of an innovative Medicare managed care program on the quality of nursing home residents. The Gerontologist, 44(1), 95–103.


Levin, H.M., & McEwan, P.J. (2001) Cost-Effectiveness Analysis, 2nd edn. Sage Publications, Thousand Oaks, CA.


National Chronic Care Consortium. (2001) Provider Survey Report: Summary of 2000 Survey of MSHO Care Coordinators, Nurse Practitioners, and Physicians. National Chronic Care Consortium, Washington, DC. Available at: http://www.cnnonline.org/products/M09001.pdf.


Patton, M.Q. (2001) Qualitative Research and Evaluation Methods, 3rd edn. Sage Publications, Thousand Oaks, CA.


Reuben, D.B. (2002) Organizational interventions to improve health outcomes of older persons. Medical Care, 40, 416–428.


Shadish, W.R., Cook, T.D., & Campbell, D.T. (2001) Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Houghton Mifflin, Boston, MA.


Stone, R.I., & Katz, R. (1996) Thoughts on the future of integrating acute and long-term care. In: Annual Review of Gerontology and Geriatrics, Volume 16 (eds. R. Newcomer & A. Wilkersin), pp. 217–248. Springer, New York, NY.


Villagra, V.G. (2004) Integrating disease management into the outpatient delivery system during and after managed care. Health Affairs, Web Exclusive, May 19, (W4)281–(W4)283.


Wagner, E.H., Austin, B., Davis, C., et al. (2001) Improving chronic illness care: translating evidence into action. Health Affairs, 20(6), 64–78.


Whitelaw, N.A., & Warden, G.L. (1999) Reexamining the delivery system as part of Medicare reform. Health Affairs, 18(1), 132–143.


Wiener, J.M., & Stevenson, D.G. (1998) State policy on long-term care for the elderly. Health Affairs, 17(3), 81–100.


Only gold members can continue reading. Log In or Register to continue

Stay updated, free articles. Join our Telegram channel

Apr 9, 2017 | Posted by in NURSING | Comments Off on Evaluation methods

Full access? Get Clinical Tree

Get Clinical Tree app for offline access