Resources to Facilitate Advanced Practice Nursing Outcome Research

248249CHAPTER 12

Resources to Facilitate Advanced Practice Nursing Outcome Research


Denise Bryant-Lukosius, Ruth Martin-Misener, Joan Tranmer, Faith Donald, Linda Brousseau, and Alba DiCenso


Chapter Objectives

1.  Describe the impact of dedicated national research programs for building advanced practice nursing (APN) research capacity and expertise

2.  Provide an overview of the PEPPA (Participatory, Evidence-Based, Patient-Focused Process for Advanced Practice Nursing Role Development, Implementation, and Evaluation) framework and how it has been enhanced and applied to guide the successful design, implementation, and evaluation of APN roles

3.  Describe additional tools and resources developed to support the selection and evaluation of APN-sensitive outcomes relevant to different stages of role development


Chapter Discussion Questions

1.  What five features of complex health care interventions are consistent with common characteristics of APN roles?

2.  What are common barriers to conducting meaningful APN role evaluations?

3.  How can participatory action research (PAR) be used to inform the development and implementation of APN roles? Why is this process important?

4.  Provide examples of APN role structures, processes, and outcomes. What are the objectives for evaluating structures, processes, and outcomes at three distinct stages of role development?




250Based on UK Medical Research Council guidelines, APN roles meet the criteria of a complex health care intervention (Craig et al., 2008). They involve a number of interacting components, including role competencies and related activities (clinical, educational, research, leadership, consultation, collaboration), which when enacted in combination are greater than the sum of their parts (Canadian Nurses Association [CNA], 2008). These activities are often directed at several target groups (patients and families, nurses and other health care providers, organizations, and health systems) to address difficult health care problems and to achieve a variety of outcomes relevant to each target group. APN roles also need to be highly flexible and responsive to the dynamic needs and contexts of the patient populations they serve and the environments and practice settings in which they work.

Like other complex health care interventions, high-quality and meaningful evaluations of APN roles require a sophisticated research skill set to determine the purpose of the evaluation and to apply the most appropriate methods for examining the role at different stages of development (Bryant-Lukosius, 2009). While numerous reviews document the effective outcomes of APN roles (Morrilla-Herrera et al., 2016; Newhouse et al., 2011; Swan, Ferguson, Change, Larson, & Smaldone, 2015), the quality of the individual APN studies included in the reviews is inconsistent. Advanced practice registered nurses (APRNs) also report that they lack the research knowledge, skills, and experience to evaluate the impact of their roles (DiCenso & Bryant-Lukosius, 2010). A common limitation of APN role evaluations is the failure to use relevant theoretical frameworks and rigorous research methods in designing both the role and the evaluation plan. This limitation contributes to poorly defined roles and evaluation designs that do not improve our understanding about the complexity of the role and the relationships between role activities and outcomes, or how these roles did or did not achieve expected outcomes. Similarly, the lack of clearly defined APN roles and poor agreement among stakeholders about expected outcomes is a barrier to selecting APN-sensitive outcomes and collecting baseline data for future comparative evaluations (Bryant-Lukosius, DiCenso, Browne, & Pinelli, 2004).

This chapter reviews work being done in Canada to further APN outcome research, by providing resources to address these five major barriers (i.e., lack of APN research expertise, guiding frameworks, role clarity, sensitive outcome measures, baseline data) to evaluate these complex health care interventions. These resources include (a) the evolution of a federally funded APN research chair and establishment of a dedicated national research unit designed to build research capacity; (b) mentorship and continuing education to support the development of novice APN researchers at the point of care; (c) guidelines for conducting economic evaluations of APN roles; (d) an enhanced conceptual framework and related toolkit for guiding the APN role design, implementation and evaluation process; and (e) a strategy for identifying sensitive outcome measures, including an APN research data collection toolkit for selecting APN-sensitive outcome measurement tools. An overview of these resources will be provided and their application to promote high-quality outcome assessments of APN roles will be examined.


From 2001 to 2011, the Canadian government’s health research agency funded a chair in APN held by Alba DiCenso, a coauthor of this chapter. The purpose of the chair was 251to increase Canada’s pool of nurse researchers with the ability to lead and conduct applied APN-related research that serves the needs of clinicians, managers, and policy makers in the health sector. Chair activities designed to achieve this goal focused on (a) the education of nurse researchers at the graduate level, (b) linkage and exchange with decision makers to ensure policy relevance and the dissemination and uptake of research results, (c) mentoring junior faculty and postdoctoral fellows to launch an APN-related research program, and (d) conducting research to inform the practice of APRNs across Canada.

Annually, through a competitive process, three Canadian graduate students (master’s and PhD level) planning to conduct APN-related health services research were accepted into the Chair Program and awarded a $10,000 bursary. In addition to the university requirements for degree completion, the Chair Program required participating students to (a) enroll in a graduate course, “Research Issues in the Introduction and Evaluation of APN Roles,” specifically developed for APN chair students, (b) write an APN-related commentary for the journal Evidence-Based Nursing, (c) complete a 90-hour practicum in a policy setting and a 90-hour research internship, (d) identify an interdisciplinary thesis committee to oversee an APN-related health services research study, (e) partner with a decision maker to identify a policy-informing thesis topic, and (f) attend monthly meetings of Chair Program students via remote technology. Chair funding permitted graduate students from across the country to enroll in the APN-related graduate course by covering travel and accommodation costs for face-to-face components.

image  Impact of the Chair Program on Building APN Research Capacity and Expertise

The Chair Program accepted 24 graduate students from five provinces (6 MSc, 16 PhD, 2 DNP) who were enrolled in nine Canadian and American universities. Those who completed their PhDs were eligible to compete for a junior faculty position funded and supervised by the Chair Program with the goal to secure postdoctoral funding. Three junior faculty were funded through the Chair Program, all of whom were successful in competing for externally funded postdoctoral fellowship awards. These plus two additional postdoctoral fellows supervised by the chair, continued to participate in the Chair Program as McMaster University–based senior faculty or as affiliate faculty from other Canadian universities. In addition to the chair students, 87 APRNs, graduate students, and health care administrators from across Canada completed the graduate course “Research Issues in the Introduction and Evaluation of APN Roles.” The final course assignment was the preparation of a research proposal related to the development, implementation, or evaluation of an APN role.

Over the course of 10 years (2001–2011), the APN Chair Program had substantive impact on the capacity to conduct APN research in Canada. The program received over $3.5 million in research funding to complete 48 APN-focused studies. Research trainees and faculty presented their study findings and other scholarly work in 605 oral and 182 poster peer-reviewed presentations at national and international conferences and in 236 peer-reviewed publications and 21 book chapters. Through linkage and exchange activities among graduate students, APRNs, researchers, and health care administrators, the Chair Program fostered the development of other national APN research enterprises, supported national collaboration within research teams conducting APN research, and 252enabled studies of national importance. For example, one clinical nurse specialist (CNS), who is also a coauthor of this chapter (D.B-L.), transitioned through each research-training opportunity provided by the Chair Program and went on to receive funding to establish a Canadian Centre of Excellence in Oncology APN. This center provides research training, mentorship, and consultative services and promotes the uptake of research findings through knowledge translation focused on APRNs in cancer control. More information about the center can be found online (

Chair students and faculty from several provinces also collaborated on a landmark national study and interdisciplinary research team, led by two coauthors of this chapter (F.D. and R.M.M.), to examine the role of nurse practitioners (NPs) in long-term care settings (Carter et al., 2016; Donald et al., 2013; Kaasalainen et al., 2013; Martin-Misener, Donald, et al., 2015; Ploeg et al., 2013; Sangster-Gormley et al., 2013). A second national study led by chair faculty and graduate students established the foundation for the next generation of APN research in Canada (DiCenso & Bryant-Lukosius, 2010). The results of this comprehensive study were summarized in a special issue of the Canadian Journal of Nursing Leadership that included 10 publications outlining evidence-based recommendations for the individual, organizational, and health system supports required to better integrate CNS and NP roles into the Canadian health care system. A major finding of this study was the need for further research about the outcomes and cost-effectiveness of APN roles. The special issue can be accessed freely online (


In July 2011, the 10-year funding for the Chair Program ended but its legacy continues with the establishment of the Canadian Centre for APN Research (CCAPNR) at McMaster University. As evidence of growing research capacity and the importance of succession planning to maintain research productivity and momentum, this center is led by senior and affiliate faculty from three provinces who are graduates of the Chair Program. In contrast to the APN Chair Program, the primary focus of the Center is on conducting research and on knowledge translation, with a continued but less prominent emphasis on the training and mentorship of APN researchers. To further the development of APN research expertise and to strengthen our understanding of these complex roles, CCAPNR has expanded its mandate to support collaborative, interdisciplinary research nationally and internationally. More information on CCAPNR can be found online (

image  Five Years of CCAPNR Activities and Influence

Over the past 5 years, CCAPNR’s research, evaluation, and knowledge transfer activities and products have made a substantial contribution to APN role introduction, implementation, and sustainability. Topics that have been the focus of CCAPNR research are summarized in Exhibit 12.1. CCAPNR has completed or is currently involved in policy, practice, educational, and research initiatives with organizations at the national level and in more than half of Canadian provinces/territories. Internationally, CCAPNR has provided consultation, education, and leadership to international organizations and in at least eight countries. CCAPNR’s knowledge translation tools and research evidence are being used around the world. Some of CCAPNR’s accomplishments include:

253EXHIBIT 12.1  Examples of Topic Areas of CCAPNR Research

image  Cost-effectiveness of CNS and NP roles

image  Optimizing health care system integration of the CNS role

image  APN activity and workload

image  APN roles and team collaboration

image  Navigation roles in primary health care

image  Optimizing RN roles in primary health care

image  NP competencies and education in primary health care

image  Indicators for evaluating the quality of NP services in primary health care

image  NP prescribing practices in primary health care

image  Impact of NPs in long-term care

image  Effectiveness of health coaching in type 2 diabetes

image  Effective use of APN roles in cancer control

image  Interprofessional models of cancer survivorship care in primary health care

APN, advanced practice nursing; CCAPNR, Canadian Centre for APN Research; CNS, clinical nurse specialist; NP, nurse practitioner.

image  Securing over $5.3 million in funded research grants, contracts, and capacity-building initiatives

image  Publishing over 100 articles in high-impact international and national peer-reviewed journals

image  Giving 100 presentations at international, national, and regional peer-reviewed conferences to researcher, policy-maker, and clinician audiences

image  Presenting 70 addresses as invited speakers at a variety of international, national, and regional forums

image  Providing leadership to international, national, and regional organizations seeking to establish, expand, or optimize APN and other specialized nursing roles through education, regulation, and policy

image  Offering innovative approaches to educate and mentor health care professionals in leading and conducting point-of-care research, quality improvement (QI), or evidence-informed decision-making (EIDM) projects

image  Establishing productive, mutually rewarding collaborations with international, national, and regional partners

image  Contributing to graduate education about APN beyond expected teaching roles

In summary, CCAPNR has evolved to become known as a leader in: APN research and evaluation; education and policy; and linkage and exchange, both nationally and internationally. Future directions include expanding our membership, and continuing to build our international reach while maintaining the strength of involvement and 254influence we have in producing and translating evidence related to APN and specialized nursing in Canada. In the following two sections, specific examples of innovative CCAPNR initiatives are described.

image  Building APN Capacity to Conduct Point-of-Care Research

The Trillium and Alberta Rose projects involved academic practice partnerships established between CCAPNR and two health care organizations in different provinces. The aim of these partnerships was to provide an applied research course and mentorship to promote the integration of research into the day-to-day practice of APRNs. The major output from the course was a research proposal or QI/evidence-based project that participants would implement following completion of the course. Participants were paired with a CCAPNR faculty mentor with similar research interests, to support the development of their research proposal or project plan and to assist with the development of research competencies over a short period of time (e.g., 4 months). Course topics were led by CCAPNR faculty and included a variety of learning strategies, such as small group seminars, peer review, and videoconferencing. A formative evaluation completed at the end of the course showed that participants were highly satisfied with the course content and the knowledge and expertise of the faculty. A pre- and postcourse evaluation demonstrated significant improvement in participants’ perceived confidence and competence in conducting research (Harbman et al., 2017).

image  Advancing Economic Evaluations of APN Roles

Controlling health care costs and improving care quality are important drivers that influence decisions to implement APN roles. To inform these decisions, CCAPNR faculty conducted a systematic review of randomized controlled trials (RCTs) evaluating the economic impact of NP and CNS roles (Donald, Kilpatrick, Reid, Carter, Martin-Misener, et al., 2014). The primary outcomes of interest were health care costs (e.g., professional, family, and hospital costs); resource use (e.g., prescriptions, diagnostic tests and procedures); length of stay; and rehospitalization. Health system utilization was also examined for all patient (e.g., mortality, morbidity, satisfaction with care, quality of life) and provider (e.g., job satisfaction and quality of care) outcomes. Published and unpublished RCTs, in all languages, conducted between January 1980 and July 2012 were eligible for the review. Forty-three RCTs were included in the review and categorized into six groups: NP-outpatient (primary care and long-term care; n = 11), NP-transition (n = 5), NP-inpatient (n = 2), CNS-outpatient (n = 11), CNS-transition (n = 13), and CNS-inpatient (n = 1). Transition roles involved services to safely transfer patients from one level of care to another or between settings in a timely manner (Naylor, Aiken, Kurtzman, Olds, & Hirschman, 2011). The review results are published in several papers according to the six groupings (Bryant-Lukosius, Carter, et al., 2015; Donald, Kilpatrick, Reid, Carter, Bryant-Lukosius, et al., 2014; Kilpatrick et al., 2014, 2015; Martin-Misener, Harbman, et al., 2017).

An important aspect of the systematic review was assessment of the quality of economic evaluations using the Quality of Health Economic Studies (QHES) tool (Marshall et al., 2015). The QHES assesses the quality of health economics studies involving 255cost-minimization, cost-effectiveness, or cost–utility analyses (Chiou et al., 2003). Based on the QHES tool, most of the 43 RCTs (77%) were found to have poor study quality and only three studies (7%) were of high quality (Marshall et al., 2015). The multifaceted domains and versatility of NP and CNS practice created challenges in applying general economic analysis guidelines to studies evaluating these roles (Marshall et al., 2015). Consequently, the research team determined that the general guidelines required adaptations to adequately address economic analyses of APN roles. Four processes were used to adapt the economic evaluation guidelines: (a) a literature review of theoretical and discussion papers about economic evaluations of CNS and NP roles; (b) a detailed analysis of quality of the 43 RCTs in this study; (c) assessment of current economic evaluation guidelines; and (d) recommendations from a panel of 15 experts from Canada and the United States (Lopatina et al., 2017).

General criteria were identified from a review of economic evaluation tools (e.g., Canadian Agency for Drugs and Technologies in Health, 2006; Drummond, Schulpher, Claxton, Stoddart, & Torrance, 2015). Four criteria were found to be applicable to APN roles without modification:

1.  Specifying the time horizon or duration of the study that is sufficient to capture important outcomes and costs of the intervention

2.  Applying modeling techniques using the results from a short-term study to evaluate and project the long-term effect

3.  Using discounting rates at 5% per year for costs and outcomes that occur beyond 12 months, especially for studies involving patients with chronic conditions

4.  Managing variability and uncertainty by conducting sensitivity analyses to determine how variations in inputs affect study results and conclusions.

Seven general economic evaluation criteria required adaptation to improve their application to APN studies. A brief overview of these criteria is provided in Table 12.1. A more fulsome summary of the criteria is presented in a manuscript for publication (Lopatina et al., 2017).


Two additional resources arising from the Chair Program are the PEPPA framework and the APN Research Data Collection Toolkit. The PEPPA framework was developed by a coauthor of this chapter (D.B-L.) as a doctoral student in the APN Chair Program (Bryant-Lukosius & DiCenso, 2004). As Chair Program students conducted their research, they used existing data collection tools and developed a variety of instruments. The APN Chair Program was often approached by researchers and decision makers for information about APN-related research data collection tools. As a result, the Chair Program, with funding from a provincial decision-maker partner, created the APN Research Data Collection Toolkit. Researchers conducting APN outcome research may find both these resources, the PEPPA framework and the APN Research Data Collection Toolkit, useful. The remainder of this chapter describes these two resources.

256TABLE 12.1    Criteria Requiring Adaptation for Economic Evaluations of APN Roles



Develop the study question

Include the target population, intervention, comparators, and study perspective. Consider the model of APN care, role domains, and if the role is alternative or complementary to usual care.

Select a suitable method of economic evaluation

Cost-effectiveness is applicable for a single unit of outcome, but may not be applicable for multiple outcomes.

Cost–utility analysis combines several outcomes into a single composite, such as a QALY, but cannot include complex measures of quality of life, nonhealth outcomes, and opportunity costs.

Cost–benefit analysis values all costs and benefits in the same monetary units to determine the value that people attach to health care services. It can be challenging to place costs on participants’ values.

Cost-minimization analysis can be used when data analysis reveals no differences in outcomes between comparison groups. Only costs are considered. Multiple outcomes may preclude this type of analysis.

Cost–consequence analysis is recommended for APN role evaluations with multiple outcomes. It provides separate reporting of all costs and outcomes.

Choose a comparator

For alternative APN roles, determine what constitutes “usual care.” In complementary models, the APN role may be added to a team to augment care. Evaluations of these roles compare outcomes of teams with and without APRNs.

Decide the study perspective

The perspective (public payer, patient, employer, or society) determines which costs, resources, and consequences are included in the analysis. The public payer perspective is commonly used but fails to capture the benefits or costs from other sectors, cost shifts to the patient and family, productivity costs, or long-term consequences, such as health promotion/prevention costs.

Measure effectiveness

Including at least 10 APRNs in an RCT may minimize bias related to variability in roles and the contexts of care (Donald, Kilpatrick, Reid, Carter, Martin-Misener, et al., 2014).

Value APN outcomes

Some APN outcomes (e.g., access to care, patient satisfaction, quality of care, impact of nonclinical role dimensions) can be challenging to measure from an economic perspective.

Determine resource use and costs

Items typically included in studies of APN roles include prescriptions, diagnostic tests, length or number of visits, length of stay, and so on. The source of the data on these items should be stated (e.g., administrative databases, current literature). A broad and inclusive approach is required for chronic conditions. This may include costs related to lost productivity for patients to attend appointments or for family members to care for sick relatives.

APN, advanced practice nursing; APRNs, advanced practice registered nurses; QALY, quality-adjusted life year; RCT, randomized controlled trial.

image  The PEPPA Framework

The PEPPA framework was developed to provide APN researchers, health care providers, administrators, and policy makers with a guide to promote the optimal development and deployment of APN roles. A critical feature of this framework is that strategies to support meaningful outcome evaluations of APN roles are incorporated throughout role planning and implementation. The underlying premise of the PEPPA framework is that the mandate of all APN roles is to maximize, maintain, or restore patient health through innovation in nursing practice and in the delivery of health services (CNA, 2008; Davies & Hughes, 2002). This mandate is consistent with international views of advanced nursing practice. There is a heightened demand worldwide for APRNs, as clinical experts, leaders, and change agents, to assist organizations in developing sustainable models of health care (Bryant-Lukosius, DiCenso, et al., 2004).

257EXHIBIT 12.2  Common Problems Associated With APN Role Implementation

image  Stakeholder confusion about APN terminology

image  Lack of clearly defined roles and role goals or outcomes

image  Role emphasis on physician replacement or support

image  Underutilization of APN scope of practice and expertise in all role dimensions

image  Failure to address role implementation barriers

image  Limited use of evidence-based approaches to guide role development, implementation, and evaluation

APN, advanced practice nursing.

Source: Bryant-Lukosius, DiCenso, et al. (2004).

An initial review of the international literature identified six frequently reported barriers that were common to the effective implementation of various types of APN roles (Bryant-Lukosius, DiCenso, et al., 2004). A subsequent study involving a review of the international literature and examination of APN role implementation in Canada reconfirmed these challenges (DiCenso & Bryant-Lukosius, 2010). Many of these barriers could be avoided through improved planning and better stakeholder understanding of the roles (Exhibit 12.2). Initial and follow-up reviews of the literature (Bryant-Lukosius, DiCenso, et al., 2004; DiCenso et al., 2010) also indicated that the costs associated with poor APN role implementation planning are high and justify the need for more thoughtful, systematic approaches to role introduction (Exhibit 12.3).

The PEPPA framework builds on earlier models recommending steps for introducing new health providers (Spitzer, 1978) and specifically APN roles (Dunn & Nicklin, 1995; Mitchell-DiCenso, Pinelli, & Southwell, 1996), by incorporating additional steps and strategies to address known barriers to successful APN role implementation. The aims of the framework are to:

image  Use relevant data to support the need and identified goals for a clearly defined role

image  Support advanced nursing practice characterized by patient-centered, health-focused, and holistic care

image  Promote the integration of APN knowledge, skills, and expertise from all role dimensions related to clinical practice, education, research, organizational leadership, and scholarly/professional practice (Canadian Association of Nurses in Oncology [CANO], 2001)

image  Create practice environments that support APN role development by engaging stakeholders from the health care team, practice setting, and health care system in role planning

image  Promote ongoing role development and model of care enhancement through continuous and rigorous evaluation of progress in achieving predetermined outcome-based goals

Only gold members can continue reading. Log In or Register to continue

Stay updated, free articles. Join our Telegram channel

Dec 7, 2017 | Posted by in NURSING | Comments Off on Resources to Facilitate Advanced Practice Nursing Outcome Research

Full access? Get Clinical Tree

Get Clinical Tree app for offline access