13.1 IMPORTANCE OF PROCESS EVALUATION

The importance of conducting a process evaluation in studies examining the effectiveness of health interventions is increasingly recognized as essential in interpreting the results of a particular study (MRC, 2019) and making valid inferences regarding the effectiveness of interventions. In general, the results of an outcome‐focused evaluation study indicate that the intervention is, overall, effective or ineffective; they fall short of delineating evidence on what exactly contributed to the observed improvement or no change in the outcomes following delivery of the intervention (Benson et al., 2018).

When the results support the effectiveness of the health intervention, it is important to know: (1) whether the expected improvement in outcomes can be confidently attributed to the intervention and not to other factors such as the interventionist–participant working alliance (an issue of internal validity); (2) what components of the intervention are most helpful (an issue of construct validity) and least helpful in inducing the hypothesized changes in the ultimate outcomes, and how can the intervention as‐a‐whole or its components be refined to optimize effectiveness; and (3) what contextual factors facilitate the delivery of the intervention and enhance clients’ experience of the ultimate outcomes and, therefore, should be accounted for in future implementation of the intervention in research and practice (issue of external validity) (Evans et al., 2015; Furness et al., 2018; Griffin et al., 2014; Masterson‐Algar et al., 2016; van Bruinessen et al., 2016; Wilkerson et al., 2019).

When the results of an outcome evaluation study do not support the effectiveness of the health intervention, it is important to explain the discrepancies between the hypothesized and the actually observed ultimate outcomes (Grant et al., 2013; Cheng & Metcalfe, 2018; Mars et al., 2013; Van de Glind et al., 2017). The main question guiding the investigation of these discrepancies is: Do the results (i.e. no change in the ultimate outcomes) reflect (1) unsuccessful intervention or theory failure, meaning that the intervention itself does not work or is inherently flawed, inadequate or deficient; or (2) unsuccessful delivery of the intervention or implementation failure, meaning that the intervention is not delivered appropriately, leading to type III error (Biron et al., 2010, 2016; Evans et al., 2015; van Bruinessen et al., 2016).

It may is wise to explore implementation failure in the context of an outcome evaluation study. If ruled out, then intervention theory failure is investigated.

The specific questions informing exploration of implementation failure include:

1. How well is the intervention delivered by interventionists? How effective is the training in preparing interventionists for providing the intervention? Is the intervention delivered with fidelity and good quality? Is the intervention or any of its components adapted? What is the type of adaptation done? Why, how, for whom, or in what conditions is the adaptation done? (Evans et al., 2015; Nurjono et al., 2018; Sharma et al., 2017; Siddiqui et al., 2018; Tama et al., 2018; van Bruinessen et al., 2016).
   
2. What contextual factors influence the delivery of the intervention by interventionists? Are all resources available and accessible? What are other barriers (e.g. sociopolitical, economic, historical, cultural) to the appropriate delivery of the intervention and its components? (Foley et al., 2016; Morgan‐Trimmer, 2015; Siddiqui et al., 2018; Van de Glind et al., 2017).
   
3. How do participating clients respond to the intervention? How do they view the intervention and its components? Do participants view the intervention as acceptable, helpful, and are they satisfied with it? Are participants exposed to the full intervention (i.e. all its components and sessions) and if not, why? Is the content covered well understood, received, and useful in enhancing participants’ understanding and management of the health problem? Do participants engage in the intervention activities? What contextual factors affect participants’ engagement and enactment of the intervention? (Al‐HadiHasan et al., 2017; Bakker et al., 2015; Hasson et al., 2012; Morgan‐Trimmer, 2015; Siddiqui et al., 2018).

Specific questions informing investigation of intervention theory failure relate to: What is the capacity of the intervention and its components in initiating the mechanism of action mediating its effects on the ultimate outcomes? Were there potential interactions among the intervention components or between the intervention and some contextual factors that weaken the potency of the intervention in producing the hypothesized changes in the ultimate outcomes? Is it possible that the intervention and its components generated an un‐hypothesized mechanism of action that lead to unintended outcomes.

As implied in the previous questions, the importance of a process evaluation rests on its focus on the intervention itself. Process evaluation provides information on the nature of the intervention (i.e. its constituent components), the extent and quality of its delivery (i.e. fidelity and influence of contextual factors) and participants’ response to the intervention (van Bruinessen et al., 2016). Process data are included in the outcome data analysis to determine what exactly contributed to (lack of) improvement in the ultimate outcomes, in what subgroups of participants, under what context (Pawson & Manzano‐Santaella, 2012; Rycroft‐Malone et al., 2018; Sorensen & Llamas, 2018). Such knowledge is critical for making valid inferences about the effectiveness of the intervention, and for refining it to optimize its effectiveness in different client subgroups and contexts.

13.2 DEFINITION AND ELEMENTS OF PROCESS EVALUATION

Although recognized as important, there is no clear conceptualization of process evaluation. Unclear conceptualization resulted in its inconsistent operationalization, and variability in the specification of its operational elements, within and across fields of study. Five approaches or frameworks have frequently been mentioned in recent literature as informing the conduct of process evaluation; these are summarized in Table 13.1. Different operational elements of fidelity are proposed in the five frameworks. Fidelity of intervention delivery is commonly identified in three frameworks. The Consolidated Framework for Implementation Fidelity developed by Carroll et al. (2007) has guided the examination of the fidelity of intervention delivery in recent studies (Hasson et al., 2012). The Steckler and Linnan’s (2002) framework is mentioned as guiding the design and conduct of several process evaluation studies (Table 13.1). However, it has been criticized for incorporating elements indicating the research process (e.g. recruitment, reach) and elements reflecting the intervention process (e.g. fidelity, dose received), leading to confusion in the conceptualization and the operationalization of process evaluation. Siddiqui et al. (2018) and van Bruinessen et al. (2016) advocate the need to distinguish between the research process and the intervention process and to focus process evaluation on the delivery of the intervention. The focus on the intervention process is also supported by the work of other researchers, who operationalized process evaluation into three main elements: delivery of the intervention, context of delivery, and mechanism of action.

1. Delivery of the intervention

Health interventions are provided by interventionists in research and health professionals in practice, or through technology. This element of process evaluation involves:

• Monitoring the fidelity with which the intervention components are provided: As explained in Chapter 9, fidelity is operationalized as the extent to which the interventionists adhered to the intervention manual.
  
• Exploring adaptations of the intervention: The exploration seeks to identify what components (content and activities) are modified, why, how, for whom, and in what conditions; whether the mode and dose of delivery are altered; and whether the adaptations are consistent with the principles explained in the intervention theory for guiding tailoring, and with the allowable changes described in the manual.
  
• Assessing the quality of intervention delivery: As mentioned in Chapter 9, quality is defined as the competence of the interventionists or health professionals in providing the intervention and in interacting with clients (Dillon et al., 2018; Evans et al., 2015; Hasson et al., 2012; Haynes et al., 2014; Hogue & Dauber, 2013; Mars et al., 2013; Moore et al., 2015; Nielson & Abildgaard, 2013; Nurjono et al., 2018; Ruikes et al., 2012; Sharma et al., 2017; Siddiqui et al., 2018; Tama et al., 2018; van den Branden et al., 2015; Van de Glind et al., 2017). For technology‐based interventions, fidelity refers to the extent to which the technology functions as planned (Verwey et al., 2016).

2. Context of delivery

Context includes anything external to the intervention that may act as a barrier or facilitator to its implementation by the interventionists and participants, or through technology (Evans et al., 2015; Fridrich et al., 2015; Grant et al., 2013; Haynes et al., 2014; May et al., 2016; Moore et al., 2015; Nielson & Abildgaard, 2013; Nurjono et al., 2018; Ruikes et al., 2012; Tama et al., 2018). Context or contextual factors thought to influence the implementation of the intervention range from the country and community, which have particular economic, historical, social, cultural, and political characteristics; through site, setting or practice in which the intervention is provided, which possesses specific features related to resources, sociocultural norms underlying interactions among health professionals; to the environment in which the participants apply the treatment recommendations, which has unique physical, psycho‐social and sociocultural attributes (Benson et al., 2018; Morgan‐Trimmer, 2015; Sharma et al., 2017; Van de Glind et al., 2017).

In addition to these contextual factors, some researchers categorize characteristics of the interventionists and the participating clients as contextual factors that play out as barriers or facilitators of intervention implementation (Benson et al., 2018; May et al., 2016; Verwey et al., 2016). As discussed in Chapter 5, the interventionist and client characteristics are stipulated to influence the implementation of the intervention in the intervention theory.

3. Mechanism

This element of process evaluation is conceptualized and operationalized in two ways. The first is the intervention’s mechanism of action, also referred to as process (Grant et al., 2013; Nurjono et al., 2018). As explained in Chapter 5, the mechanism of action is responsible for the effects of the health intervention on the ultimate outcomes. It is represented by the series of linkages among changes in the immediate, intermediate (or mediators), and ultimate outcomes, as envisioned by Evans et al. (2015), Moore et al. (2015), Sharma et al. (2017), and van de Glind et al. (2017). It is argued that this conceptualization of mechanism, although important, is within the realm of outcome evaluation (Chapter 14).

The second conceptualization of mechanism reflects how clients who are exposed to the intervention react, perceive, and respond to it (Brand et al., 2019; Dalkin et al., 2015; Dillon et al., 2018; Rycroft‐Malone et al., 2018; Van Belle et al., 2016). This conceptualization of mechanism is consistent with the notions of: (1) client responsiveness advanced by Carroll et al. (2007); (2) dose received‐exposure and enactment, and satisfaction identified in Steckler and Linnann’s framework (see Table 13.1 and Chapter 9); (3) perception or appraisal of the intervention as alluded to in the work of Poston et al. (2013), Nielson et al. (2007), and van den Branden et al. (2015) and discussed in Chapter 11. Of particular interest in process evaluation is participants’ responsiveness and satisfaction with the intervention (Carroll et al., 2007; Hasson et al., 2012; Haynes et al., 2014). These two concepts represent participants’ views of the intervention’s processes (Chapter 11). Accordingly, the second conceptualization of mechanism is relevant to process evaluation.

The previously presented points support the integration of the following elements in a process evaluation. The elements are in alignment with the focus of process evaluation on the intervention itself. The elements are:

• Fidelity and adaptations of the intervention delivery by interventionists.
  
• Competence of interventionists in intervention delivery.
  
• Contextual factors, operating at the setting level, influencing delivery of the intervention.
  
• Client responsiveness, that is, exposure, engagement and enactment of the intervention.
  
• Contextual factors, operating within participants’ environment, and affecting exposure, engagement and enactment of the intervention.
  
• Perception (or satisfaction) of the intervention by participants.

13.3 METHODS USED IN PROCESS EVALUATION

Process evaluation is conducted in different studies, including preliminary small‐scale studies aimed to examine feasibility of a health intervention, studies focusing on determining the efficacy of a health intervention under controlled conditions, and studies evaluating the effectiveness of an intervention implemented in practice (Benson et al., 2018; Siddiqui et al., 2018). In all cases, the primary concerns are ascertaining the fidelity and the quality of intervention delivery, and identifying factors that affect the delivery. When process evaluation is integrated in an outcome evaluation study, the additional concern is to “unpack the black box” (Wong et al., 2012) to gain a comprehensive understanding of what contribute to the beneficial outcomes, how and under what conditions (Fletcher et al., 2016; Furness et al,. 2018; Nielson & Abildgaard, 2013).

Process evaluation is guided by the intervention theory, which is operationalized in a logic model (Sorensen & Llamas, 2018; Sharma et al., 2017; Van de Glind et al., 2017). As discussed in Chapter 5, the theory clearly defines the intervention’s active ingredients and the respective components, and highlights the components responsible for the interventions effects. Knowledge of the components, including content, activities and treatment recommendations informs the assessment of: the fidelity with which the interventionists deliver the intervention and participating clients’ responsiveness (Masterson‐Algar et al. 2016, 2018; Wells et al., 2012); and participants’ perceptions (satisfaction) of the intervention components. The intervention theory points to key contextual factors, including characteristics of clients, interventionists and setting or environment, that affect the implementation of the intervention. Understanding contextual factors informs their assessment. The intervention theory also specifies the immediate and intermediate outcomes and delineates their associations with specific intervention components, thereby providing guidance for analyzing the contribution of the intervention components to the achievement of outcomes, within and across contexts or settings (Biron et al., 2010, 2016; Sridharan & Nakaima, 2012).

The intervention theory presents conceptual and operational definitions of the intervention components, contextual factors, client responsiveness, and ultimate outcomes. These definitions form the basis for selecting or developing measures of the respective elements of process evaluation. The resulting quantitative data are analyzed descriptively and the proposed relationships are tested using advanced statistical tests. However, to gain a comprehensive and in‐depth understanding of the intervention implementation within and across contexts, qualitative methods are used (Cheng & Metcalfe, 2018). Qualitative methods are useful to explore different stakeholder groups’ (e.g. interventionist, client) views of the intervention, of influential contextual factors, and of the intervention impact, thereby complementing and supplementing the quantitative results. Accordingly, quantitative and qualitative methods are highly valued, and the integration of their respective data is important in enhancing the validity of process evaluation results (De Vlaming et al., 2010; Van de Glind et al., 2017).

Selected quantitative and qualitative methods may have to be adapted to conduct process evaluation of different health interventions. The adaptation is often necessary to accommodate variability in the nature of the intervention components and mode of delivery; the contextual factors operating at the individual client, interventionist, and setting levels; and the perspective of stakeholder groups (including research ethics board), on the methods to be used. For instance, direct observation of intervention delivery may not be relevant for technology‐based health interventions or it may not be acceptable or agreeable to clients (e.g. observation of the provision of intimate care for persons with dementia or of individual sessions addressing personal sexual issues). Furthermore, there are a few established measures of the elements of process evaluation. Those available have to be adapted for consistency with the conceptual and operational definitions of relevant concepts proposed in the intervention theory. Because of this variability, general principles and methods for assessing each element of process evaluation are presented next.