Critical Appraisal of Quantitative and Qualitative Research for Nursing Practice

The nursing profession continually strives for evidence-based practice (EBP), which includes critically appraising studies, synthesizing the findings, applying the scientific evidence in practice, and determining the practice outcomes (Brown, 2014; Doran, 2011; Melnyk & Fineout-Overholt, 2011). Critically appraising studies is an essential step toward basing your practice on current research findings. The term critical appraisal or critique is an examination of the quality of a study to determine the credibility and meaning of the findings for nursing. Critique is often associated with criticize, a word that is frequently viewed as negative. In the arts and sciences, however, critique is associated with critical thinking and evaluation—tasks requiring carefully developed intellectual skills. This type of critique is referred to as an intellectual critical appraisal. An intellectual critical appraisal is directed at the element that is created, such as a study, rather than at the creator, and involves the evaluation of the quality of that element. For example, it is possible to conduct an intellectual critical appraisal of a work of art, an essay, and a study.

The idea of the intellectual critical appraisal of research was introduced earlier in this text and has been woven throughout the chapters. As each step of the research process was introduced, guidelines were provided to direct the critical appraisal of that aspect of a research report. This chapter summarizes and builds on previous critical appraisal content and provides direction for conducting critical appraisals of quantitative and qualitative studies. The background provided by this chapter serves as a foundation for the critical appraisal of research syntheses (systematic reviews, meta-analyses, meta-syntheses, and mixed-methods systematic reviews) presented in Chapter 13.

This chapter discusses the implementation of critical appraisals in nursing by students, practicing nurses, nurse educators, and researchers. The key principles for implementing intellectual critical appraisals of quantitative and qualitative studies are described to provide an overview of the critical appraisal process. The steps for critical appraisal of quantitative studies, focused on rigor, design validity, quality, and meaning of findings, are detailed, and an example of a critical appraisal of a published quantitative study is provided. The chapter concludes with the critical appraisal process for qualitative studies and an example of a critical appraisal of a qualitative study.

When are Critical Appraisals of Studies Implemented in Nursing?

In general, studies are critically appraised to broaden understanding, summarize knowledge for practice, and provide a knowledge base for future research. Studies are critically appraised for class projects and to determine the research evidence ready for use in practice. In addition, critical appraisals are often conducted after verbal presentations of studies, after a published research report, for selection of abstracts when studies are presented at conferences, for article selection for publication, and for evaluation of research proposals for implementation or funding. Therefore nursing students, practicing nurses, nurse educators, and nurse researchers are all involved in the critical appraisal of studies.

Students’ Critical Appraisal of Studies

One aspect of learning the research process is being able to read and comprehend published research reports. However, conducting a critical appraisal of a study is not a basic skill, and the content presented in previous chapters is essential for implementing this process. Students usually acquire basic knowledge of the research process and critical appraisal process in their baccalaureate program. More advanced analysis skills are often taught at the master’s and doctoral levels. Performing a critical appraisal of a study involves the following three steps, which are detailed in this chapter: (1) identifying the steps or elements of the study; (2) determining the study strengths and limitations; and (3) evaluating the credibility and meaning of the study findings. By critically appraising studies, you will expand your analysis skills, strengthen your knowledge base, and increase your use of research evidence in practice. Striving for EBP is one of the competencies identified for associate degree and baccalaureate degree (prelicensure) students by the Quality and Safety Education for Nurses (QSEN, 2013) project, and EBP requires critical appraisal and synthesis of study findings for practice (Sherwood & Barnsteiner, 2012). Therefore critical appraisal of studies is an important part of your education and your practice as a nurse.

Critical Appraisal of Studies by Practicing Nurses, Nurse Educators, and Researchers

Practicing nurses need to appraise studies critically so that their practice is based on current research evidence and not on tradition or trial and error (Brown, 2014; Craig & Smyth, 2012). Nursing actions need to be updated in response to current evidence that is generated through research and theory development. It is important for practicing nurses to design methods for remaining current in their practice areas. Reading research journals and posting or e-mailing current studies at work can increase nurses’ awareness of study findings but are not sufficient for critical appraisal to occur. Nurses need to question the quality of the studies, credibility of the findings, and meaning of the findings for practice. For example, nurses might form a research journal club in which studies are presented and critically appraised by members of the group (Gloeckner & Robinson, 2010).

Skills in critical appraisal of research enable practicing nurses to synthesize the most credible, significant, and appropriate evidence for use in their practice. EBP is essential in agencies that are seeking or maintaining Magnet status. The Magnet Recognition Program was developed by the American Nurses Credentialing Center (ANCC, 2013) to “recognize healthcare organizations for quality patient care, nursing excellence, and innovations in professional nursing,” which requires implementing the most current research evidence in practice (see http://www.nursecredentialing.org/Magnet/ProgramOverview.aspx).

Your faculty members critically appraise research to expand their clinical knowledge base and to develop and refine the nursing educational process. The careful analysis of current nursing studies provides a basis for updating curriculum content for use in clinical and classroom settings. Faculty serve as role models for their students by examining new studies, evaluating the information obtained from research, and indicating which research evidence to use in practice. For example, nursing instructors might critically appraise and present the most current evidence about caring for people with hypertension in class and role-model the management of patients with hypertension in practice.

Nurse researchers critically appraise previous research to plan and implement their next study. Many researchers have a program of research in a selected area, and they update their knowledge base by critically appraising new studies in this area. For example, selected nurse researchers have a program of research to identify effective interventions for assisting patients in managing their hypertension and reducing their cardiovascular risk factors.

Research proposals are reviewed for funding from selected government agencies and corporations. Private corporations develop their own format for reviewing and funding research projects (Grove et al., 2013). The peer review process in federal funding agencies involves an extremely complex critical appraisal. Nurses are involved in this level of research review through national funding agencies, such as the National Institute of Nursing Research (NINR, 2013) and the Agency for Healthcare Research and Quality (AHRQ, 2013).

What are the Key Principles for Conducting Intellectual Critical Appraisals of Quantitative and Qualitative Studies?

An intellectual critical appraisal of a study involves a careful and complete examination of a study to judge its strengths, weaknesses, credibility, meaning, and significance for practice. A high-quality study focuses on a significant problem, demonstrates sound methodology, produces credible findings, indicates implications for practice, and provides a basis for additional studies (Grove et al., 2013; Hoare & Hoe, 2013; Hoe & Hoare, 2012). Ultimately, the findings from several quality studies can be synthesized to provide empirical evidence for use in practice (O’Mathuna, Fineout-Overholt, & Johnston, 2011).

The major focus of this chapter is conducting critical appraisals of quantitative and qualitative studies. These critical appraisals involve implementing some key principles or guidelines, outlined in Box 12-1. These guidelines stress the importance of examining the expertise of the authors, reviewing the entire study, addressing the study’s strengths and weaknesses, and evaluating the credibility of the study findings (Fawcett & Garity, 2009; Hoare & Hoe, 2013; Hoe & Hoare, 2012; Munhall, 2012). All studies have weaknesses or flaws; if every flawed study were discarded, no scientific evidence would be available for use in practice. In fact, science itself is flawed. Science does not completely or perfectly describe, explain, predict, or control reality. However, improved understanding and increased ability to predict and control phenomena depend on recognizing the flaws in studies and science. Additional studies can then be planned to minimize the weaknesses of earlier studies. You also need to recognize a study’s strengths to determine the quality of a study and credibility of its findings. When identifying a study’s strengths and weaknesses, you need to provide examples and rationale for your judgments that are documented with current literature.

Box 12-1 Key Principles for Critically Appraising Quantitative and Qualitative Studies

1. Read and critically appraise the entire study. A research critical appraisal involves examining the quality of all aspects of the research report.

2. Examine the organization and presentation of the research report. A well-prepared report is complete, concise, clearly presented, and logically organized. It does not include excessive jargon that is difficult for you to read. The references need to be current, complete, and presented in a consistent format.

3. Examine the significance of the problem studied for nursing practice. The focus of nursing studies needs to be on significant practice problems if a sound knowledge base is to be developed for evidence-based nursing practice.

4. Indicate the type of study conducted and identify the steps or elements of the study. This might be done as an initial critical appraisal of a study; it indicates your knowledge of the different types of quantitative and qualitative studies and the steps or elements included in these studies.

5. Identify the strengths and weaknesses of a study. All studies have strengths and weaknesses, so attention must be given to all aspects of the study.

6. Be objective and realistic in identifying the study’s strengths and weaknesses. Be balanced in your critical appraisal of a study. Try not to be overly critical in identifying a study’s weaknesses or overly flattering in identifying the strengths.

7. Provide specific examples of the strengths and weaknesses of a study. Examples provide evidence for your critical appraisal of the strengths and weaknesses of a study.

8. Provide a rationale for your critical appraisal comments. Include justifications for your critical appraisal, and document your ideas with sources from the current literature. This strengthens the quality of your critical appraisal and documents the use of critical thinking skills.

9. Evaluate the quality of the study. Describe the credibility of the findings, consistency of the findings with those from other studies, and quality of the study conclusions.

10. Discuss the usefulness of the findings for practice. The findings from the study need to be linked to the findings of previous studies and examined for use in clinical practice.

Critical appraisal of quantitative and qualitative studies involves a final evaluation to determine the credibility of the study findings and any implications for practice and further research (see Box 12-1). Adding together the strong points from multiple studies slowly builds a solid base of evidence for practice. These guidelines provide a basis for the critical appraisal process for quantitative research discussed in the next section and the critical appraisal process for qualitative research (see later).

Understanding the Quantitative Research Critical Appraisal Process

The quantitative research critical appraisal process includes three steps: (1) identifying the steps of the research process in studies; (2) determining study strengths and weaknesses; and (3) evaluating the credibility and meaning of study findings. These steps occur in sequence, vary in depth, and presume accomplishment of the preceding steps. However, an individual with critical appraisal experience frequently performs two or three steps of this process simultaneously.

This section includes the three steps of the quantitative research critical appraisal process and provides relevant questions for each step. These questions have been selected as a means for stimulating the logical reasoning and analysis necessary for conducting a critical appraisal of a study. Those experienced in the critical appraisal process often formulate additional questions as part of their reasoning processes. We will identify the steps of the research process separately because those new to critical appraisal start with this step. The questions for determining the study strengths and weaknesses are covered together because this process occurs simultaneously in the mind of the person conducting the critical appraisal. Evaluation is covered separately because of the increased expertise needed to perform this step.

Step 1: Identifying the Steps of the Research Process in Studies

Initial attempts to comprehend research articles are often frustrating because the terminology and stylized manner of the report are unfamiliar. Identifying the steps of the research process in a quantitative study is the first step in critical appraisal. It involves understanding the terms and concepts in the report, as well as identifying study elements and grasping the nature, significance, and meaning of these elements. The following guidelines will direct you in identifying a study’s elements or steps.

Guidelines for Identifying the Steps of the Research Process in Studies

The first step involves reviewing the abstract and reading the study from beginning to end. As you read, think about the following questions regarding the presentation of the study:

• Was the study title clear?

• Was the abstract clearly presented?

• Was the writing style of the report clear and concise?

• Were relevant terms defined? You might underline the terms you do not understand and determine their meaning from the glossary at the end of this text.

• Were the following parts of the research report plainly identified (APA, 2010)?

• Introduction section, with the problem, purpose, literature review, framework, study variables, and objectives, questions, or hypotheses

• Methods section, with the design, sample, intervention (if applicable), measurement methods, and data collection or procedures

• Results section, with the specific results presented in tables, figures, and narrative

• Discussion section, with the findings, conclusions, limitations, generalizations, implications for practice, and suggestions for future research (Fawcett & Garity, 2009; Grove et al., 2013)

We recommend reading the research article a second time and highlighting or underlining the steps of the quantitative research process that were identified previously. An overview of these steps is presented in Chapter 2. After reading and comprehending the content of the study, you are ready to write your initial critical appraisal of the study. To write a critical appraisal identifying the study steps, you need to identify each step of the research process concisely and respond briefly to the following guidelines and questions.

1. Introduction

a. Describe the qualifications of the authors to conduct the study (e.g., research expertise conducting previous studies, clinical experience indicated by job, national certification, and years in practice, and educational preparation that includes conducting research [PhD]).

b. Discuss the clarity of the article title. Is the title clearly focused and does it include key study variables and population? Does the title indicate the type of study conducted—descriptive, correlational, quasi-experimental, or experimental—and the variables (Fawcett & Garity, 2009; Hoe & Hoare, 2012; Shadish, Cook, & Campbell, 2002)?

c. Discuss the quality of the abstract (includes purpose, highlights design, sample, and intervention [if applicable], and presents key results; APA, 2010).

2. State the problem.

a. Significance of the problem

b. Background of the problem

c. Problem statement

3. State the purpose.

4. Examine the literature review.

a. Are relevant previous studies and theories described?

b. Are the references current (number and percentage of sources in the last 5 and 10 years)?

c. Are the studies described, critically appraised, and synthesized (Brown, 2014; Fawcett & Garity, 2009)? Are the studies from referred journals?

d. Is a summary provided of the current knowledge (what is known and not known) about the research problem?

5. Examine the study framework or theoretical perspective.

a. Is the framework explicitly expressed, or must you extract the framework from statements in the introduction or literature review of the study?

b. Is the framework based on tentative, substantive, or scientific theory? Provide a rationale for your answer.

c. Does the framework identify, define, and describe the relationships among the concepts of interest? Provide examples of this.

d. Is a map of the framework provided for clarity? If a map is not presented, develop a map that represents the study’s framework and describe the map.

e. Link the study variables to the relevant concepts in the map.

f. How is the framework related to nursing’s body of knowledge (Alligood, 2010; Fawcett & Garity, 2009; Smith & Liehr, 2008)?

6. List any research objectives, questions, or hypotheses.

7. Identify and define (conceptually and operationally) the study variables or concepts that were identified in the objectives, questions, or hypotheses. If objectives, questions, or hypotheses are not stated, identify and define the variables in the study purpose and results section of the study. If conceptual definitions are not found, identify possible definitions for each major study variable. Indicate which of the following types of variables were included in the study. A study usually includes independent and dependent variables or research variables, but not all three types of variables.

a. Independent variables: Identify and define conceptually and operationally.

b. Dependent variables: Identify and define conceptually and operationally.

c. Research variables or concepts: Identify and define conceptually and operationally.

8. Identify attribute or demographic variables and other relevant terms.

9. Identify the research design.

a. Identify the specific design of the study (see Chapter 8).

b. Does the study include a treatment or intervention? If so, is the treatment clearly described with a protocol and consistently implemented?

c. If the study has more than one group, how were subjects assigned to groups?

d. Are extraneous variables identified and controlled? Extraneous variables are usually discussed as a part of quasi-experimental and experimental studies.

e. Were pilot study findings used to design this study? If yes, briefly discuss the pilot and the changes made in this study based on the pilot (Grove et al., 2013; Shadish et al., 2002).

10. Describe the sample and setting.

a. Identify inclusion and exclusion sample or eligibility criteria.

b. Identify the specific type of probability or nonprobability sampling method that was used to obtain the sample. Did the researchers identify the sampling frame for the study?

c. Identify the sample size. Discuss the refusal number and percentage, and include the rationale for refusal if presented in the article. Discuss the power analysis if this process was used to determine sample size (Aberson, 2010).

d. Identify the sample attrition (number and percentage) for the study.

e. Identify the characteristics of the sample.

f. Discuss the institutional review board (IRB) approval. Describe the informed consent process used in the study.

g. Identify the study setting and indicate whether it is appropriate for the study purpose.

11. Identify and describe each measurement strategy used in the study. The following table includes the critical information about two measurement methods, the Beck Likert scale and a physiological instrument to measure blood pressure. Completing this table will allow you to cover essential measurement content for a study (Waltz, Strickland, & Lenz, 2010).

a. Identify each study variable that was measured.

b. Identify the name and author of each measurement strategy.

c. Identify the type of each measurement strategy (e.g., Likert scale, visual analog scale, physiological measure, or existing database).

d. Identify the level of measurement (nominal, ordinal, interval, or ratio) achieved by each measurement method used in the study (Grove, 2007).

e. Describe the reliability of each scale for previous studies and this study. Identify the precision of each physiological measure (Bialocerkowski, Klupp, & Bragge, 2010; DeVon et al., 2007).

f. Identify the validity of each scale and the accuracy of physiological measures (DeVon et al., 2007; Ryan-Wenger, 2010).

Variable Measured	Name of Measurement Method (Author)	Type of Measurement Method	Level of Measurement	Reliability or Precision	Validity or Accuracy
Depression	Beck Depression Inventory (Beck)	Likert scale	Interval	Cronbach alphas of 0.82-0.92 from previous studies and 0.84 for this study; reading level at 6th grade.	Construct validity—content validity from concept analysis, literature review, and reviews of experts; convergent validity of 0.04 with Zung Depression Scale; predictive validity of patients’ future depression episodes; successive use validity with the conduct of previous studies and this study.
Blood pressure	Omron blood pressure (BP) equipment (equipment manufacturer)	Physiological measurement method	Ratio	Test-retest values of BPs in previous studies; BP equipment new and recalibrated every 50 BP readings in this study; average three BP readings to determine BP.	Documented accuracy of systolic and diastolic BPs to 1 mm Hg by company developing Omron BP cuff; designated protocol for taking BP average three BP readings to determine BP.

12. Describe the procedures for data collection.

13. Describe the statistical analyses used.

a. List the statistical procedures used to describe the sample (Grove, 2007).

b. Was the level of significance or alpha identified? If so, indicate what it was (0.05, 0.01, or 0.001).

c. Complete the following table with the analysis techniques conducted in the study: (1) identify the focus (description, relationships, or differences) for each analysis technique; (2) list the statistical analysis technique performed; (3) list the statistic; (4) provide the specific results; and (5) identify the probability (p) of the statistical significance achieved by the result (Grove, 2007; Grove et al., 2013; Hoare & Hoe, 2013; Plichta & Kelvin, 2013).

Purpose of Analysis	Analysis Technique	Statistic	Results	Probability (p)
Description of subjects’ pulse rate	Mean Standard deviation Range	M SD range	71.52 5.62 58-97
Difference between adult males and females on blood pressure	t-Test	t	3.75	p = 0.001
Differences of diet group, exercise group, and comparison group for pounds lost in adolescents	Analysis of variance	F	4.27	p = 0.04
Relationship of depression and anxiety in older adults	Pearson correlation	r	0.46	p = 0.03

14. Describe the researcher’s interpretation of findings.

a. Are the findings related back to the study framework? If so, do the findings support the study framework?

b. Which findings are consistent with those expected?

c. Which findings were not expected?

d. Are the findings consistent with previous research findings? (Fawcett & Garity, 2009; Grove et al., 2013; Hoare & Hoe, 2013)

15. What study limitations did the researcher identify?

16. What conclusions did the researchers identify based on their interpretation of the study findings?

17. How did the researcher generalize the findings?

18. What were the implications of the findings for nursing practice?

19. What suggestions for further study were identified?

20. Is the description of the study sufficiently clear for replication?

Step 2: Determining the Strengths and Weaknesses in Studies

The second step in critically appraising studies requires determining strengths and weaknesses in the studies. To do this, you must have knowledge of what each step of the research process should be like from expert sources such as this text and other research sources (Aberson, 2010; Bialocerkowski et al., 2010; Brown, 2014; Creswell, 2014; DeVon et al., 2007; Doran, 2011; Fawcett & Garity, 2009; Grove, 2007; Grove et al., 2013; Hoare & Hoe, 2013; Hoe & Hoare, 2012; Morrison, Hoppe, Gillmore, Kluver, Higa, & Wells, 2009; O’Mathuna et al., 2011; Ryan-Wenger, 2010; Santacroce, Maccarelli, & Grey, 2004; Shadish et al., 2002; Waltz et al., 2010). The ideal ways to conduct the steps of the research process are then compared with the actual study steps. During this comparison, you examine the extent to which the researcher followed the rules for an ideal study, and the study elements are examined for strengths and weaknesses.

You also need to examine the logical links or flow of the steps in the study being appraised. For example, the problem needs to provide background and direction for the statement of the purpose. The variables identified in the study purpose need to be consistent with the variables identified in the research objectives, questions, or hypotheses. The variables identified in the research objectives, questions, or hypotheses need to be conceptually defined in light of the study framework. The conceptual definitions should provide the basis for the development of operational definitions. The study design and analyses need to be appropriate for the investigation of the study purpose, as well as for the specific objectives, questions, or hypotheses. Examining the quality and logical links among the study steps will enable you to determine which steps are strengths and which steps are weaknesses.

Guidelines for Determining the Strengths and Weaknesses in Studies

The following questions were developed to help you examine the different steps of a study and determine its strengths and weaknesses. The intent is not for you to answer each of these questions but to read the questions and then make judgments about the steps in the study. You need to provide a rationale for your decisions and document from relevant research sources, such as those listed previously in this section and in the references at the end of this chapter. For example, you might decide that the study purpose is a strength because it addresses the study problem, clarifies the focus of the study, and is feasible to investigate (Brown, 2014; Fawcett & Garity, 2009; Hoe & Hoare, 2012).

1. Research problem and purpose

a. Is the problem significant to nursing and clinical practice (Brown, 2014)?

b. Does the purpose narrow and clarify the focus of the study (Creswell, 2014; Fawcett & Garity, 2009)?

c. Was this study feasible to conduct in terms of money commitment, the researchers’ expertise; availability of subjects, facilities, and equipment; and ethical considerations?

2. Review of literature

a. Is the literature review organized to demonstrate the progressive development of evidence from previous research (Brown, 2014; Creswell, 2014; Hoe & Hoare, 2012)?

b. Is a clear and concise summary presented of the current empirical and theoretical knowledge in the area of the study (O’Mathuna et al., 2011)?

c. Does the literature review summary identify what is known and not known about the research problem and provide direction for the formation of the research purpose?

3. Study framework

a. Is the framework presented with clarity? If a model or conceptual map of the framework is present, is it adequate to explain the phenomenon of concern (Grove et al., 2013)?

b. Is the framework related to the body of knowledge in nursing and clinical practice?

c. If a proposition from a theory is to be tested, is the proposition clearly identified and linked to the study hypotheses (Alligood, 2010; Fawcett & Garity, 2009; Smith & Liehr, 2008)?

4. Research objectives, questions, or hypotheses

a. Are the objectives, questions, or hypotheses expressed clearly?

b. Are the objectives, questions, or hypotheses logically linked to the research purpose?

c. Are hypotheses stated to direct the conduct of quasi-experimental and experimental research (Creswell, 2014; Shadish et al., 2002)?

d. Are the objectives, questions, or hypotheses logically linked to the concepts and relationships (propositions) in the framework (Chinn & Kramer, 2011; Fawcett & Garity, 2009; Smith & Liehr, 2008)?

5. Variables

a. Are the variables reflective of the concepts identified in the framework?

b. Are the variables clearly defined (conceptually and operationally) and based on previous research or theories (Chinn & Kramer, 2011; Grove et al., 2013; Smith & Liehr, 2008)?

c. Is the conceptual definition of a variable consistent with the operational definition?

6. Design

a. Is the design used in the study the most appropriate design to obtain the needed data (Creswell, 2014; Grove et al., 2013; Hoe & Hoare, 2012)?

b. Does the design provide a means to examine all the objectives, questions, or hypotheses?

c. Is the treatment clearly described (Brown, 2002)? Is the treatment appropriate for examining the study purpose and hypotheses? Does the study framework explain the links between the treatment (independent variable) and the proposed outcomes (dependent variables)? Was a protocol developed to promote consistent implementation of the treatment to ensure intervention fidelity (Morrison et al., 2009)? Did the researcher monitor implementation of the treatment to ensure consistency (Santacroce et al., 2004)? If the treatment was not consistently implemented, what might be the impact on the findings?