6

Basic Concepts: Sampling, Reliability and Validity

Sampling in research

This chapter introduces the ideas of reliability and validity in the context of what it means to sample from a wider population. In Chapter 3, we noted that different methodological approaches often have different research aims associated with them. In the same way, the different approaches and aims are generally associated with different ways of sampling. However, perhaps the most basic distinction in sampling in research is the one between sampling in quantitative and qualitative research. The specifics of different sampling techniques used in qualitative and quantitative approaches are examined in detail in Chapters 7 and 13.

In this chapter, basic notions of the rationale behind sampling are explored. We start from a quantitative perspective because this is the one most commonly encountered in medical and healthcare research studies. Even when a qualitative study is reported in the literature, the sampling technique may still be one initially derived from quantitative approaches. We also believe that the notion of sampling is not peculiar to research, but is part of our everyday lives, and many of our everyday concepts of sampling are most easily understood from a quantitative perspective. At the same time, we do not ignore qualitative perspectives on sampling, and you will see these are mentioned throughout this chapter.

Samples and populations

Most of us are familiar with the national census. Infrequently, the whole population of the UK is presented with a series of forms to fill in, asking for information about their characteristics. The sheer size of this undertaking is so vast that it only takes place once every 10 years. In between times, things change, and government organisations continue to need information which reflects those changes. In the absence of resources to repeat the whole population census or survey, smaller surveys are conducted on a limited sample of the population. Similarly, in most research studies, it would be very difficult to survey, say, every person who had undergone a hernia operation and ask them about their satisfaction with care. In this situation, health researchers follow the same approach as these government surveys, and sample a certain number of patients.

In both these examples, we make the assumption that the sample is similar to the population from which it is drawn. In the first case, this is the population of all people in the UK; in the second, it is the population of all people who have had hernia operations. This is also the commonsense way in which we typically talk about samples and populations. Nor is this simply a consequence of our having been brought up with scientific jargon thrust on us from school onwards, via teachers and the media. For example, the New Testament story of Jesus begins with a survey. Mary and Joseph travel to Bethlehem essentially to be counted for tax purposes.

Sampling as an everyday pursuit

In our own experience, we think implicitly in terms of populations and samples. Imagine you wanted to get wedding photos taken, or special 50th Wedding Anniversary photos for your parents. The consequences of making the wrong choice of photographer are potentially expensive, both financially and emotionally. Before employing a photographer, you will probably ask to see a sample of her work. You might actually go to several photographers in the neighbourhood getting sample photographs from each, before finally deciding. When you do both these things, you are actually making a number of sophisticated sampling decisions which are precisely similar to the decisions faced by researchers.

Consider the following issues. When you ask to see samples of a photographer’s work, you expect that these will be reflective of all the photographs (the population of photographs) she has taken. Of course, you accept that she will have chosen her best work to show you, but that probably matters less to you than seeing the kind of photos she takes. If she specialises in highly formal posed shots, her sample will reflect that, and that information is useful to you in making your decision because you assume it is reflective of her work as a whole. As we will see later, you have accepted that the sample will not be random but will be selected by the photographer in a non-random way. This, however, does not concern you because the issue you are concerned with (style of photos) is not dependent on randomness to be an adequate sample of the photographer’s work (unless she takes photographs in different styles and has chosen to show you only one such style). Many sampling decisions in research are variants of these two issues: What characteristics of the population do I wish to capture in the sample? What approach to sampling will best capture these?

One aspect of decision making about sampling we have not covered so far is the notion of what one’s aim is in capturing particular characteristics of the population in the sample. Yet, in the photograph example, this point emerges readily. Your aim, of course, is to get the best possible set of pictures for your wedding or your parents,given the circumstances. These circumstances give rise to a whole series of further sampling decisions. To give just two examples of circumstances that affect our sampling decisions, what you really are aiming for in this example is to get the best possible set of photographs within a reasonable time, at a price you can afford. Because of the need to meet the reasonable time aim, you need only sample from photographers you know do not have a long waiting list. Similarly, to meet this aim, it will only be practical to sample a certain number of photographers (perhaps only those in your nearest large town or nearest city). Finally, to satisfy the at a price you can afford aim, there is no point in sampling from photographers whose work you know you cannot afford.

If your aim is different, your approach to sampling will be different. If you want to know what kind of range of wedding photographs exists (say, because you have not seen many and want to see what different sorts look like), you will no longer be concerned with the at a price I can afford goal. You just want to know about style, not about price, because you are not buying yet, although price may affect your decision later. The within a reasonable time goal does not concern you much either (except if you spend so long looking at wedding photos that you run out of time before the wedding!). In this situation, you are much more likely to want a broad ranging sample. First, you probably want a random selection of each photographer’s work, rather than ones that photographer thinks are suitable for you. Second, you will want to see the work of a great many photographers, not just local ones. You may even go so far as to take a random sample of all UK photographers, if time and money are no object. This is because you are aware that the broader the sample, the better, in giving you responses which are broad ranging.

Finally, your first concern might be to engage a photographer who is sympathetic to the needs of bride and groom on their wedding day, or your parents on the day of their 50th Wedding Anniversary. In this situation, your sample may be very much smaller than in the second example we gave above, because the kind of information you want from them may be very detailed, which in turn will affect the amount of time it takes you to gather it from them. In this case, you may decide on personal visits from a small range of local photographers, during which you discuss the event, their approach to it, and also get an idea of their personality. Here, you have accepted that the sample is likely to be quite unreflective of the whole population of photographers, or even photographers in your town for hire at a good price, because you believe this disadvantage in your approach to sampling is offset by the advantage to you in gaining the very detailed information you need from a few respondents.

In each of these situations, our ideas about sampling are different because of the different aims we have. We also adjust our ideas of what constitutes the population. In the first example, we have defined the population as wedding photographers who can do the job in a reasonable time, in the second as all photographers, and in the third as local photographers. Naturally, as you will have gathered from the above, all these supposed populations are actually themselves samples. For example, wedding photographers are themselves a subsample of the population of all professional photographers, who are again a subsample of the population of all photographers. This is where confusion sometimes arises, but is best settled by considering a sample to be a smaller element examined from some larger entity. The population is that larger entity, but may itself be a smaller element of some still larger entity. Ultimately, all photographers are simply a sample of a broader population of all human beings. We define the population according to our needs, and sample from it in the same way.

Turning to healthcare, the sampling decisions we took above are also evident in much health research. We may, for example, survey hernia patients in our own hospital, even if we know that that hospital’s approach to hernia repair and its care is very different from many other hospitals’ care. If our aim is simply to monitor and improve care in our own hospital, that does not matter. If we wish to make some broader claim (say, about the best way to organise care for hernia patients in general) we will accept the inconvenience of having take a broader sample, perhaps choosing other hospitals to sample from on a random basis, and employing some systematic form of sampling within each institution. On the other hand, if we are at a stage where we want detailed accounts of the patient’s experience of the patient journey in repair of hernias, including what their impressions were of what happened to them, their feelings at the time, their attitudes to others around them, the atmosphere on the ward while they were being treated, and so on, our decision will be similar to the one described in our description of the search for a sympathetic photographer. We will choose a very few people, gain a great deal of information from them and accept that this information may be particular to them rather than reflective of the personal experiences of hernia repair patients as a whole. Naturally, we would probably also want to argue that the insights we had gained from them would be useful in considering the possible experiences of others, but the rationale we gave for making such an assertion would not be on grounds of the way in which the individuals in the sample might reflect individuals in the population at large.

Samples and representativeness

On several occasions, we have described samples as being more or less reflective of the populations from which they are drawn. This is the concept of representativeness and is a basic notion in sampling. Roadside breathalyser tests measure alcohol levels in the human body. If it were the case that the sample of such levels did not represent the levels present in a person’s body as a whole, there would be little chance of gaining a conviction for drink driving, and we should agree that this would be fair enough, because the sample would not be telling us about the actual population of alcohol levels in that person’s body at the time. Indeed, there have been a number of recent celebrated court cases in which contamination of human tissue samples has led to the acquittal of the accused. These cases are essentially examples of situations where there could be no confidence that the samples were truly representative of the population of that person’s tissue.

Once again, if we extend this idea to healthcare practice, we can see that it would be futile to base interventions on samples that were not representative of the whole. If a blood sample is not representative of the patient’s blood as a whole, it does not tell us much about the population (all that patient’s blood) or, by extension, about their health status. The same is true of psychological measures. If we measure a person’s level of depression using an accepted scale, we do so because we believe that the scale gives us a meaningful sample of their mood, and base our interventions on its results. This is because we are confident that the scale results are representative of the patient’s mood as a whole (the population of their mood levels, if you like).

In qualitative research, many researchers have wanted to deny the importance of representativeness. This argument is often misunderstood, however. Qualitative researchers are certainly concerned with representativeness, even when they wish only to describe the experiences of their participant group, rather than drawing inferences from that group to a wider population. This is because they will want us to have confidence that the responses they have gained from their participants are, in fact, representative of what their respondents think (in other words, that they are a representative sample of those people’s thoughts). The central difference between approaches to representativeness in quantitative and qualitative research is that the two traditions emphasise different issues in defining representativeness.

In quantitative research, our confidence in representativeness comes overwhelmingly from sampling technique. Have we sampled in such a way that we can be sure that this individual, or this sample of blood, is truly representative of the population from which we claim to have drawn it? In qualitative research, we are more likely to be concerned with theoretical issues. Do the responses genuinely reflect some characteristic of the whole? In our photography examples, both quantitative and qualitative concerns are reflected in the first example. We want a representative sample of photographers (albeit within certain constraints). However, we are not concerned that the examples of their work may be non-random because, whatever the sample they choose, we know it represents a theoretically important aspect of their work – their style. In the second example, quantitative notions prevail. Representativeness of the sample is the key element, and all our efforts will be geared to ensuring this (see Chapter 14). Finally, in the third example, our approach is very much at the qualitative end of the spectrum in sampling terms. There is no representativeness of the sample from the quantitative perspective, but it is fit for the researcher’s purpose, because it allows the in-depth exploration necessary to bring to the surface issues of concern to the researcher. Moreover, the researcher does not need to worry that the responses of the participants will not be representative of the broader range of their thoughts, feelings and attitudes. Provided the information is sensitively gathered, the responses will, by definition, bring to the surface aspects of the photographer’s personality and so on, and this is the information required by you in this context.

Validity and reliability

Both qualitative and quantitative research are concerned with the principles of validity and reliability, although, as we shall see in Chapters 12, 13 and 14, the two research approaches have slightly different criteria for determining these and sometimes different terminology to reflect the underlying constructs. Nevertheless, qualitative and quantitative researchers are generally united in the view that the entity being examined (rather than something else) should emerge from the results of a study (validity) and that it should do so in a systematised way (reliability).

Validity refers to several ways in which we can be confident that the thing under investigation is truly emerging. Thus, we speak of a measure as having validity if it genuinely measures the thing it claims to measure. A structured interview which claimed to measure quality of life, but actually measured depression levels would be an invalid measure of quality of life, because low mood is only one part of this. Any conclusions we drew about quality of life based on such a measure would likewise be invalid. We also describe research studies as having validity. Typically, this terminology has been borrowed and extended from experimental research and is discussed in more detail in Chapters 13 and 14.

Reliability is the extent to which an entity is measured in a consistent way. To be reliable, a measure needs to be repeatable (giving similar responses in the same conditions) and reproducible (giving similar responses in different conditions).

Validity, reliability and sampling

Internal validity refers to our confidence that changes in a patient’s status are caused by the treatment they are given, rather than some extraneous event (upbringing, personality, genetic inheritance) and is greatly increased by the use of random sampling (see Chapters 14, 15 and 18). In this situation, adequate sampling increases confidence that we are truly measuring the relationship between treatment and changes in patient status – the thing we claim to be measuring.

The notion of external validity is related to generalisability and refers to how confident we can be that results found in our sample reflect attributes of the population. As we suggested in the photography example, we are not always concerned with this, and the more tightly we define the sample and population, the more we decrease external validity. For example, one enduring criticism of randomised controlled trials (see Chapter 17) is that, although participants may have been randomly sampled from the population concerned (thus increasing internal validity), the population (e.g. males between 18 and 65 with no other illnesses attending South of England teaching hospitals and willing to participate) is itself so untypical that generalisability to the greater population of hernia repair patients is low. In this situation, selective sampling decreases external validity. Validity is low if we claim the study is genuinely measuring the responses of all hernia repair patients.

Reliability in sampling extends the idea of reliability of measurement by asking if the sampling strategy and techniques collect from the population in a replicable way. For a sampling strategy to be reliable, it should garner the same results from a population every time we sample, provided the attribute being examined remains unchanged. In most situations, a random sampling approach is an important beginning in ensuring such reliability of results, but other issues such as interviewer technique and instrument reliability also impact heavily on reliability of data collection. Although they seem unrelated to sampling itself, actually there are obvious relationships between sampling approach and data collection approach and technique. For example, there would be little point in taking a random sample of cancer patients if we wanted to undertake, say, a dozen in-depth interviews with them about their experiences of delays in diagnosis and treatment. A random sample with such a small number would never give us sufficient participants who had experienced delays and could talk in depth about them and had a broad range of experiences, but the use of in-depth interviews is arguably the most reliable way of getting such information.

Horses for courses

This final point gives us, once again, the key feature of approaches to sampling. There are no right and wrong ways to sampling, only more and less appropriate ones under any given set of circumstances. Appropriate sampling should reflect the aims of the research study, as well as the practical constraints under which the study takes place. Our decisions about sampling take place within these two contexts.

Stay updated, free articles. Join our Telegram channel

Join