12

Limitations of Qualitative Research

Introduction

Qualitative research methods are valuable in helping to gather descriptions of other people’s ideas, thoughts, feelings, attitudes and values. As we have noted, the most commonly used data collection method is the interview, while the most frequently used method of data analysis is a version of content analysis.

Qualitative approaches to healthcare research are of a more recent origin than quantitative. Also, such approaches do not, necessarily, follow the scientific method seen in quantitative research. The degree to which the scientific approach can be applied to human beings, given their difference to other objects in the world is a matter of debate. The fact that people have consciousness and can think and ponder on their situation, means that quantitative data may not best represent a given person’s situation over time. Put simply, we can change our minds. This, in turn, means that any form of research into human subjects can only ever offer a glimpse of a situation at a particular time and in a particular context. We would, then, be advised to be careful about the degree to which we generalise our findings in any human research.

In this chapter, the limitations of the qualitative approach to research are examined. It should be noted, in advance, that – perhaps because of its newness and difference from quantitative research – some of those in the health and caring professions still view it with suspicion. Some also compare the approach rather too directly with quantitative research and expect the rules of quantitative research to apply to qualitative. They do not. We are both members of Local Research Ethics Committees. Such committees are in place throughout the UK to consider and approve medical and healthcare proposals. Any research that is likely to involve patients or healthcare workers has, first, to be approved by such committees. We are still surprised by the questions sometimes raised about qualitative proposals (e.g. ‘Where is the hypothesis?’, ‘Isn’t this study a rather subjective one?’). To raise these questions, as we have noted in previous chapters, is to miss the point of qualitative research. The point in, such work, is to describe and to collect subjective views from a range of people. Qualitative research findings cannot, by the nature of the sampling and data collection methods used, be generalised out to larger populations. The point of qualitative research is to describe and illuminate.

It would be a mistake, then, to criticise qualitative research in terms of the conditions applied to quantitative. The enterprise does not claim to be a ‘scientific’ one in the commonly accepted sense. Indeed, many qualitative commentators would question the degree to which any human research can be scientific. However, we cannot reject all the criticisms laid against qualitative research and it will be valuable to explore some of them.

Reliability and validity

These general concepts are examined in Chapter 6. Put simply, qualitative and quantitative researchers are generally united in the view that the entity being examined (rather than something else) should emerge from the results of a study (validity) and that it should do so in a systematised way (reliability). However, in qualitative research, those issues are not so clear-cut as they are in quantitative. There are, perhaps, three reasons for this.

First, the sampling methods used in qualitative research are rarely, if ever, representative. No attempts are made to obtain random samples from a larger population. Instead, as we have seen, samples are usually convenience or purposive and sometimes both. A convenience sample is one that ‘is to hand’: it is made up of people who are prepared to take part in the study. A purposive sample is a sample of people who are likely to have a view on the topic that is being investigated. Given that sampling is not randomised and, therefore, not representative of a larger population, we cannot fully address sampling validity issues in qualitative research.

Second, data collection methods, in qualitative studies, usually involve either semi-structured interviews or observation. Both of these methods have built-in problems in terms of keeping everything uniform. One interview may not be anything like the others, in a series. Interviews may vary in terms of their overall ‘quality’, their tone, and so on, over time. Again, these are a threat to both the validity and reliability of the project.

Third, people, themselves, are variable. As we shall see, people may give different responses to different people, they may offer different views on different days. Similarly, the researchers’ own ‘changeability’ comes into the equation: one researcher’s treatment of a qualitative dataset may be quite different from that of a colleague. In quantitative research (as long as the appropriate procedures are followed) this variability is not the case: a range of quantitative researchers, handling the same dataset, with the same computer program, will come up with the same results. The findings in a qualitative study are not so likely to be so ‘clean’ in this way and this further limits the degree to which we can claim to have carried out a reliable and valid study.

However, we can, as far as possible, undertake certain checks in these respects. By attempting to ‘bracket’ or put to one side our own views on the topic being researched, we can cut down the likelihood of our bringing our own views into the analysis and subsequent discussion of data and findings. We can also check our analysis of the data with another research to see whether or not there is a reasonable case for our being able to say that we have been as objective and careful in our analysis as is possible. This checking process is not without its own problems. We might want to say that simply asking another colleague to check our system of analysis is to, potentially, bring in another level of error. With a check of this sort, we may also pick up the second researcher’s biases.

Another device that is sometimes used for checking the appropriateness and validity of our analysis of the data is to discuss that analysis with the respondents, after the event. Again, such a system is hardly flawless. It seems likely that the respondents will not, themselves, be researchers and may have no comments to make on the processes that were completed. The respondents may also be pleasantly or unpleasantly surprised by the analysis and suggest changes based on this. Finally, the respondents may simply accept what the researcher has done – believing, perhaps, that the ‘researcher knows best’.

In a well designed, quantitative study, an important feature is that a study should be able to be replicated and similar findings be found (once it is established that the original study’s processes were sound). In qualitative research, it is rare that this is the case – and mostly for the reasons identified above. There are too many uncontrolled variables and possible sources of error for qualitative studies to be able to be replicated and similar findings identified. This is not so much of a limitation of the qualitative approach, but more an appreciation that the qualitative process is a very different one to the quantitative.

Many of the points in this section are revisited in the sections that follow. It is the subjective nature of qualitative research that is so often a problem for those who would question the usefulness of it. It is a subjective process both from the points of view of the respondents and also from the point of view of the researcher. It will be useful, therefore, to revisit some of the above points in a slightly different context.

Two overriding points need to be made in respect of a general approach to validity and reliability. First, the qualitative researcher must, like all researchers, be as honest and as careful in her work on any given project. Second, the researcher should always look for disconfirmation of her findings. At all stages, through the analysis and writing up of a qualitative project, the researcher should be asking: ‘have I got this right?’, ‘is something else going on here?’ or ‘could things be otherwise?’ Most research yields up new knowledge in very small increments. Any researcher – including the qualitative researcher – should be wary of any findings that seem very different from what has gone before. All researchers should, by nature perhaps, be conservative and not prone to expect large numbers of new findings. If such findings are forthcoming, they should ring a bell of caution in the head of the researcher. Such unusual new findings often, in the end, point to errors in the research process or to faulty assumptions on the part of the researcher.

Generalisability

Bearing in mind the point, made above, that the aim of qualitative research is not to generalise, it still remains a limitation of the approach. If research is intended to inform practice, then we might hope that we can draw evidence from it to apply to situations other than the ones dealt with in the research. Qualitative research is very limited in this respect. We cannot take a group of people’s views and apply them to another context, another place or another group of people. We may read them and think ‘Yes! This sounds familiar to me!’ or ‘I am surprised at these findings!’ but we cannot, then, extrapolate from them and decide that we have hard evidence that can be applied in other contexts.

It is interesting to read the conclusions drawn from many qualitative studies. Some contain recommendations. These are sometimes in the format seen in quantitative studies, where the researcher suggests changes or modifications to practice that might result from considering the findings. It is arguable, given the points noted above, that qualitative researchers cannot make such recommendations. If it is true that findings from such studies are limited to a particular time, place and sample, then it follows that such findings should not generate recommendations for other times, places and populations.

Such lack of generalisability does not, however, mean that qualitative findings are not of value. As we have noted, qualitative work illuminates, offers glimpses of how other people work and think. Such findings can help us consider our own situation and learn from the thoughts, feelings and actions of others. They can allow us to compare and contrast our own views of practice with those of others. The limitation does mean, though, that we must always bear this ‘generalisation’ rule in mind and not automatically conclude that we have obtained hard evidence to apply to our own practice. This tendency to offer recommendations and advice to others at the end of qualitative research reports is often to be noted in those who are new to qualitative methods (although it is a tendency not always missing in those with considerable experience, either!).

The nature of human beings

Human beings, because of the fact of their consciousness, are in a constant state of flux and change. If we pause, for a moment, and consider our own thinking processes, we may note that such processes are not linear: we do not, necessarily, think logically or consistently. We are affected by our emotions, by the situation we find ourselves in and – sometimes – simply by whim. We are often contradictory in the things we think. A simple, everyday example, may serve to illustrate this. If we are driving a car and another driver forces his way in front of us, we consider this bad driving. If, however, we force our own way through traffic, we may feel that what we do is necessary. We often have one rule for ourselves and another for other people.

Similarly, we can very easily change our minds and our views. What I believed two years ago, I may not believe today: what I thought to be the case, yesterday, may not be what I believe today. This must be an important challenge for the qualitative researcher, who, it would seem, is committed to capturing the thoughts, feelings and actions of others. What such researchers may have caught is merely a snapshot of a person’s thoughts, on a given day, in a particular situation. If the researcher were to conduct interviews with this person, on a different day or in a different context, he or she might gather very different data. The question must therefore be asked: ‘To what degree are these findings, to any degree, an accurate representation of what another person thinks or feels?’

We also react to different people in different ways. When we are interviewed, we are also reacting to the person conducting the interview. Do we like him or her? Do we feel intimidated, threatened, superior or inferior to him or her? What, do we consider, is his or her motives in asking a question? What will he or she do with our utterances? These factors and many more influence the sort of responses we make to the questions we are asked. Most people, as students, have experienced the situation, in a classroom or lecture theatre, in which, on being asked a question by a lecturer, have thought: ‘What sort of answer is required here?’ Similarly, in research interviewing, the subject attempts to best-guess the response that they believe is being asked for. We do not simply offer an honest and open response. We modify our replies in accordance with all sorts of feelings and views we have about what is going on at the time.

Perhaps the most clear-cut example of this takes place in a job interview. Our responses to questions are clearly aimed at ‘making a good impression’. We are, for obvious reasons, trying to present ourselves in the best possible light. We may exaggerate our good points and play down our weaker ones. Issues of confidence, self esteem and self-concept also come into play in the research interview. Few of us, in any situation, want to present ourselves in a bad light. We want other people to like us and we want to be appreciated. We may even want to offer what we perceive to be the ‘right’ answer.

An example, from a cultural study, conducted by one of the writers may help to illustrate this. The study was conducted in Thailand, where one of the cultural norms is to be polite and not to argue or contradict those perceived as being in a superior or ‘one-up’ position. The researcher visited a restaurant, in Thailand, with a respondent in the study. He noted that the owner of the restaurant was Chinese and (for reasons not, now clear) pointed this out to the respondent, who agreed. The researcher and respondent met on other occasions and the restaurant became known, by shorthand, as the Chinese restaurant (even though it only served Thai food). In a later exchange, the researcher pondered on where, in China, the owner might have come from. The respondent replied that the owner had probably never been to China, as that owner was Thai!

The example illustrates a complicated, cultural issue. First, the researcher offered a flat, unquestioned, view of the origins of the owner. In order for the researcher not to ‘lose face’ the respondent did not challenge or deny the assertion. Only later, and possibly with some difficulty, did the respondent point out the mistake that had been made. If further exchanges had not taken place, the mistake may have remained unchallenged.

In this case, the researcher quickly learned not to make such flat assertions and, working with a Thai co-researcher, often checked assumptions and ideas about ‘what was going on’ to clarify and modify his understanding. However, it seems fairly likely that, in many qualitative studies, (a) assumptions are made and go unchallenged and (b) respondents, for various reasons, attempt to ‘keep the researcher happy’.

All of this is complicated by the fact that we often do not appreciate our own biases and prejudices: we simply cannot account for all the things that we think and believe. This is true for both researcher and respondent. The researcher is often blind to his or her own values and beliefs and so is the respondent. Thus, the reporting, by both parties, of a clean and accurate account of any given situation seems fraught with difficulty. Errors, of varying degrees can creep into a study both from researchers and respondents. If the findings from such studies are then built into a ‘theory’ or into ‘recommendations’, errors may be built upon errors.

If it is true that respondents are changeable and that their thought processes vary from day to day, it is also true for researchers. The researcher may well analyse data some time after they are collected. This allows time for further errors to emerge. The research has to attempt to recall the interview and reconstruct it in his or her mind. However, many life events have taken place since the original encounter and it seems unlikely that recall of the interview, after the event, is necessarily an accurate one. The views of conversations emerging from a tape recorder or from written notes are not going to be the same as the ones that took place in real-time.

Data analysis

We have already noted one problem of data analysis. Analysis cannot, normally, take place as interviews happen (although the researcher is probably making some initial attempt at analysis during an interview).

The most detailed analysis takes place after the event. As we have noted, this can lead to bias. Our memory of events is rarely accurate – even if we have recorded interviews carefully. Our ‘reading’ of an interview, at a later date, is likely to be different to the reading of it as it took place. The ‘live’ interview is quite different to the recorded one.

One of us recalls, in a different study, undertaking a fairly lengthy interview and being impressed by the quality of that interview. He felt that much ground had been covered and that many useful things had been gleaned from the encounter. He was surprised, some time later, on playing back the tape of the interview that he found it very difficult to understand what had taken place! The ‘good’ interview had become difficult to fathom. Live interviews involve a range of dynamics that simply cannot be captured on tape or in notes. In a live interview, we respond to each other, we smile, use eye contact and understand each other in ways that are not easily recorded. Perhaps, in the future, more use will be made of visual as well as audio recording – both as means of data collection and as means of data reporting. However, it seems unlikely that any sort of recording will ever quite capture the atmosphere and the shared meanings that are present in the original encounter.

In most forms of the content analysis of qualitative data, the processes are subjective ones. Although, in the grounded theory approach to qualitative research, it is often claimed that data categories ‘emerge’ out of the data, it seems obvious to both of us that it is the researcher who devises those categories. To argue that such categories emerge is similar to saying that a block of stone ‘contains’ the eventual sculpture that the skilled artist produces. There is no sculpture in the block of stone until the sculptor fashions it. Similarly, there are no categories in a series of interview transcripts until the researcher develops them. And that process cannot be seen as an objective process. In the end, the qualitative researcher has to decide upon a series of headings – of his or her devising – and then identify which elements of the data fit under those headings. We might argue, then, that a different researcher might devise a different set of headings or that he or she might file data under those headings in quite a different way. These facts also point to another form of potential limitation: we cannot guarantee the relationship between the original dataset and what the researcher does with that dataset.

In summary then, we might say that, on a different day, with a different researcher, the data produced in a qualitative study may be different from this particular set. We might further say that what the researcher does when he or she analyses those data may vary from what another researcher might do. In the end, we are left to note a certain arbitrariness in the qualitative research process. We cannot argue that what we have captured, in any given study, is a clear picture of a person or persons’ views, nor that the findings we have can, in any sense, be extrapolated out to other groups of people. We are left, in the most extreme view, with a very particular interpretation of a very particular situation.

We might elaborate this just a little further. It seems possible that a respondent, reading what has been done with the data he or she has supplied might have a variety of responses to the account. He or she might say, at least:

(a) Yes! I agree with this account! It is a very fair representation of what I said and what I believe to be true.

(b) Yes! I agree that this is what I said at the time … however, on reflection, I was wrong and I no longer feel that this represents what I believe to be true.

(c) No! This is not an accurate representation of what I said.

Oddly enough, although the first option seems the most reassuring one, we might even question the holding of this view. If we believe, as seems to be borne out by our everyday experience, that people are dynamic, changing beings, we might wonder at a person’s ability to make such a statement with any conviction! The second option is likely to trouble us, perhaps, and make us wonder about the validity of what we have done as researchers and to question, in various ways, the value of doing a qualitative study. The third option, perhaps, might lead us to believe we have done ‘bad research’ and not been careful enough in our procedures. However, as we have noted, perhaps we should not be surprised if the third option came to be expressed, occasionally, given the seeming arbitrariness of the processes involved in analysing qualitative data.

Despite all this, we can, I believe, justify the use of qualitative methods in an almost literary way. We can, perhaps, view qualitative data as a series of stories. A moment’s thought will reveal that many aspects of our lives involve what might be called ‘storytelling’. First, we tell ourselves stories. We invent reasons for why we and others do things. We offer ourselves explanations for our own and other people’s behaviours, thoughts and feelings. We also offer other people stories. An example might be as follows. One of us, when driving into the centre of Cardiff, a busy capital city, will often comment, to his wife, on how it is busy on that day. On almost every occasion, his wife offers an ‘explanation’ of why that is the case (e.g. ‘People are getting out earlier to the shops, today’ or ‘People want to get to be able to park in town’). These are examples of ‘stories’: the author’s wife clearly cannot know why the city seems busy nor even know if the city is any busier than usual! In this way, though, we tell each other stories.

Similarly, in qualitative research, we are, perhaps, presented with stories about human lives, groups and cultures. Such stories should not be read too literally – for the reasons, at least, discussed above. Those stories can, however, help us to appreciate, a little more clearly, some aspects of the human condition. They illustrate human lives in a way that statistical information cannot. We can read these stories and, again, have various reactions to them, for example:

(a) Yes! I can understand this person’s point of view and his or her experiences.

(b) What this person says surprises me.

(c) I do not believe what I am reading.

(d) I had never thought about this before, in this sort of way.

Clearly, many other reactions are possible, but the point is that the stories that emerge out of qualitative data are not necessarily versions of ‘the truth’ (and, in the end no research is offering that). Instead, they are illuminative accounts that help us to think more clearly – or in different ways – about a healthcare topic. Or they allow us glimpses of aspects of the human condition of which, before, we were unaware. For these reasons, perhaps, qualitative research – although of a different type and order to quantitative – can be found to be valuable.

From the point of view of evidence-based healthcare, qualitative methods are taking their time in being accepted as mainstream. One reason for this is the problem of generalisation: while it is possible to generalise from quantitative studies, such generalisation is not possible from qualitative studies. However, hopefully, the illuminative aspect of qualitative research will soon be more widely recognised and appreciated. We need human, subjective accounts of life (and death) alongside the more objective, detached, quantitative ones. Quantitative research cannot account for all aspects of the human condition – particularly for the more personal, subjective aspects. To some degree, these can be captured in qualitative research, and qualitative accounts can be very useful in attempting to offer the evidence base a personal view of healthcare. Or, to put it another way: where people are concerned, you cannot simply count everything.

Stay updated, free articles. Join our Telegram channel

Join