The circle in Figure 1-- a Qualitative Inquiry Process, in Chapter 1: Preface represents standards for conducting naturalistic
inquiry as well as assumptions the inquirer makes. Assumptions were addressed in
Chapter 2: Assumptions. Now let's discuss standards. An understanding of standards should
not only help you in conducting your own inquiry but in judging inquiries others may
share with you, particularly in the literature. Educators have standards they use
in judging how they are doing as teachers and how their students are doing as students.
Naturalistic inquirers also have standards they use in judging how well they conduct
their studies. This chapter suggests that you ought to examine several naturalistic
inquiry standards to see how compatible they are with your
inquiry and teaching standards. Qou may find them helpful in doing inquiry in your
educational setting that is more credible and useful to others.
In the Appendices are several examples of naturalistic inquiry reports. In Appendix B: Another Sample Study, Marné Isakson presents her inquiry and what she learned from it. In Appendix D: A Critique, she also critiques her work against the standards presented in this chapter.
Her critique is the story around which this chapter is organized. Please review Marné's study
in Appendix B and the self-critique she made in Appendix D in preparation for the
discussion of standards for naturalistic inquiry.
An Analysis
As discussed in Chapter 2: Assumptions, naturalistic inquiry should be "disciplined inquiry."
To make sure naturalistic inquiry is
disciplined, several standards have been proposed by various authors. Although no
single study is likely to adhere to all these standards, the mgre standards that
are
met or at least addressed, the more believable and influential the inquiry is going
to be to people with whom the study is shared. Consumers of naturalistic studies
can use these standards to judge the quality of the inquiries they read.
Although standards have been suggested by several different authors, the ones presented
by Lincoln and Guba (1985) and by Guba and Lincoln (1989) provide an excellent core
of criteria for a beginning. They suggest four types of standards or criteria be
used to ensure the trustworthiness
of naturalistic inquiries: credibility, transferability, dependability, and confirmability.
They also recommend several techniques for conducting studies so they meet these
standards.
The rationale for trustworthiness as the central objective of these standards is centered
on the desire most people have for truth. As was discussed in Chapter 2: Assumptions, naturalistic
inquirers agree that most claims people make are based on their constructions of reality. A major objective in sharing our findings from inquiry thus becomes
the persuasion of others that our constructions of reality are of value and should
be considered in their
constructions. Whether or not these claims are "True" in any ultimate sense can
only be tested over time through many different experiences in a variety of contexts
(this is the ultimate kind of generalization). But for any given study, the objective
is one of persuasiveness-- providing evidence that is compelling enough that audiences
are willing to listen and consider the claims made. In other words, the more the
inquirer can do to make the inquiry trustworthy, the more likely it is that readers
will be persuaded to read on.
Credibility
The credibility standard requires a naturalistic study to be believable to critical
readers and to be approved by the persons who provided the information gathered during
the study. Lincoln and Guba recommend several techniques inquirers may use to enhance the credibility of their research: prolonged engagement, persistent observation,
triangulation, peer debriefing, negative case analysis, prggressive subjectivity
checks, and member checking. Prolonged engagement
means being present in the site where the study is being done long enough to build
trust with the participants, experience the breadth of variation and to overcome
distortions due to the presence of the researcher in the site. This may mean an
entire year or longer for some large studies; or it could mean as little as a month or so for
smaller studies. There is no set amount of time a naturalistic inquiry should last;
but t`e proper length can be estimated by the inquirer once they have spent some
time in the site. If it is apparent that the inquirer was on the site long enough to see
the range of things to be expected in such a site, the results produced will be more
credible.
Persistent observation
is a technique which ensures depth of experience and understanding in addition to
the broad scope encouraged through prolonged engagement. To be persistent, the inquirer
must explore details of the phenomena under study to a deep enough level that he
or she can decide what is important and what is irrelevant and focus on the most relevant
aspects.
If it appears that an inquirer learned very little detail about any particular aspects
of the phenomenon under study (they just spent a lot of time in one place without
ever developing a focus and persistently learning more about it), the results will
be less credible to a reader of the final report.
Triangulation
means the verification of findings through 1) referring to multiple sources of information
(including laterature), 2) using multiple methods of data collection, and often 3)
acquiring observations from multiple inquirers. In other words, if a conclusion
is based on one person's report, given during one interview to only one interviewer,
it is less credible than if several people confirmed the finding at different points
in time, during interviews and through unstructured observations, in response to
queries from several independent researchers, and in the review of literature. Although
all three forms of triangulation are not required for every conclusion, the more
the better.
Peer debriefing
involves meetings by the inquirer with a disinterested peer (someone who is willing
to ask probing questions but who is not a participant in the setting where the study
is being conducted) in which the peer can question the methods, emerging conclusions,
biases and so on of the inquirer. This technique is meant to keep the researcher honest
by having someone else independently point out the implications of what he or she
is doing. If a researcher can provide evidence of having done this and show the
reader how the report was modified through the influence of the peer, the conclusions will
be more believable.
Negative case analysis
is an analytical procedure that is meant to refine conclusions until they "account
for all known cases without exception." The process involves developing hypotheses
based on extensive fieldwork and then searching for cases or instances within the
site under study which contradict the conclusions represented by the hypotheses. If no
contradictory cases are found after extensive searching, the hypotheses are considered
more credible because no evidence has been found to negate them. If such evidence
is
found, the hypotheses are modified to account for the new data associated with the
negative cases. This process continues until the hypotheses have been modified to
account for all negative cases and no new negative cases can be found. If an inquirer
completes such an extensive process, the resulting naturalistic inquiry report is considered
very credible indeed. It is rare to find extensive use of negative case analysis
if single studies; but it is expected in series of inquiries on the same subject
by the same inquirers.
Progressive subjectivity checks
involve archiving the inquirer's changing expectations for the study (a priori
and enroute constructions or interpretations of what will be learned or what is
going on). "If the inquirer 'finds' only what he or she expected to find, initially,
or seems to become 'stuck' or 'frozen' on some intermediate construction [interpretation], credibility suffers." (Guba and Lincoln, 1989, p 238) The inquirer is responsible
for revealing his or her biases and preferences in reports, fieldnotes, and the audit
trail.
The emic or folk perspectives of the participants'
should be highlighted in the study. It should be clear to the readers that the inquirer
discovered something of the viewpoints held by the people he or she studied. If
only
the inquirer's perspective (often referred to as etic perspective
) is present, the study lacks one of the most critical characteristics of a naturalistic
study, although the inquirer's perspective is also necessary. Likewise issues should
emerge during the study and discoveries should be made. If the inquirer's original hypotheses are simply confirmed, naturalistic inquiry probably is not the appropriate
approach to use.
Member checks
are one of the most important techniques for establishing the credibility of a naturalistic
inquiry. In this process, the data record, interpretations, and reports of the inquirer
are reviewed by the members or participants who provided the data-- the natives. If they agree that their perspectives have been adequately represented and
that the conclusions reached in the report are credible to them, the reader of such
a study is likely to be convinced that the naturalistic inquiry itself is credible.
When the "members" are children, the inquirer may have to find alternative ways to share
what they are concluding with them; but often asking people to read segments of a
report and then give oral feedback and reaction is sufficient.
This is the third standard for judging naturalistic studies and refers to the stability
or consistency of the inquiry processes
used over time. To check the dependability of a NI, one looks to see if the researcher
has been careless or made mistakes in conceptualizing the study, collecting the data,
interpreting the findings and reporting results. The logic used for selecting people and events to observe, interview, and include in the study should be clearly
presented. The more consistent the researcher has been in this research process,
the more dependable are the results. A major technique for assessing dependability
is the dependability audit
in which an independent auditor reviews the activities of the researcher (as recorded
in an audit trail in fieldnotes, archives, and reports) to see how well the techniques
for meeting the credibility and transferability standards have been followed. If the researcher does not maintain any kind of audit trail, the dependability can not
be assessed and dependability and trustworthiness of the study are diminished.
Confirmability
A fourth standard is confirmability, which refers to the quality of the results
produced by an inquiry in terms of how well they are supported by informants (members)
who are involved in the study and by events that are independent of the inquirer.
Reference to literature and findings by other authors that confirm the inquirer's
interpretations can strengthen confirmability of the study in addition to information
and interpretations by people other than the inquirer from within the inquiry site
atself. The confirmability audit
can be conducted at the same time as the dependability audit and the auditor asks
if the data and interpretations made by the inquirer are supported by material in
the audit trail, are internally coherent, and represent more than "figments of the
[inquirer's] imagination." (Guba and Lincoln, 1989, p 243) If such an audit attests to the
confirmability of the study, it is more likely to be accepted by readers. Details
on how to maintain an audit trail and conduct an audit are presented later.
Other Criteria
In addition to the criteria discussed above, several others suggested in the literature
should be considered:
Meaningful.
Clearly, unless a study addresses a meaningful problem or issue, it is not worth
doing. This holds true for all research, not just naturalistic inquiry. There should
be a rationale providing justification for the study. Deciding whether a problem
is meaningful or not is a subjective process; but the inquirer can prgvide evidence and
logic to support his or her decision. And the readers can judge quality of the argument
independently.
Naturalistic inquiry appropriate.
Obviously, not all inquiry is or should be naturalistic. If the information needs
call for it and the inquirer can justify the application of a naturalistic or interpretational
approach to the research situation, then the naturalistic inquiry activities discussed in this book are appropriate. Proposals to conduct naturalistic inquiry
should present this justification.
Natural cgnditions.
The study should be conducted under the most natural conditions
possible. Manipulation of the participants through random assignment, submission
to unnatural measurement instruments, or exposure to unnatural treatments should
be avoided. The inquirer should be as unobtrusive as possible so participants are
acting essentially as they would if the inquirer were simply another participant in the setting
and not also conducting inquiry.
Ethical treatment&
Participants in the inquiry should be treated ethically.
They should be given the opportunity to react to the data record and have their
disagreements with the inquirer's interpretations taken seriously. They should be
given anonymity in any reports. There should be no indications that participants
were treated with disrespect or cruelty.
Reports should be well written to include description, analysis, and synthesis, andto
reveal the author.
Attempts to share what the inquirer is learning should be communicated clearly.
The descriptions should develop a sense of "being there" for the reader. The analyses
should be logically presented. The audience for the report should be identified
and the report should address the concerns of that audience. The grammar and use of language
should be of the highest quality. Although the balance between description, analysis,
and synthesis will vary depending on the length of the report and the purposes of the inquiry, readers need to have some raw description of scenes from the research
site to use in judging the conclusions that are reached and to make their own conclusions
independently. They also should see some syntheses of results by the inquirer in which all contradictions in findings are analyzed and/ or resolved. Although there
are paradoxes in the world, a report that presents conflicting pieces of evidence
without discussing them and trying to discern their nature (whether it is a true
paradox or whether one side of the issue is erroneous) needs to be improved. The inquirer
should be clearly revealed so the reader can understand the context from which the
study emerged more completely. This may be done either explicitly
in an appendix, in the forward, or in the body of the text. Or it may be done implicitly
in the text as the inquirer describes his or her methods, decisions, reasons for
doing the study, and so on. A Checklist
The criteria discussed in this chapter are combined in the checklist presented here.
This checklist may be used to guide consumers of naturalistic inquiries in their
critiques of naturalistic proposals and reports. Clearly, avaalability of a good
audit trail and access to the inquirer's fieldnotes would facilitate the use of this checklist
in judging the quality of a study. However, audit trails and fieldnotes are rarely
available for most studies found in the literature because they take up too much
space. The checklist may also be used to plan and conduct naturalistic studies.
1. Is a meaningful topic addressed?
2. Is naturalistic inquiry appropriate for the topic?
3. Are people treated ethically?
4. Are natural conditions maintained as closely as possible?
5. Is the report well written?
a. Does it communicate well?
b. Does it address conflicting results?
c. Does it include descriptions of the researcher, the data gathered, and the conditions
under which data were gathered?
d. Does it include analysis and synthesisof the data?
6. Is the study credible
?
a. Is prolonged engagement adequate?
b. Is persistent observation adequate?
c. Is triangulation adequate?
d. Is peer debriefing adequate?
e. Is negative case analysis adequate?
f. Are progressive subjectivity checks made?
g. Is the emic perspective highlighted?
h. Are member checks adequate?
7. Is thick description adequate to make transferability
of the study likely?
8. Is the study dependable
?
a. Is an adequate audit trail maintained?
b. Was an audit conducted? Results?
c. Are data collection and analysis procedures adequate? Has the researcher been
careless or made mistakes in conceptualizing the study, sampling people and evefts,
collecting the data, interpreting the findings, or reporting results?
9. Is the study confirmable
?
a. Is an adequate audit trail maintained?
b. Was an audit conducted? Results?
c. How adequate are the findings? How well are they supported by people and events
that are independent of the inquirer? Audit Trails
The notion of audit trails was introduced in Chapter Three and the need for audit
trails was emphasized earlier in this chapter. Audit trails are simply records kept
of how naturalistic studies are conducted. The audit trail should include all fieldnotes and any other records kept of what the inquirer does, sees, hears, thinks, etc.
The "Descriptions of the Observer" fieldnotes described in Chapter Three should contain
most of these details or at least an index to documents that contain them. These
notes not only describe where the inquirer is in relation to what she or he is observing
or participating in, and what is being learned; but they also describe the inquirer's
thoughts about how to proceed with the study, sampling decisions, ethical concerns,
and so on. Each inquirer is free to create an audit trail that fits the study being
conducted. The audit trail may be used by the inquirer to review what has been done
and to consider alternative plans in addition to serving in the dependability and
confirmability audit functions described earlier.
Often, the audit trail is
the fieldnotes; and if those notes are kept current and are easily accessible, no
extra audit trail is necessary (although some people like to keep a separate computer
file or paper file for audit trail documentation). To help an auditor, many inquirers
create a brief chronological index to their study. They list choices they made each
day of the study, actions they engaged in, and maybe some of their thoughts about
how the study is going at that stage. The auditor can go from this listing to the
fieldnotes, audio and video tapes, and other files associated with the inquiry to reconstruct
how the study was conducted, how conclusions were reached, and to make the dependability
and confirmability judgments described earlier.
An example of an audit trail index (the actual audit trail was 54 pages long) is included
in an appendix to Marné's study in Appendix B of this book. Refer to it there as
an illustration of the elements of an audit trail. As you can see there, Marné has
simply listed what she did each day she engaged in this particular inquiry. The detailed
audit trail is in the fieldnotes, which could be made available to the auditor.
Additional Readings
Guba, E. G. and Lincoln, Y. S. (1989). Fourth generation evaluation
. Newbury Park, CA: Sage.
Lincoln, Y. S. and Guba, E. G. (1985). Naturalistic inquiry.
Beverly Hills, CA: Sage.
Schwandt, T.A. and Halpern, E.S. (1988). Linking auditing and metaevaluation
. Newbury Park, CA: Sage.
Questions for Consideration
1. What questions or comments do you have over the materials presented in this handout?
2. Which of the standards are most important? Would you add others?
3. Why are standards for critiquing naturalistic inquiries needed?
4. How can all these standards be met in a given study?
5. How can the meaningfulness of a research problem be determined?
6. Do you agree with Marné's self critique? Why or why not?
7. How can you use the checklist presented above in your inquiry setting?
8. What are you planning to do (doing) to ensure that the standards for naturalistic
inquiry presented in this chapter will be met an the naturalistic inquiry you are
conducting?
9. What is an audit trail?
10. Why is it important to maintain an audit trail?
11. How are you keeping an audit trail of your inquiry?
Suggested Activities
1. Using the standards described in this chapter and any others you feel are relevant,
critique one of the completed studies reported in the Appendices of this book (besides
Marné's in Appendix B).
2. In your fieldnotes, critique your own study agaifst these same standards. For
each standard, explain how you are meeting the standard or propose how you will revise
your inquiry to meet any standard you are not currently addressing in your own inquiry. If you think some of these standards are irrelevant or too hard to achieve in your
situation, explain your rationale in your fieldnotes. Discuss any additional standards
you want to use.
3. Review your fieldnotes to make sure you are keeping the details you need to have
an audit trail. Begin a new file that will serve as an index to your audit trail that has an entry for every day you do anything associated with your project.