Chapter 3: Issues of Trustworthiness in qualitative and mixed method studies

Rather than issues of validity that we saw in quantitative studies, qualitative (and mixed methods) have trustworthiness issues. The first issue is credibility, which is comparable to internal validity. This is getting at the credibility of your data, common methods used are triangulation, prolonged contact, member checks, and saturation. You want to show that your data are as accurate as possible.

The second issue is transferability, which is comparable to external validity. This is getting at the generabilizability of your data to other groups. Common methods used are thick description and a variation in participant selection.

The third issue is dependability, comparable to reliability. You want to show the accuracy of your data methods, common methods are audit trails and triangulation. Triangulation is accomplished by asking the same research questions of different study participants and by collecting data from different sources and by using different methods to answer those research questions. Member checks occur when the researcher asks participants to review both the data collected by the interviewer and the researchers' interpretation of that interview data. Participants are generally appreciative of the member check process, and knowing that they will have a chance to verify their statements tends to cause study participants to fill in any gaps from earlier interviews.

The fourth issue is confirmability, comparable to objectivity. This is the degree to which the findings are the product of the focus of the study and not of the biases of the researcher One way to do this is through an audit trail. An adequate trail (or records) should be left to enable the auditor to determine if the conclusions, interpretations, and recommendations can be traced to their sources and if they are supported by the inquiry.

If you are using another coder(s), you must show how you will demonstrate intercoder reliability. Interrater or intercoder reliability is used to reduce bias by having multiple people code the data. How you go about this and how you resolve any discrepancies needs to be detailed. 

Next time I will post an updated blog index. Do you have an issue or a question that you would like me to discuss in a future post? Would you like to be a guest writer? Send me your ideas! leann.stadtlander@waldenu.edu

Chapter 3: Threats to Validity- construct and statistical validity

The final section under Threats to Validity for quantitative and mixed method studies asks you to describe any threats to construct or statistical conclusion validity. Let's start with a definition of construct. A construct is an attribute, proficiency, ability, or skill that happens in the human brain and is defined by established theories. For example, "resilience" is a construct. It exists in theory and has been observed to exist in practice.

Construct validity has traditionally been defined as the experimental demonstration that a test is measuring the construct it claims to be measuring. Such an experiment could take the form of a differential-groups study, wherein the performances on the test are compared for two groups: one that has the construct and one that does not have the construct. If the group with the construct performs better than the group without the construct, that result is said to provide evidence of the construct validity of the test. An alternative strategy is called an intervention study, wherein a group that is weak in the construct is measured using the test, then taught the construct, and measured again. If a non-trivial difference is found between the pretest and posttest, that difference can be said to support the construct validity of the test.

There are a large number of threats to construct validity- too many to discuss here, but I do suggest that you take a look at http://www.socialresearchmethods.net/kb/consthre.php when you are ready to write this section. The author does a very nice job laying out the many types of possible threats.

The last issue is statistical conclusion validity. Statistical conclusion validity is the degree to which our conclusions about the relationship between your variables based on the data are correct or ‘reasonable’. This is getting at the two types of statistical errors that can occur: type I (finding a difference or correlation when none exists) and type II (finding no difference when one exists). Statistical conclusion validity concerns the qualities of the study that make these types of errors more likely. Statistical conclusion validity involves ensuring the use of adequate sampling procedures, appropriate statistical tests, and reliable measurement procedures. If you would like a more in depth discussion of this topic, please see http://www.socialresearchmethods.net/kb/concthre.php 

Next time we will continue our review - Chapter 3: Issues of Trustworthiness in qualitative and mixed method studies. Do you have an issue or a question that you would like me to discuss in a future post? Would you like to be a guest writer? Send me your ideas! leann.stadtlander@waldenu.edu

Chapter 3: Threats to Validity- internal validity quant and mixed methods

Previously we looked at external validity in quantitative and mixed methods studies, today we look at internal validity. Internal validity refers to whether an experimental treatment/condition makes a difference or not, and whether there is sufficient evidence to support the claim. Some threats to internal validity include:

•History--the specific events which occur between the first and second measurement. People are affected by elements outside of the study. Let's use an extreme example of this, say you were interested in fear of flying and gave people a survey examining this variable on Sept 9, 2001. The participants then went through a de-sensitization training for a week and came back on Sept 16, 2001 and were retested. They are also going to be affected by an historical event outside of the study- the traumatic events of 9/11, and you would need to account for this.

•Maturation--the processes within subjects, which act as a function of the passage of time. i.e. if the project lasts a few years, most participants may improve their performance regardless of treatment. As people age they change, so if you were doing a study that lasted any period of time, you need to realize that they will change without your intervention. This is often why a control group is used, so the normal changes that occur can be compared with those of the treatment.

•Testing--the effects of taking a test on the outcomes of taking a second test. Simply taking a test can change how people think, they also cannot forget what they read in the first test. So if you give a second test people will have thought about their first answers and may change them in the second test because of that thinking process.

•Instrumentation--the changes in the instrument, observers, or scorers, which may produce changes in outcomes. Many things can affect the results, minor changes in wording, having additional people in the testing area, having different people score the test all may change the results.

•Statistical regression--It is also known as regression to the mean. This threat is caused by the selection of subjects on the basis of extreme scores or characteristics. Give me forty worst students and I guarantee that they will show immediate improvement right after my treatment, not because of my great treatment, but because they expect to do better.

•Selection of subjects--the biases which may result in selection of comparison groups. Randomization (Random assignment) of group membership is a counter-attack against this threat. However, keep in mind that randomization is only effective with large samples.

•Experimental mortality--the loss of subjects. For example, if you require people to participate in multiple training sessions, some will drop out. Those who stay in the project all the way to end may be more motivated to learn and thus achieved higher performance.

•Selection-maturation interaction--the selection of comparison groups and maturation interacting which may lead to confounding outcomes, and erroneous interpretation that the treatment caused the effect. A great example is if you had girls in a class assigned to one treatment and the boys assigned to another treatment. You compare them and discover there is a treatment difference. However, you do not know if it is the treatment that caused the differences or was it any differences in development between the girls and the boys.

In this section of your paper, you need to think through the various internal validity issues and how you will address them.

Next time we will examine threats to validity- construct and statistical validity. Do you have an issue or a question that you would like me to discuss in a future post? Would you like to be a guest writer? Send me your ideas!

Chapter 3: Threats to Validity: external validity – quantitative and mixed methods studies

The threats to validity section in the quantitative and mixed methods checklist is one of the trickiest to understand. Let's see if I can make some sense of it for you. It begins with threats to external validity. External validity is the extent to which the researcher can conclude that results apply to a larger population (generalizability). Here are some common threats to external validity.

•Reactive or interaction effect of testing--a pretest might increase or decrease a subject's sensitivity or responsiveness to the experimental variable. Therefore, giving a pretest changes your participants, they will respond differently later because they took the pretest.

•Interaction effects of selection biases and the experimental variable. You may unintentionally choose people that have particular biases. For example if you are doing an online survey about use of the internet- you will only have people participate who are already comfortable enough with the computer and internet to choose to participate in an online survey. You will be missing people who are not comfortable with computers.

•Reactive effects of experimental arrangements--it is difficult to generalize to non-experimental settings if the effect was attributable to the experimental arrangement of the research. So, if you are doing some type of experiment in a controlled setting (picture a quiet psychology lab room), there is no way to know what will happen when a similar occasion occurs in the real world.

•Multiple treatment interference--as multiple treatments are given to the same subjects, it is difficult to control for the effects of prior treatments. People cannot "unlearn" something, so whatever has happened to them previously will affect future learning/ experiences.

In this section of chapter 3, you need to think through what are the threats to external validity in your study. Keep in mind that no study is perfect, it is ok, in fact, it is expected that there will be issues. The important thing is that you recognize them.

Next time, we will look at threats to validity- internal validity quant and mixed methods. Do you have an issue or a question that you would like me to discuss in a future post? Would you like to be a guest writer? Send me your ideas! leann.stadtlander@waldenu.edu

Chapter 3: Instrumentation in mixed methods studies

For mixed methods, the instrument section has to accommodate both qual and quant instruments. For the qualitative components, identify each instrument (observation sheet, interview protocol, focus group protocol, videotape, audiotape, artifacts, archival data, and other kinds of data collection instruments). State the source of each item and provide the permission for it in your appendix.

For published instruments, identify who developed it, where and with what populations it has been used. State why you think it is appropriate for your study, and any cultural or context issues that might be present with your population.

If you are designing qualitative instruments, explain how you developed them – what was the basis for them? How will you establish content validity?

Similarly, for the quantitative components, explain the background of each instrument. Discuss validity and reliability in previous studies and where it has been used before.

The next section is how you will recruit participants for each component (qual and quant). Go into detail on how and where the data will be collected for each component.

Finally, you need to lay out your data analysis plan for each component. For the quantitative aspects, indicate your hypotheses and what statistical tests will be used for each. How will you interpret the results? For the qualitative portion, indicate how you will code the transcripts and how you will handle discrepant cases. Then you need to integrate the two types of data and compare their results. How will you do this? 

Next time we will continue our review - Chapter 3: Threats to Validity. Do you have an issue or a question that you would like me to discuss in a future post? Would you like to be a guest writer? Send me your ideas! leann.stadtlander@waldenu.edu

Chapter 3: Instrumentation in quantitative studies

Previously, we looked at instrumentation in qualitative studies, this time we move to quantitative studies. Generally, students will be relying on previously published instruments for quantitative studies. I do not recommend developing new surveys for a doctoral study. I will explain more on this below.

You will begin this section by discussing each instrument you will be using, who developed it and when. Then indicate why you have chosen it, and why it is appropriate for your study. You will need permission to use the instrument from the developer- include it in your appendix. Then go into detail about the published validity and reliability values that are relevant for your study (those using similar populations). Finally, you will discuss when and with what populations it has been used and how validity and reliability were established for each study.

If you are developing your own instrument, first describe the basis for its development. Did it come through items indicated in the literature? Did you or plan to do a pilot study to refine the questions? You will need to provide evidence for its reliability and validity, this typically requires extensive testing (100s of participants). Finally, show how the instrument will answer your research questions. If you are developing your own instrument, it will require quite a bit of testing and additional work; again, I do not recommend this for a dissertation.

If you are doing an intervention involving manipulation of an independent variable,

there are issues that will come up with the IRB, address them early! As far as c. 3, identify any materials that will be used in the intervention. Indicate who developed the materials and where they have been used in the past (you also may need permission from the developer to use them). If you developed them, indicate how that was done. Provide evidence that another agency will sponsor the intervention.

Next, you need to operational each variable. So for example, if you are interested in resilience, define it and how you will be measuring it. Then talk about how each variable or score is calculated and what the scores represent. Give an example item from each scale/ subscale.

The final portion of this section is your data analysis plan. Mention what software you will use, how you will clean the data (how you will handle missing data, and make sure there are not any extreme outliers). Restate your research questions and hypotheses from c 1. Then, for each hypothesis describe the statistical tests that will be used. If you are doing many statistical tests, you need to account for that by using a correction statistic (Bonferroni's is common- it reduces your p value, based on the number of tests). If you are using covariates and/or have confounding variables, you need to discuss it. Finally, how will you interpret the results (key parameter estimates, confidence intervals and/or probability values, odds ratios, etc.). 

Next time we will continue our review - Chapter 3: Instrumentation in mixed methods studies. Do you have an issue or a question that you would like me to discuss in a future post? Would you like to be a guest writer? Send me your ideas! leann.stadtlander@waldenu.edu

Chapter 3: Instrumentation in qualitative

In qualitative studies, if you are talking to people and not using archival data, you will design most of your instruments.  You need to identify each data collection instrument and provide the source of it, if you did not design it (some examples: observation sheet, interview protocol, focus group protocol) There are also archival data sets which would need to be identified and the source (e.g., video-tape, audio-tape, artifacts, archived data).

If you are using historical or legal documents are used as a source of data (unusual to use), demonstrate the reputability of the sources and justify why they represent the best source of data. Then you want to clearly demonstrate the link between the data collection instruments and your research questions.

For published data collection instruments.

Explain who developed the instrument and provide the date of publication. Detail where and with which participant group it been used previously. You then need to justify its use in the current study (that is, context and cultural specificity of protocols/instrumentation) and whether modifications will be or were needed.

Describe how content validity will be or was established (how do you know it is looking at what you think it is?) A common way to do this is to use an expert panel. Discuss any context- and culture-specific issues specific to the population while developing the instrument. An example might be that if you are using an interview protocol that was designed for adults, and you want to use it with adolescents, you would need to change some of the language.

For researcher-developed instruments

What did you use as the basis for instrument development (some examples might be from the literature or from doing a pilot study)? Again, you need to describe how content validity will be or was established (how do you know it is looking at what you think it is?). Finally, you want to describe how your instruments will answer the research questions.

Next time we will continue our review - Chapter 3: Instrumentation in quantitative studies. Do you have an issue or a question that you would like me to discuss in a future post? Would you like to be a guest writer? Send me your ideas! leann.stadtlander@waldenu.edu