Chapter 4: Data Analyses: Qualitative and Mixed Methods

Today we will examine the section of c.4 in qualitative studies and mixed methods: Data Analyses. This section asks that you clearly describe how you went about analyzing your qualitative data. To begin, you will outline the overall process that you used to move inductively from coded units to larger representations including categories and themes. If you followed a specific methodologist's methods (e.g., Creswell) cite him or her.

Next, you move into the specifics of your data by describing the specific codes, categories, and themes that emerged from the data using quotations as needed to emphasize their importance. Keep in mind that you are walking the reader through the process of your data analysis, so share specifics- how did you make decisions as to what were themes?

Finally, in most studies you will have a person or two who discussed experiences or ideas that were outside the normal experience of the others in your sample, these are called discrepant cases. Describe how these discrepant cases differed from the rest of the sample and how they were factored into the analysis. You should also consider whether there are obvious reasons for their differences, are these individuals older, younger, or in some other way different from others in the sample?

Next time we will talk about Chapter 4: Trustworthines. Do you have an issue or a question that you would like me to discuss in a future post? Would you like to be a guest writer? Send me your ideas! leann.stadtlander@waldenu.edu 

c.4: Data Analyses: Quantitative

Today we will take a look at the results section for quantitative studies. Start out by reporting descriptive statistics "that appropriately characterize the sample." What does this mean? Look at frequencies for your demographics, such as gender, marital status, etc. For continuous variables (not in categories) you will need to compute the means and standard deviations or standard errors (check with your committee as to which they prefer). An example of such a variable is age, the convention is to give these stats like this (M = 43 yr., SD = 5.2). You will also discuss any total scores or subscores that you may have calculated and their distribution.

The next step is to discuss and evaluate statistical assumptions as appropriate to the study. All statistical tests have specific assumptions that must be considered (see Pallant, 2013, for an in-depth discussion of them).  Let's take as an example, the assumptions for parametric tests (e.g., t-tests, analysis of variance): using an interval or ratio scale of measurement, random sampling, independence of observations (no measurement is influenced by another), a normal distribution, and homogeneity of variance (samples have similar variances). There are techniques to check these assumptions, and you would discuss in this section which ones you used and the results.

Next, you report your findings, organized by research questions and/or hypotheses. Include the exact statistics and associated probability values (some examples: t(32)=3.1, p < .01; r(N=45)= .16, p > .05). A reminder- if the probability is < .05 (less than), it is considered significant; if it is > than.05 (greater than) it is not significant. You should include confidence intervals around the statistics, as appropriate (check with your committee). Include effect sizes, as appropriate (e.g., R²,^; check with your committee as to what they prefer).

If you had multiple conditions, you may need to do post-hoc tests. Report the type and results of post-hoc analyses.  You may have additional statistical tests of hypotheses that emerged from the analysis of main hypotheses and you will need to report those. Finally, you may wish to clarify your results with tables and figures, include those as specified in APA manual. There is very specific formatting for these- so check it out in the manual.

Next time we will talk about c.4: Data Analyses: Qualitative. Do you have an issue or a question that you would like me to discuss in a future post? Would you like to be a guest writer? Send me your ideas! leann.stadtlander@waldenu.edu

Pallant, J. (2013). The SPSS Survival Manual, 5^th edition. Open University Press.

c.4: Demographics and Data Collection

The mixed methods and qualitative checklists have section called Demographics, in which you should discuss the relevant demographic characteristics of your participants. Typical items include gender, race, and age, as well as any characteristics specific to your study. For example, if you interviewed homeless teen mothers, it would be important to know how long they have been on their own and the age of their children.

For all methods, the next section is Data Collection. You need to describe when the study was done (for example, months and year). Describe how you recruited your participants, and how many participated in all phases of the study. If you had to change any of your data collection procedures from what was listed in c. 3, indicate how and why it was changed (and that you went through IRB to do so).

For Qualitative and Mixed Methods Studies. Describe the location of your study, how often you met with participants and the length of time both for individual interviews/surveys and for the total study. Next, describe how you recorded your interviews and how they were transcribed. If you encountered any unusual circumstances during your data collection describe it and how it affected your data collection (e.g., equipment failure, a participant died between interviews, etc.).

For Quantitative Studies. Describe your demographics as discussed above. Describe how representative your sample is to the population of interest or how proportional it is to the larger population if non-probability sampling is used (external validity). Provide results of basic univariate analyses that justify inclusion of covariates in your model, if applicable.  

For this section, keep in mind that your reader should have a good picture of how you did your study, and would be able to replicate it based upon your description.

Next time we will talk about c.4: Data Analyses: Quantitative. Do you have an issue or a question that you would like me to discuss in a future post? Would you like to be a guest writer? Send me your ideas! leann.stadtlander@waldenu.edu

Chapter 3: Ethical Procedures

For the ethical procedures section, you will begin with stating what agreements you have received to get access to your participants or your data. Some examples might be if you are interviewing people through an agency, you will need an agreement with the agency (See the IRB site for samples of such agreements). Include a copy of the agreement in your appendix.

Next indicate your approval number and date from the Walden IRB, if you need to get approvals from other IRBS, list them here. Discuss any ethical concerns about recruitment materials and a plan to address them. Some examples might include how you will recruit (such as with a flyer), your consent form, and if you are using children and assent form for them. Then you will need to discuss any ethical concerns related to your data collection, such as people refusing participation, stopping midway, and having any possible adverse reactions to the study. Some things to remember that will help with this section, participants have the right to stop whenever they want. You do however, have to decide what you will do with their data- will you include it or exclude it? Anytime your participants might have issues arise from your study questions you need to find a way to help them. You are not allowed to counsel them, but you could provide phone numbers/ info on low cost counseling or hotlines.

If there is any possibility of participants revealing any personal medical, educational information or illegal activity (e.g., child or elderly abuse, drug use), you must have a plan as to how you will handle it.

Next, you need to describe how you will protect your data. State if the data is considered confidential (you know who provided it) or anonymous (you do not know who provided it). If data are confidential, and it is possible to determine who provided it, extra protections are required (an example is if there are 2 women who work at a given agency with 30 men, you will want to disguise them so it isn’t possible for the reader to identify them). You will want to maintain the data on password protected flash drives and limit access to it. State that you will destroy all data after 7 years (what APA suggests).

Finally, describe any special ethical considerations for your study, such as doing a study at your workplace, conflict of interest issues, and if you are using incentives, such as gift cards. Incentives are not recommended in general. See the IRB web site on these issues; they have additional information listed there.

We have now finished reviewing the three chapters that make up the proposal. Next time we will begin examining Chapter 4: Results. Do you have an issue or a question that you would like me to discuss in a future post? Would you like to be a guest writer? Send me your ideas! leann.stadtlander@waldenu.edu

Chapter 3: Issues of Trustworthiness in qualitative and mixed method studies

Rather than issues of validity that we saw in quantitative studies, qualitative (and mixed methods) have trustworthiness issues. The first issue is credibility, which is comparable to internal validity. This is getting at the credibility of your data, common methods used are triangulation, prolonged contact, member checks, and saturation. You want to show that your data are as accurate as possible.

The second issue is transferability, which is comparable to external validity. This is getting at the generabilizability of your data to other groups. Common methods used are thick description and a variation in participant selection.

The third issue is dependability, comparable to reliability. You want to show the accuracy of your data methods, common methods are audit trails and triangulation. Triangulation is accomplished by asking the same research questions of different study participants and by collecting data from different sources and by using different methods to answer those research questions. Member checks occur when the researcher asks participants to review both the data collected by the interviewer and the researchers' interpretation of that interview data. Participants are generally appreciative of the member check process, and knowing that they will have a chance to verify their statements tends to cause study participants to fill in any gaps from earlier interviews.

The fourth issue is confirmability, comparable to objectivity. This is the degree to which the findings are the product of the focus of the study and not of the biases of the researcher One way to do this is through an audit trail. An adequate trail (or records) should be left to enable the auditor to determine if the conclusions, interpretations, and recommendations can be traced to their sources and if they are supported by the inquiry.

If you are using another coder(s), you must show how you will demonstrate intercoder reliability. Interrater or intercoder reliability is used to reduce bias by having multiple people code the data. How you go about this and how you resolve any discrepancies needs to be detailed. 

Next time we will continue our review - Chapter 3: Ethical Procedures. Do you have an issue or a question that you would like me to discuss in a future post? Would you like to be a guest writer? Send me your ideas! leann.stadtlander@waldenu.edu

Chapter 3: Threats to Validity- construct and statistical validity

The final section under Threats to Validity for quantitative and mixed method studies asks you to describe any threats to construct or statistical conclusion validity. Let's start with a definition of construct. A construct is an attribute, proficiency, ability, or skill that happens in the human brain and is defined by established theories. For example, "resilience" is a construct. It exists in theory and has been observed to exist in practice.

Construct validity has traditionally been defined as the experimental demonstration that a test is measuring the construct it claims to be measuring. Such an experiment could take the form of a differential-groups study, wherein the performances on the test are compared for two groups: one that has the construct and one that does not have the construct. If the group with the construct performs better than the group without the construct, that result is said to provide evidence of the construct validity of the test. An alternative strategy is called an intervention study, wherein a group that is weak in the construct is measured using the test, then taught the construct, and measured again. If a non-trivial difference is found between the pretest and posttest, that difference can be said to support the construct validity of the test.

There are a large number of threats to construct validity- too many to discuss here, but I do suggest that you take a look at http://www.socialresearchmethods.net/kb/consthre.php when you are ready to write this section. The author does a very nice job laying out the many types of possible threats.

The last issue is statistical conclusion validity. Statistical conclusion validity is the degree to which our conclusions about the relationship between your variables based on the data are correct or ‘reasonable’. This is getting at the two types of statistical errors that can occur: type I (finding a difference or correlation when none exists) and type II (finding no difference when one exists). Statistical conclusion validity concerns the qualities of the study that make these types of errors more likely. Statistical conclusion validity involves ensuring the use of adequate sampling procedures, appropriate statistical tests, and reliable measurement procedures. If you would like a more in depth discussion of this topic, please see http://www.socialresearchmethods.net/kb/concthre.php

Next time we will continue our review - Chapter 3: Issues of Trustworthiness in qualitative and mixed method studies. Do you have an issue or a question that you would like me to discuss in a future post? Would you like to be a guest writer? Send me your ideas! leann.stadtlander@waldenu.edu

Chapter 3: Threats to Validity- internal validity quant and mixed methods

Previously we looked at external validity in quantitative and mixed methods studies, today we look at internal validity. Internal validity refers sto whether an experimental treatment/condition makes a difference or not, and whether there is sufficient evidence to support the claim. Some threats to internal validity include:

•History--the specific events which occur between the first and second measurement. People are affected by elements outside of the study. Let's use an extreme example of this, say you were interested in fear of flying and gave people a survey examining this variable on Sept 9, 2001. The participants then went through a de-sensitization training for a week and came back on Sept 16, 2001 and were retested. They are also going to be affected by an historical event outside of the study- the traumatic events of 9/11, and you would need to account for this.

•Maturation--the processes within subjects, which act as a function of the passage of time. i.e. if the project lasts a few years, most participants may improve their performance regardless of treatment. As people age they change, so if you were doing a study that lasted any period of time, you need to realize that they will change without your intervention. This is often why a control group is used, so the normal changes that occur can be compared with those of the treatment.

•Testing--the effects of taking a test on the outcomes of taking a second test. Simply taking a test can change how people think, they also cannot forget what they read in the first test. So if you give a second test people will have thought about their first answers and may change them in the second test because of that thinking process.

•Instrumentation--the changes in the instrument, observers, or scorers, which may produce changes in outcomes. Many things can affect the results, minor changes in wording, having additional people in the testing area, having different people score the test all may change the results.

•Statistical regression--It is also known as regression to the mean. This threat is caused by the selection of subjects on the basis of extreme scores or characteristics. Give me forty worst students and I guarantee that they will show immediate improvement right after my treatment, not because of my great treatment, but because they expect to do better.

•Selection of subjects--the biases which may result in selection of comparison groups. Randomization (Random assignment) of group membership is a counter-attack against this threat. However, keep in mind that randomization is only effective with large samples.

•Experimental mortality--the loss of subjects. For example, if you require people to participate in multiple training sessions, some will drop out. Those who stay in the project all the way to end may be more motivated to learn and thus achieved higher performance.

•Selection-maturation interaction--the selection of comparison groups and maturation interacting which may lead to confounding outcomes, and erroneous interpretation that the treatment caused the effect. A great example is if you had girls in a class assigned to one treatment and the boys assigned to another treatment. You compare them and discover there is a treatment difference. However, you do not know if it is the treatment that caused the differences or was it any differences in development between the girls and the boys.

In this section of your paper, you need to think through the various internal validity issues and how you will address them. 

Next time we will examine threats to validity- construct and statistical validity. Do you have an issue or a question that you would like me to discuss in a future post? Would you like to be a guest writer? Send me your ideas!