Chapter 4 Analyses: Mixed Method

By definition, a mixed method study incorporates both quantitative and qualitative methods. There are several ways to analyze the data; I am going to discuss two. In the first method, the quantitative data is analyzed and the qualitative simply adds to an understanding of that data through quotes.  

In the second method, the data are analyzed together. Typically, the quantitative data is analyzed first, using more sophisticated analyses, such as multiple regression. Then the qualitative data is transformed into quantitative data (see Creswell and Plano Clark, 2011). To do this you will need to develop a scoring rubric for the qualitative themes and codes that you want to incorporate. Let's use the qualitative data from last time (see May 20 post), in which we had 3 sample responses. I developed coding as follows: 

101- grandmother, death, visual imagery, with mother, crying (negative emotion)

102- grandmother, visual imagery, auditory, empathy (recognizing other's emotions)

103- mom, eating, auditory imagery, feelings of comfort 

I need to decide what is important to include in my data analysis. I want to include any person mentioned; if a female relative, it will be designated as 1, a male relative will be 2, nonrelative female 3, etc. I am also interested in the use of imagery; if they mention visual images it will be 1, no visual will be 0. Auditory imagery will be 1, no auditory will be 0. Emotions in the self will be 1, and emotions mentioned in others will be 2. I would then enter this into SPSS along with the participants' scores on the quantitative measures that were used. The resulting SPSS entry would look something like this: 

Subnum	Survey1Tot	Survey2Tot	Person	Vismage	AudImage	Emotions
101	56	24	1	1	0	1
102	32	23	1	1	1	2
103	86	26	1	0	0	1

 

Then I will analyze the data using probably correlations and chi square to see if the scores on the surveys are related at all to the memories that the participants reported. Presumably, I would have some theoretical rationale for doing this. 

If you would like to see what a mixed method analysis looks like in a journal, please see:  

Stadtlander, L., Giles, M., Sickel, A., Brooks, E., Brown, C., Cormell, M., Ewing, L., Hart, D., Koons, D., Olson, C., Parker, P., Semenova, V., & Stoneking, S. (2013). Independent Living Oldest-Old and Their Primary Health Provider: A Mixed Method Examination of the Influence of Patient Personality Characteristics. Journal of Applied Gerontology. (Available in Walden Library's Psychology/ Sage database) 

Next time we will look at the final dissertation chapter – Chapter 5, Summary. 

 

Creswell, J. W. & Plano Clark, V.L. (2011) Designing and conducting mixed methods research, 3rd ed. Thousand Oaks, CA: Sage.

 

Chapter 4 Analyses: Qualitative

You have completed your interviews and maintained your field notes, the next step is to do the transcriptions. If you can do the transcriptions as you go, all the better! You should include every utterance of the interviewee, including umms, laughter, etc. Incorporate your field notes and observations into the transcription (e.g., subject head down, begins to cry). The transcription is the most time consuming aspect, in my opinion.  

You now have a ton of information from all of your interviews- what do you do with it? First, I suggest reading through all of your interviews several times. Think about them as you read them, what are the people saying? What are they NOT saying? Sometimes, what they avoid discussing is as interesting as what they do say. Try to put yourself in their place; you want to understand what their life is like. 

Once you have a good feel for the data, the next step is to begin the analysis. There are a number of software programs available, such as Nvivo and Qualrus, but I am going to recommend a low tech method that I generally use. In this method a matrix or table is used in word. Picture a table as shown below for each interview question. Each person’s response is copied into the table. You then carefully read each response and develop the coding for each one. As you do this, questions to ask yourself: What are the key points that are mentioned? What is the underlying emotions associated with the response? You will want to go through this process several times. The first time, use the participants' words in the codes. Then go through again and think about larger concepts that are present in different people's responses- can you find an overarching code?

 

Sub#	What is your earliest memory?	coding
101	I remember my grandmother being sick, she had cancer. She passed away. I remember crying. It was summer time. I remember seeing her in her bed, not moving, with her eyes closed. I was with my mom.
102	I remember my grandmother holding me in her arms with a polka dot cap. I remember her holding me in her arms, rocking me, trying to get me to go to sleep. The look in her eyes of happiness and content, of having me there as she rocked me to sleep with her polka dot night cap. I remember vaguely maybe the TV being on, maybe the news. I remember an air conditioner in the wall going.
103	I was sitting at a table eating a sandwich and my mom was singing a song to me. It was a song I enjoyed her singing to me – I enjoyed the song and it made me feel comforted. I was focused on my mom’s voice and hearing the story in the song.

 

What coding would you give for each of the examples above? Try it and then see if yours are similar to mine (at the end of this post). It is always a good idea to have someone else independently code the responses and compare yours to theirs. For any that you differ on, try to reach a compromise. You might see if anyone in your dissertation class is at similar place in his or her dissertation process and would be willing to trade coding checks. If not, check with your committee member- perhaps they know another student who might work. Note that if you go this route, the person should sign a confidentiality agreement which will need to be submitted to the IRB along with a change of procedure including them.

Once the coding is complete, the next step is to look for similarities across your interviews based up on the coding; these similarities are called themes. Were there any themes in the examples presented above (of course your data will have more people, 3 people do not make up a theme)? I have listed two possible ones at the end of this post. 

Once themes are determined, the next step is to begin writing up c. 4, data analysis. You will want to give your themes and provide examples of quotes illustrating them. Follow the qualitative checklist for how the chapter is to be arranged. 

Next time we will look at the mixed method analyses. 

My coding and themes

101- grandmother, death, visual imagery, with mother, crying (negative emotion)

102- grandmother, visual imagery, auditory, empathy (recognizing other's emotions)

103- mom, eating, auditory imagery, feelings of comfort

Themes: female relative, visual and auditory imagery

Chapter 4: Quanitative Analysis

A request. As we are nearing the completion of the first dissertation review in the blog, what else would you like me to address in future posts? Please send me an email with your ideas and questions! leann.stadtlander@waldenu.edu 

You have collected all of your quantitative data, entered it into SPSS (or downloaded it), and double checked your data entry. Great… now what?? It can feel intimidating to view your first real data set and know you need to figure out what to do with it. I like to start with getting an overall feel for what is going on by calculating means and frequencies. Let’s take a moment and review what to use when. 

If your variable is continuous, meaning there are no categories that you set up previous to the study you can calculate means and standard deviations. An example of a continuous variable would be where you asked people to enter their age today (e.g., 32, 56, etc.). Examples of categories would be gender: 1 = female, 2 = male or age: 1 = 20-30, 2 = 30-40, etc.). Hopefully, these terms are sounding familiar, if not go back to your stats book and review. A good reference for SPSS is

Pallant, J. (2010). SPSS survival guide, 4th ed. Berkshire England: McGraw Hill 

For categorical variables (e.g., gender, education level) you can do a frequency table and get a feel for how your data looks. Make sure you don’t have any data entry errors- they will show up as a weird number, e.g., you have gender coded as 1 & 2 and a 21 shows up in the frequency table. Go back to the original data and double check it.  

One issue you may need to consider is what to do with missing data, for example from people skipping questions. There are a number of ways to deal with this issue. Check with your methodologist to see what they prefer. They will probably ask you how many cells (individual data points) are missing, for which variables are they missing, and what is the largest number missing per individual, so be prepared for those questions. 

Once missing data issues are resolved, my usual next step is run correlations between my variables just to get a feel for what is going on. I then do scatterplots for any that show up as significant. Again, I am just trying to get a feel for the data. My old undergrad stats professor used to say that you need to “take a bath in your data.” I like that idea, you need to understand the relationships before getting immersed in the formal data analysis. 

You should have developed an analysis plan in your proposal, so now is the time to go to that. What happens if you just can’t figure things out? Contact your committee members and ask for help. As a committee member I sometimes have students send me their data set and I play with it a little, then I can talk them through issues.

Sometimes students decide to hire statistical consultants. Personally, I am not a fan of this. I prefer that the student figure out the stats with the help of his or her committee. The problem with a consultant is that you don’t really understand what they did and why. Even if they explain it, you really don’t have the level of understanding that you should. It misrepresents your knowledge level. People reading your dissertation will assume that you did the analyses and are capable of doing it (and perhaps teaching it!) again. If you must use a stats consultant, my advice is to rerun all of the analyses that they do, so you understand them too. 

Another aspect to consider is keeping track of all the analyses that you run and what they show. There are several ways to do this. You can simply save all of your SPSS outputs in a separate file on your computer (my least favorite, because then you have to reopen each to see it). Another way is to print out all of your data outputs and save them in a file or binder. My own favorite way to keep my data output is to copy it or rewrite it as I go into a word file. The advantage of this is that I am keeping everything together that is relevant (you will generate a lot of irrelevant info as you go). Do keep in mind that SPSS tables are not in APA format, so any that you want to use in your paper will need to rewritten.  

I also find it helpful to think through what I am finding with each analysis (even though this technically goes in c. 5, I find it helpful to think about it at the analysis stage). Let’s work through an example, I find that my variable education level is correlated with my dependent variable, emotional intelligence (EI) total score. So my first question is which education level has a higher EI score? I could do a scatterplot or could just calculate the means for each education level (use analyze/ descriptive stats/explore). I then find that people with a graduate degree have a higher emotional intelligence score than people with a high school diploma in the sample. Is that what previous research has found? What if this is not the relationship other researchers have reported? I need to consider why my sample may be different. I check what else is correlated with education- perhaps I find that for this sample, gender is highly related to emotional intelligence. Do another scatterplot between gender and education. Whoa- all of the graduate level participants are female. Could that be the cause of the education and emotional intelligence score correlation?  

Remember your methodologist can help with any issues you may have. There are also statistical consultants available for tricky issues, a faculty member can ask questions on your behalf. Next time, we will look at qualitative analyses.

Data Collection Problems and Resolutions

Things happen in the real world when you are conducting a study, as the researcher it is your responsibility to both recognize the issue and resolve it. I am going to talk today about some of the most common issues that may arise. 

Confidentiality breach. Imagine that you are doing a program evaluation. You will survey participants before the program begins and again after the program. In order to match each person’s pre and post program survey, you will have them write their names on each form. Somehow during the data collection a few surveys get left behind. This is a breach of confidentiality. Someone else could see the responses and know who wrote them. In a program setting, that may not seem a serious offense, but consider if it was one’s supervisor that saw a derogatory comment about themselves. 

What to do? First, as soon as the loss is realized, get the surveys back in your possession. Second, report the breach to your committee and the IRB. They may want you to notify the individuals involved, but let them make that decision. You also may want to consider not using names in studies, instead ask people to pick a number or phrase that they will enter on the pre survey and remember for the post survey. 

Questionable data. When you are collecting in-person data, you may have someone act strangely, making you suspect drugs or alcohol use. Another scenario is that someone hands in a “completed” survey in a few minutes, as compared to the 20 minutes others took.  

What to do? First, make sure you are safe in the case of strange behavior (I once had a participant break the chair in which he was sitting from rocking and bouncing the chair due to amphetamine use). If there are any concerns, thank them for coming and get them out the door quietly. Second, mark their data forms, with the behavior that concerns you. Later when you analyze data, check whether their data deviates from the norm. Discuss the issue with your methodologist and make a joint decision about whether to include the data in the analyses. 

Participant Withdrawal. Occasionally you may have participants indicate that they want to stop and do not want to continue in your study. This is most common with special populations, such as the elderly and ill.  

What to do? First, I suggest asking if a break might help (and if you can recognize the need for one early, all the better). Second, as frustrating as it may be, you should just thank them for coming and let them go. 

Incomplete data. Thus far I have been addressing in-person research concerns, but one that is very common with online data collection is participants skipping questions. Some standardized surveys are invalid if questions are skipped.  

What to do? In person, you can do a quick check to see if any data is missing. Online is trickier. Most survey programs (like survey monkey) allow the option for requiring answers to the questions. There are some positives and negatives to consider with this. The positive is that your data will be complete, with no missing responses. The negative is some people will quit the survey if they can’t skip responses.  

Inappropriate disclosure. In interviews, you may have someone tell you things that are inappropriate, such scenarios include child or elder abuse, and disclosing medical or education information. 

What to do? As soon as you realize where they are going, stop them. Say that this is information you should not be told and move on with the study. The exception, is if you are a mandated reporter (check the laws in your state, some states say that everyone is a mandated reporter for child or elder abuse), in this case you are ethically required to report the incident to the appropriate authorities. I suggest first consulting with your committee and the IRB. Your status as a mandated reporter must be disclosed in the consent form. Such disclosures are unusual, but you need to be prepared in case they do happen. 

Whenever you run into something unexpected, know that you can contact your methodologist and/ or the IRB for help. Next time, we will consider issues in quantitative data analyses.

 

Collecting Data

Finally you have all the approvals and get to collect your data! A few pointers for this phase. Make sure you follow the procedures that were listed in your IRB application exactly, this is also called your research protocol. The IRB does random audits of students’ data, so be sure you are scrupulous about maintaining your protocol. Any changes (even minor ones) must be approved by the IRB through the “Change of Procedure” form. If you make any major changes, also be sure you notify your committee member and URR, and get their okay on it. 

One detail that is rarely mentioned, is the need for organization of your data. Let’s consider a couple of different scenarios. If you are using any type of paper collection methods (surveys, questionnaires, etc.) keep everything in one place, such as a file folder or large envelope. Don’t throw away or destroy anything until your total project is complete (even if you enter the data into software), you may need to double check your originals and if you are audited by the IRB, you will need the originals. 

If you are downloading your data directly from an outside source, e.g., survey monkey, secondary data, be sure that you double check that the data makes sense. Sometimes things get messed up in the transfer. Save your data in a couple of places, just in case of computer issues. 

Always double check your data entry, it is worth having someone else check you. Your results are only as accurate as your data entry. 

One question that students doing qualitative often have is whether they can deviate from their IRB approved interview questions. You are allowed to ask follow up questions to clarify the participants’ responses, but you may not go into new topics. If you believe that you need an additional interview question(s), you must complete a “change of procedure” form with the IRB, you will also need to show how the new interview questions relate to your research questions. 

If you are interacting directly with your participants through interviews or other methods, keep in the back of your mind that these people are doing you a great favor by being in your study. You could not get your degree without them. So be accommodating to their schedule, be polite, and thank them for participating. You have an obligation to all future researchers to show your participants that involvement in research is important and enjoyable. 

As you wait for the data to come in, use this time to go back through your proposal and change everything related to your study from future tense (“the study will explore…”) to past tense (“the study explored…”). Do not rely on a search and replace, I guarantee you will have to later go back and fix errors. Instead, read through the paper and correct them. You can also save yourself time in the future by double checking the Dissertation Guidebook as to form and style issues. Make sure you have followed the rules of grammar and writing as closely as possible. 

Sometimes, things don’t go as planned during data collection. Next time we will explore data collection problems and their resolution.

IRB: Ethics Review

Once your proposal has been accepted, the next step is the Institutional Review Board review (ethics board). The first step is to go to the research center, find the IRB website, and download the IRB application. One of the first items listed is that you need to take a short ethics course online (it is free), I recommend the NIH one. When you are done with the course, save the certificate to your computer, you will need to submit it to the IRB. Double check the IRB website for any relevant information sheets about your population. 

What is the IRB looking for? They want to know that you have thought out every detail of your study. The most common error students make is not providing enough detail, or not being clear in the information provided. As you begin to complete the IRB application, think about each item and consider: will someone unfamiliar with your study understand what you have written? Keep in mind that the board typically does not read your proposal, they rely on the information you provide in your application. 

The IRB is most concerned with your interactions with your participants, so go into great detail in this area. Think through each aspect of your study, some examples: Who will be approaching possible participants? What will they say? What happens to participants at each step of the process? Go into excruciating detail. If you find yourself assuming things (e.g., “I’ll use a room at XX for the interview”) stop and rethink it- do you have permission to use the room? Will it be private? Can people easily get there and find it? Will using the building give the impression that the organization is sponsoring the study? An example, of the last question would be using a church, you would want to be clear that the church is not affiliated in any way with your study. 

Let’s work through a fake study and think through some of the issues in the research methods. Let’s say we want to look at car shopping decisions and the relationship with self-efficacy (self-confidence). The plan is to do a mixed method study, we’ll use a standardized, brief self-efficacy survey and briefly interview the customer (What was your decision at the dealership regarding buying a car? What led you to make the decision that you did?). We decide that we want to use a local Nissan dealership, so first we approach the general manager, John Smith, and see if there is some way to make this work. John is open to the dealership participating but does not want us to interfere with customers in their decision making process. We initially discuss having everyone that speaks with a salesperson complete the 5 question survey. A brief consent form will be included and ask whether a researcher can contact them by phone and asks for their name and phone number. We will then call each customer that agrees and ask our questions. 

What are issues we need to consider? First we need a letter from John Smith stating everything that his people will and won’t do, and any information that the dealership will provide to us (so let’s wait on getting the letter until we have finalized the details). Second, we want to make it very clear that the researchers are not affiliated in any way with the dealership, that the person is not required to participate, and no one will treat them differently regardless of their choice in participating. One issue the IRB is going to have is with the salesperson giving out the survey, which gives the appearance of the dealership sponsoring the study. Is there a way around this? Perhaps at the end of the salesperson interaction he or she could give them a flyer with a link to an online survey (and the consent form) on survey monkey? As a researcher we need to consider that this is going to lower our participant rate, but will make it more acceptable to the IRB. We will probably have to continue the study longer and will need to mention this issue in the limitation section of the paper. 

Let’s consider our study’s inclusion criteria. People may participate if they are 18 years and older and if they are able to read English, which is the language of the survey. We should ask their age at the beginning of the survey. The exclusion criteria are being under 18 and unable to read English.

What other demographics might be a factor in purchasing a car? Perhaps marital status, having children, sharing decision making, employment, income per year? We would want to check the literature, make a decision, and add these into the survey. How will we determine who is interviewed? We will need to preset a rule for this. It could be the first 10 people who agree to be interviewed or if we think marital status is important we could say the first 5 with a partner/ spouse and the first 5 single/ divorced people who volunteer will be selected. The important thing is to select the criteria in advance and stick to it. 

Here is our current method:

1.      Get permission letter from John Smith

2.      Prepare flyer for salespeople with a link in survey monkey

3.      Prepare survey monkey with a consent form, the survey, and ask if person can be contacted for an interview, and have them give their email or phone number.

4.      Have a brief meeting with salespeople explaining the study and that we would like them to give a flyer to every customer, whether they buy a car or not.

5.      Wait to see if we get participants

6.      Contact those willing to be interviewed

7.      Call for interviews, record interviews

8.      Transcribe interviews

9.      Analyze qualitative data, developing coding

10.   Download survey data from survey monkey

11.   Analyze survey results compared to coding

12.   Send results to participants and dealership 

After a month, we only have three participants! We need to rethink our method, some possible options: we could go to the dealership and hand out the flyer personally as the person is leaving the dealership, sometimes the personal contact will help. We could increase the number of dealerships, each will require that we go through the first 4 steps listed above. Whatever, our final decision we will need to complete a “Change of Procedure” form for the IRB (available on their web site).  

A few final thoughts on the IRB process, you may need to go through several revisions of your methods. Work through each of the IRB’s comments and think them through. If you can’t figure out a solution, ask for a conference call with the IRB and your chair and they will brainstorm with you. Again, always be professional in your interactions, they are not trying to delay your progress. They just want your study to be the best one possible that protects you and your participants.  

Next time we will take a look at data collection. 

Moving On: The URR and proposal defense

Once your committee has approved your proposal (Chapters 1-3), each of your committee members completes a rubric. Your chair will submit your proposal to turnitin and assuming it is acceptable, the proposal, rubrics, and turnitin report will all be sent to your University Research Reviewer (URR). The URR is a faculty member that is assigned by the research center, this person provides an independent review of the paper and double checks that all is well. Why this step? The problem is that your committee is primarily looking at specific aspects of your paper, the content expert is concentrating on the literature. The methods expert is concentrating on the research methods. Unfortunately, they are focused on the trees, and need someone else to see the forest. Does everything hang together? Are the individual parts consistent? You will probably need to make some additional revisions for the URR. 

Once the URR approves your paper, he or she will complete a rubric and submit it to the research center. You will then receive an email indicating that you may schedule your one-hour proposal defense conference call. You will need to select several possible days and times that will work for you and send those to your committee with your time zone (check with your chair first, he or she may like to follow a different procedure). Once there are a couple of times that work for everyone, your chair will submit a conference call request form to the research center. You will be notified of the time selected and how to call in. 

In advance of the call you will want to prepare a power point for your committee. You will be given 20 minutes to talk about your project, so keep the slides to less than 25. It is better to have just an outline on the slides, you can always write out everything you want to say and read it during your presentation. Concentrate on the methods and analyses with just enough literature to put it into context. Check with your chair how they like it done. Send your committee the power point prior to the defense call. 

On the day of the defense, have your computer on with your paper open for reference. Have any other papers available that you might need, as well as your power point. It is also a good idea to have some water available and make sure you have an environment where you won’t be distracted or disturbed. Once you call in, your committee will greet you and try to put you at ease. Your chair will leave the conference briefly to turn on the recorder, there will be announcement when it starts. Your chair will probably indicate the date and who is on the call for the recorder and then it will be your turn to go through the presentation. 

A few things to know, first this is not a confrontational situation. I know it is called a defense, but it is really just a chance for the committee to hear about the project one more time and catch anything that may have been missed. It is typically very collegial, with everyone trying to make it the best project possible. After your 20 minute talk, your committee will ask you questions. They are just trying to clarify any issues that might arise during the data collection. It is okay to say that you don’t know something (but say that you will check it out, find the answer, and report back). 

Take a few moments at this stage of the process to reflect on your progress, celebrate! 

After the call, your chair will submit a form to the research center indicating that you passed the defense. You will then receive a notice that your next step is the IRB (Institutional Review Board) this is the ethics review and takes place before you collect any data. We’ll take a look at the IRB process next time.