Data Collection Problems and Resolutions

Things happen in the real world when you are conducting a study, as the researcher it is your responsibility to both recognize the issue and resolve it. I am going to talk today about some of the most common issues that may arise. 

Confidentiality breach. Imagine that you are doing a program evaluation. You will survey participants before the program begins and again after the program. In order to match each person’s pre and post program survey, you will have them write their names on each form. Somehow during the data collection a few surveys get left behind. This is a breach of confidentiality. Someone else could see the responses and know who wrote them. In a program setting, that may not seem a serious offense, but consider if it was one’s supervisor that saw a derogatory comment about themselves. 

What to do? First, as soon as the loss is realized, get the surveys back in your possession. Second, report the breach to your committee and the IRB. They may want you to notify the individuals involved, but let them make that decision. You also may want to consider not using names in studies, instead ask people to pick a number or phrase that they will enter on the pre survey and remember for the post survey. 

Questionable data. When you are collecting in-person data, you may have someone act strangely, making you suspect drugs or alcohol use. Another scenario is that someone hands in a “completed” survey in a few minutes, as compared to the 20 minutes others took.  

What to do? First, make sure you are safe in the case of strange behavior (I once had a participant break the chair in which he was sitting from rocking and bouncing the chair due to amphetamine use). If there are any concerns, thank them for coming and get them out the door quietly. Second, mark their data forms, with the behavior that concerns you. Later when you analyze data, check whether their data deviates from the norm. Discuss the issue with your methodologist and make a joint decision about whether to include the data in the analyses. 

Participant Withdrawal. Occasionally you may have participants indicate that they want to stop and do not want to continue in your study. This is most common with special populations, such as the elderly and ill.  

What to do? First, I suggest asking if a break might help (and if you can recognize the need for one early, all the better). Second, as frustrating as it may be, you should just thank them for coming and let them go. 

Incomplete data. Thus far I have been addressing in-person research concerns, but one that is very common with online data collection is participants skipping questions. Some standardized surveys are invalid if questions are skipped.  

What to do? In person, you can do a quick check to see if any data is missing. Online is trickier. Most survey programs (like survey monkey) allow the option for requiring answers to the questions. There are some positives and negatives to consider with this. The positive is that your data will be complete, with no missing responses. The negative is some people will quit the survey if they can’t skip responses.  

Inappropriate disclosure. In interviews, you may have someone tell you things that are inappropriate, such scenarios include child or elder abuse, and disclosing medical or education information. 

What to do? As soon as you realize where they are going, stop them. Say that this is information you should not be told and move on with the study. The exception, is if you are a mandated reporter (check the laws in your state, some states say that everyone is a mandated reporter for child or elder abuse), in this case you are ethically required to report the incident to the appropriate authorities. I suggest first consulting with your committee and the IRB. Your status as a mandated reporter must be disclosed in the consent form. Such disclosures are unusual, but you need to be prepared in case they do happen. 

Whenever you run into something unexpected, know that you can contact your methodologist and/ or the IRB for help. Next time, we will consider issues in quantitative data analyses.

 

Collecting Data

Finally you have all the approvals and get to collect your data! A few pointers for this phase. Make sure you follow the procedures that were listed in your IRB application exactly, this is also called your research protocol. The IRB does random audits of students’ data, so be sure you are scrupulous about maintaining your protocol. Any changes (even minor ones) must be approved by the IRB through the “Change of Procedure” form. If you make any major changes, also be sure you notify your committee member and URR, and get their okay on it. 

One detail that is rarely mentioned, is the need for organization of your data. Let’s consider a couple of different scenarios. If you are using any type of paper collection methods (surveys, questionnaires, etc.) keep everything in one place, such as a file folder or large envelope. Don’t throw away or destroy anything until your total project is complete (even if you enter the data into software), you may need to double check your originals and if you are audited by the IRB, you will need the originals. 

If you are downloading your data directly from an outside source, e.g., survey monkey, secondary data, be sure that you double check that the data makes sense. Sometimes things get messed up in the transfer. Save your data in a couple of places, just in case of computer issues. 

Always double check your data entry, it is worth having someone else check you. Your results are only as accurate as your data entry. 

One question that students doing qualitative often have is whether they can deviate from their IRB approved interview questions. You are allowed to ask follow up questions to clarify the participants’ responses, but you may not go into new topics. If you believe that you need an additional interview question(s), you must complete a “change of procedure” form with the IRB, you will also need to show how the new interview questions relate to your research questions. 

If you are interacting directly with your participants through interviews or other methods, keep in the back of your mind that these people are doing you a great favor by being in your study. You could not get your degree without them. So be accommodating to their schedule, be polite, and thank them for participating. You have an obligation to all future researchers to show your participants that involvement in research is important and enjoyable. 

As you wait for the data to come in, use this time to go back through your proposal and change everything related to your study from future tense (“the study will explore…”) to past tense (“the study explored…”). Do not rely on a search and replace, I guarantee you will have to later go back and fix errors. Instead, read through the paper and correct them. You can also save yourself time in the future by double checking the Dissertation Guidebook as to form and style issues. Make sure you have followed the rules of grammar and writing as closely as possible. 

Sometimes, things don’t go as planned during data collection. Next time we will explore data collection problems and their resolution.

IRB: Ethics Review

Once your proposal has been accepted, the next step is the Institutional Review Board review (ethics board). The first step is to go to the research center, find the IRB website, and download the IRB application. One of the first items listed is that you need to take a short ethics course online (it is free), I recommend the NIH one. When you are done with the course, save the certificate to your computer, you will need to submit it to the IRB. Double check the IRB website for any relevant information sheets about your population. 

What is the IRB looking for? They want to know that you have thought out every detail of your study. The most common error students make is not providing enough detail, or not being clear in the information provided. As you begin to complete the IRB application, think about each item and consider: will someone unfamiliar with your study understand what you have written? Keep in mind that the board typically does not read your proposal, they rely on the information you provide in your application. 

The IRB is most concerned with your interactions with your participants, so go into great detail in this area. Think through each aspect of your study, some examples: Who will be approaching possible participants? What will they say? What happens to participants at each step of the process? Go into excruciating detail. If you find yourself assuming things (e.g., “I’ll use a room at XX for the interview”) stop and rethink it- do you have permission to use the room? Will it be private? Can people easily get there and find it? Will using the building give the impression that the organization is sponsoring the study? An example, of the last question would be using a church, you would want to be clear that the church is not affiliated in any way with your study. 

Let’s work through a fake study and think through some of the issues in the research methods. Let’s say we want to look at car shopping decisions and the relationship with self-efficacy (self-confidence). The plan is to do a mixed method study, we’ll use a standardized, brief self-efficacy survey and briefly interview the customer (What was your decision at the dealership regarding buying a car? What led you to make the decision that you did?). We decide that we want to use a local Nissan dealership, so first we approach the general manager, John Smith, and see if there is some way to make this work. John is open to the dealership participating but does not want us to interfere with customers in their decision making process. We initially discuss having everyone that speaks with a salesperson complete the 5 question survey. A brief consent form will be included and ask whether a researcher can contact them by phone and asks for their name and phone number. We will then call each customer that agrees and ask our questions. 

What are issues we need to consider? First we need a letter from John Smith stating everything that his people will and won’t do, and any information that the dealership will provide to us (so let’s wait on getting the letter until we have finalized the details). Second, we want to make it very clear that the researchers are not affiliated in any way with the dealership, that the person is not required to participate, and no one will treat them differently regardless of their choice in participating. One issue the IRB is going to have is with the salesperson giving out the survey, which gives the appearance of the dealership sponsoring the study. Is there a way around this? Perhaps at the end of the salesperson interaction he or she could give them a flyer with a link to an online survey (and the consent form) on survey monkey? As a researcher we need to consider that this is going to lower our participant rate, but will make it more acceptable to the IRB. We will probably have to continue the study longer and will need to mention this issue in the limitation section of the paper. 

Let’s consider our study’s inclusion criteria. People may participate if they are 18 years and older and if they are able to read English, which is the language of the survey. We should ask their age at the beginning of the survey. The exclusion criteria are being under 18 and unable to read English.

What other demographics might be a factor in purchasing a car? Perhaps marital status, having children, sharing decision making, employment, income per year? We would want to check the literature, make a decision, and add these into the survey. How will we determine who is interviewed? We will need to preset a rule for this. It could be the first 10 people who agree to be interviewed or if we think marital status is important we could say the first 5 with a partner/ spouse and the first 5 single/ divorced people who volunteer will be selected. The important thing is to select the criteria in advance and stick to it. 

Here is our current method:

1.      Get permission letter from John Smith

2.      Prepare flyer for salespeople with a link in survey monkey

3.      Prepare survey monkey with a consent form, the survey, and ask if person can be contacted for an interview, and have them give their email or phone number.

4.      Have a brief meeting with salespeople explaining the study and that we would like them to give a flyer to every customer, whether they buy a car or not.

5.      Wait to see if we get participants

6.      Contact those willing to be interviewed

7.      Call for interviews, record interviews

8.      Transcribe interviews

9.      Analyze qualitative data, developing coding

10.   Download survey data from survey monkey

11.   Analyze survey results compared to coding

12.   Send results to participants and dealership 

After a month, we only have three participants! We need to rethink our method, some possible options: we could go to the dealership and hand out the flyer personally as the person is leaving the dealership, sometimes the personal contact will help. We could increase the number of dealerships, each will require that we go through the first 4 steps listed above. Whatever, our final decision we will need to complete a “Change of Procedure” form for the IRB (available on their web site).  

A few final thoughts on the IRB process, you may need to go through several revisions of your methods. Work through each of the IRB’s comments and think them through. If you can’t figure out a solution, ask for a conference call with the IRB and your chair and they will brainstorm with you. Again, always be professional in your interactions, they are not trying to delay your progress. They just want your study to be the best one possible that protects you and your participants.  

Next time we will take a look at data collection. 

Moving On: The URR and proposal defense

Once your committee has approved your proposal (Chapters 1-3), each of your committee members completes a rubric. Your chair will submit your proposal to turnitin and assuming it is acceptable, the proposal, rubrics, and turnitin report will all be sent to your University Research Reviewer (URR). The URR is a faculty member that is assigned by the research center, this person provides an independent review of the paper and double checks that all is well. Why this step? The problem is that your committee is primarily looking at specific aspects of your paper, the content expert is concentrating on the literature. The methods expert is concentrating on the research methods. Unfortunately, they are focused on the trees, and need someone else to see the forest. Does everything hang together? Are the individual parts consistent? You will probably need to make some additional revisions for the URR. 

Once the URR approves your paper, he or she will complete a rubric and submit it to the research center. You will then receive an email indicating that you may schedule your one-hour proposal defense conference call. You will need to select several possible days and times that will work for you and send those to your committee with your time zone (check with your chair first, he or she may like to follow a different procedure). Once there are a couple of times that work for everyone, your chair will submit a conference call request form to the research center. You will be notified of the time selected and how to call in. 

In advance of the call you will want to prepare a power point for your committee. You will be given 20 minutes to talk about your project, so keep the slides to less than 25. It is better to have just an outline on the slides, you can always write out everything you want to say and read it during your presentation. Concentrate on the methods and analyses with just enough literature to put it into context. Check with your chair how they like it done. Send your committee the power point prior to the defense call. 

On the day of the defense, have your computer on with your paper open for reference. Have any other papers available that you might need, as well as your power point. It is also a good idea to have some water available and make sure you have an environment where you won’t be distracted or disturbed. Once you call in, your committee will greet you and try to put you at ease. Your chair will leave the conference briefly to turn on the recorder, there will be announcement when it starts. Your chair will probably indicate the date and who is on the call for the recorder and then it will be your turn to go through the presentation. 

A few things to know, first this is not a confrontational situation. I know it is called a defense, but it is really just a chance for the committee to hear about the project one more time and catch anything that may have been missed. It is typically very collegial, with everyone trying to make it the best project possible. After your 20 minute talk, your committee will ask you questions. They are just trying to clarify any issues that might arise during the data collection. It is okay to say that you don’t know something (but say that you will check it out, find the answer, and report back). 

Take a few moments at this stage of the process to reflect on your progress, celebrate! 

After the call, your chair will submit a form to the research center indicating that you passed the defense. You will then receive a notice that your next step is the IRB (Institutional Review Board) this is the ethics review and takes place before you collect any data. We’ll take a look at the IRB process next time.

Chapter 1: Beginning and end

A request. If you follow the blog by email, could you send me a quick email leann.stadtlander@waldenu.edu or text 406-451-6412 and let me know? I am trying to determine how many people are following it. Thanks!
 

Chapter 1 is the official beginning of your proposal and usually the last chapter written, it is a very important chapter in that it sets up the reader to understand an overview of the study and to appreciate the need for it. Let’s start with an outline from the checklist: Introduction, Background, Problem Statement, Purpose, Research Questions and Hypotheses, Theoretical and Conceptual Framework, Nature of the Study, Definitions, Assumptions, Scope and Delimitations, Limitations, Significance, Summary. I am sure that the first thing you notice is that there are many sections, each of these will be quite short- think of them as concise summaries.  

Again there are many resources to help with each of the sections (see the writing center and research center). I am just going to talk about the area that students have the most problems with, the introduction. The introduction needs to build a concise case as to the need for the study. Start general (think demographics) becoming more narrow. Clearly point out the gap in the literature and how your study addresses this gap. Also, mention the social change implications of the study. By the end of the introduction, the reader should be convinced that your study is the next logical step in the progression of the topic. They should know what type of study you are doing (e.g., survey, interviews) and how this method adds to filling the gap in the literature. 

You have a draft of the first 3 chapters, then what? First give yourself a cheer and celebrate, then begin the rewriting process. Rewriting?? Yes, you have done a first draft, hopefully including all of the necessary parts, but it is rough, and needs a lot of polishing. I suggest reading through the draft in full, make notes to yourself (I use track changes) and mark areas that are not complete, that may not be clear to someone reading it for the first time, or that needs more support with citations. Then start at the beginning and read each sentence aloud, is there a way to make it clearer, more concise? Picture your grandmother who knows nothing about psychology reading it, would she understand that sentence? Check for any pronouns (they, he, and she), is it clear who the pronouns are referring to? Check your plurals versus possessives (this makes me crazy when they are wrong): plurals (e.g. “girls”) do not have an apostrophe, possessives do have an apostrophe (e.g., “the girl’s bike;” “the girls’ bikes”).  

Do you know a former English major? Someone who is a great writer? If so, ask them to read through your paper and offer suggestions.  

Let’s consider for a minute, accepting criticism. No one likes to be criticized, but it is part of life and particularly part of academia. Instead of taking it personally, remember you are a student, isn’t part of that learning new things? Don’t you want the best paper and project that it is possible to have? The only way it will happen is to listen to criticism. Let’s assume for a moment that you have just received an edited draft from your committee member, how to you handle this? I suggest reading through their comments then setting it aside for a few hours- to a day. Think about them, instead of reacting. When you are calm and ready, start at the beginning and take each comment as a learning experience. Fix it, then look for similar issues in the paper and fix those. Go on to the 2^nd comment, etc. I look at comments and suggestions as a challenge, can I fix these and have the person read the paper again and not find any issues? When you are done with your revisions read through the paper one more time (at least) and make sure it reads well.

Next time we’ll take a look at the URR step and proposal defense.

The Care & Feeding of Committee Members

 

What is appropriate behavior for a dissertation student toward his or her chair and committee member? In a traditional brick and mortar program you would see the behavior modeled by students further along in your program, but it is difficult at an online school to know what is normal.

The reality is it varies tremendously, so the first bit of advice I have for you is to ask what your chair and committee member prefer. If your chair does not suggest it, ask for a phone call to get you started, and discuss how your chair likes to structure their class and what their expectations are of students. What are they expecting from you in the first quarter? How will they decide if you should get a “U” or an “S” in the dissertation course? How would they prefer for you to address them? 

In advance, think about how you like to work… are you independent and good at personal goal setting? Then perhaps ask if you could send them your draft once a month (or whatever the 2 of you agree is appropriate). Are you a person who has a hard time structuring your time and goal setting? Then ask for help as to what is realistic. You might want to set up regular meetings or times when you will need to submit drafts. Talk to your chair about your needs, and how they can be accommodated. 

Also ask your committee member when they would like to see a draft. Many will prefer that you wait until the proposal has been approved by your chair, unless you have specific methodological or content issues on which you want their advice.  

A few general bits of advice, be polite, ask questions when you need help, no one expects you to know everything. A “thank you” for good advice, prompt response, etc. means a lot to faculty (think of it as positive reinforcement, they will tend to repeat the good behavior).  

This is important: No one will know you are having problems, unless you say something! Typically, if they don’t hear from you, the faculty’s assumption is that you simply don’t care or aren’t working on the paper. There are many people who can help if you run into problems. If you have writing issues, contact the writing center. They have editors that will work with you. If one of your committee members advises you go to the writing center then listen and try not to be defensive. They are trying to save you time and tuition money! If you have committee issues, contact your program coordinator or program director. Be assertive, it is your money and time being spent, however, always be professional in interactions with faculty and administrators. Emotional responses are never a viable solution. 

Faculty are people too, with busy schedules and family obligations, it is not realistic to expect immediate feedback. However, if you have not heard from a faculty member at all within a couple of days of sending an email, write and double check that they received it. Ask if they can give you an estimate of when you might get a response. Again, be polite and professional.  

Remember, every faculty member who reads your paper is allowed 10 business days. I know it is hard to wait, but go on to the next stage. If you are waiting to get feedback on your prospectus, then go ahead and start on Chapter 2. If you are waiting on your proposal approval, work on your IRB application and power point for your defense. There is always something to work on. If you can’t find anything else, do an update in the literature and see if anything new has come out that could be helpful.

What do you do if you are having problems with your chair? First, write out an email outlining the issues, say how you would like it to be resolved- be very professional- no emotional response. Often it helps to take on some of the responsibility. It is a good idea to let it sit for a day to make sure that you are not being impulsive. As the health psych program coordinator, I am always happy to read such email drafts and give suggestions. Here is an example of a an appropriate email: 

Dr. X,

I am concerned that it is often taking 3 weeks for you to review my proposal drafts; the delay is really slowing my progress on the paper. Is there anything I can do to make this process go a little faster? If it would help, I could just send you a couple of pages at a time. Ideally, I would like to be able to get a response on my writing within a few days to a week. Do you have any further ideas? 

If you feel that you don’t get a satisfactory response, the next step would be to contact your program coordinator or program director and ask for help.  

One final bit of advice, keep a record of when you submit papers to faculty, when you get a response, and their comments. It is a record that may come in handy in the future (this is where a research journal comes in handy!). 

Next time, we will finish up the proposal with Chapter 1. 

C.3, Research Methods: The heart of the paper

Chapter 3 is the most important part of your proposal; it will detail what you are actually doing in the study. Let's start again by seeing what is in the template and checklist, note that each of the research methods has a different set of requirements.  

I suggest beginning your draft by writing down each of the requirements for your study: 

Qualitative

Introduction, Research Design and Rationale, Role of the Researcher, Methodology, Instrumentation, Data Analysis Plan, Issues of Trustworthiness, Ethical Procedures, Summary 

Quantitative

Introduction, Research Design and Rationale, Methodology, Instrumentation, Threats to Validity, Ethical Procedures, Summary 

Mixed Methods

Introduction, Research Design and Rationale, Role of the Researcher, Methodology, Instrumentation, Data Analysis Plan (for both quantitative and qualitative), Threats to Validity, Issues of Trustworthiness, Ethical Procedures, Summary 

There are many resources for these sections in the writing center and research center; I am going to concentrate on the ones that lead to the most issues for students: Methodology and Ethics. Naturally, every study is very different, but a few general comments can be made.  

Remember that your Methodology section should be written in enough detail that someone could replicate your study exactly based on your description. Some things to consider, how many people will participate? You need a rationale for the number, whether it is from a power calculation for quantitative studies (Google "power analysis calculator" for info) or convention for qualitative ones (cite your sources). 

How will you recruit participants? You want to use flyers? Where will you post them? You will need to include the flyer in the appendix. You want to recruit from an online support group? Which ones? What will you say about the study? Include that info in the appendix. You will need written permission from any groups that agree to help you, those letters will go in the appendix. 

How will the participants interact with you? An online survey? Where will it be posted? How will they access it? In person interview? Where will it be done? How long will it take? You will need to include any surveys and interview questions in the appendix. 

All participants will be required to sign a consent form, and anyone under 18 needs an assent form (parents sign the consent form). Any forms that the participant will see need to be in the appendix. 

There are many possible ethical issues; one of the most common scenarios is when you want to use people at your workplace. There are many concerns with this, read the information sheets on the IRB website. Similarly, if you would like to recruit vulnerable populations (some examples, children, elderly, ill people, prisoners, military), read the information sheets on the IRB website, and if you think your population fits these cases, send the IRB an email and talk to them (IRB@waldenu.edu). The IRB will help you work through issues. 

One final resource I want to mention, if you are having a difficult time working out issues with your methods or your analyses sections, Walden has methodological and statistical consultants. Here is the tricky part- students are not allowed to contact them directly, however, your chair or methodologist can contact them on your behalf. They are very knowledgeable and can help resolve those tricky issues. 

Next time we will take a look at the care & feeding of committee members.