Chapter 3: Instrumentation in quantitative studies

Previously, we looked at instrumentation in qualitative studies, this time we move to quantitative studies. Generally, students will be relying on previously published instruments for quantitative studies. I do not recommend developing new surveys for a doctoral study. I will explain more on this below.

You will begin this section by discussing each instrument you will be using, who developed it and when. Then indicate why you have chosen it, and why it is appropriate for your study. You will need permission to use the instrument from the developer- include it in your appendix. Then go into detail about the published validity and reliability values that are relevant for your study (those using similar populations). Finally, you will discuss when and with what populations it has been used and how validity and reliability were established for each study.

If you are developing your own instrument, first describe the basis for its development. Did it come through items indicated in the literature? Did you or plan to do a pilot study to refine the questions? You will need to provide evidence for its reliability and validity, this typically requires extensive testing (100s of participants). Finally, show how the instrument will answer your research questions. If you are developing your own instrument, it will require quite a bit of testing and additional work; again, I do not recommend this for a dissertation.

If you are doing an intervention involving manipulation of an independent variable,

there are issues that will come up with the IRB, address them early! As far as c. 3, identify any materials that will be used in the intervention. Indicate who developed the materials and where they have been used in the past (you also may need permission from the developer to use them). If you developed them, indicate how that was done. Provide evidence that another agency will sponsor the intervention.

Next, you need to operational each variable. So for example, if you are interested in resilience, define it and how you will be measuring it. Then talk about how each variable or score is calculated and what the scores represent. Give an example item from each scale/ subscale.

The final portion of this section is your data analysis plan. Mention what software you will use, how you will clean the data (how you will handle missing data, and make sure there are not any extreme outliers). Restate your research questions and hypotheses from c 1. Then, for each hypothesis describe the statistical tests that will be used. If you are doing many statistical tests, you need to account for that by using a correction statistic (Bonferroni's is common- it reduces your p value, based on the number of tests). If you are using covariates and/or have confounding variables, you need to discuss it. Finally, how will you interpret the results (key parameter estimates, confidence intervals and/or probability values, odds ratios, etc.). 

Next time we will continue our review - Chapter 3: Instrumentation in mixed methods studies. Do you have an issue or a question that you would like me to discuss in a future post? Would you like to be a guest writer? Send me your ideas! leann.stadtlander@waldenu.edu

Chapter 3: Instrumentation in qualitative

In qualitative studies, if you are talking to people and not using archival data, you will design most of your instruments.  You need to identify each data collection instrument and provide the source of it, if you did not design it (some examples: observation sheet, interview protocol, focus group protocol) There are also archival data sets which would need to be identified and the source (e.g., video-tape, audio-tape, artifacts, archived data).

If you are using historical or legal documents are used as a source of data (unusual to use), demonstrate the reputability of the sources and justify why they represent the best source of data. Then you want to clearly demonstrate the link between the data collection instruments and your research questions.

For published data collection instruments.

Explain who developed the instrument and provide the date of publication. Detail where and with which participant group it been used previously. You then need to justify its use in the current study (that is, context and cultural specificity of protocols/instrumentation) and whether modifications will be or were needed.

Describe how content validity will be or was established (how do you know it is looking at what you think it is?) A common way to do this is to use an expert panel. Discuss any context- and culture-specific issues specific to the population while developing the instrument. An example might be that if you are using an interview protocol that was designed for adults, and you want to use it with adolescents, you would need to change some of the language.

For researcher-developed instruments

What did you use as the basis for instrument development (some examples might be from the literature or from doing a pilot study)? Again, you need to describe how content validity will be or was established (how do you know it is looking at what you think it is?). Finally, you want to describe how your instruments will answer the research questions.

Next time we will continue our review - Chapter 3: Instrumentation in quantitative studies. Do you have an issue or a question that you would like me to discuss in a future post? Would you like to be a guest writer? Send me your ideas! leann.stadtlander@waldenu.edu

Chapter 3: Participant Selection- qualitative and mixed methods

The section on participant selection for qualitative and mixed methods is similar to the quantitative, except you need to think of it in terms of the methodology. So again, identify your population – to whom will the results generalize? In qualitative/ mixed method studies the population is generally smaller than quantitative, so if you are interested in women who have been in domestic violence relationships, think about what age range will be included? What geographic area? All of these issues limit the generalizability.

Identify and explain your sampling strategy. For example, will you use snowball sampling? How will that happen? How will you do your initial recruitment? Why is this the best method for your specific study? How will you know participants meet your inclusion criteria? Using the previous example, how will you know that they have been in abusive relationships?

How many participants do you need? Why did you decide on that number (support it with literature)? For mixed methods studies, you will also need to explain how many participants you need for each aspect, and you will need a power analysis for the quantitative portion. Talk to your methodologist about this.

Carefully describe how you will identify, contact, and recruit your participants. Be very detailed. Remember if you plan to rely on any other people/ organizations for referrals or help in any way, you will need a letter of agreement from them, spelling out exactly what they will do and provide.

In qualitative studies, you need to consider the idea of saturation, meaning when you are not getting any new information from participants. How does this fit with your sample size? If you have not reached saturation by the time you have talked to all of your required sample, what will you do? (hint, keep doing interviews).

Next time we will continue our review - Chapter 3: Instrumentation in qualitative studies. Do you have an issue or a question that you would like me to discuss in a future post? Would you like to be a guest writer? Send me your ideas! leann.stadtlander@waldenu.edu

Chapter 3: Additional Information, quantitative

There are three special cases that are listed in the quantitative checklist that we will examine today: a pilot study, conducting an intervention, and archival data. As you may remember, I am a fan of pilot studies, but you do need to explain why you are doing one and who will be participating in it. Pilot studies are typically done to give you an opportunity to practice your procedures, check the clarity of the measure, and to determine how long the full study will take (so you can report that in your consent form).

Doing an intervention, which means introducing any new training, treatment, or information to your participants is a very tricky proposition at Walden. Even something as simple as having a control group may come under the intervention rules. So if you have been thinking along these lines, I caution you to write to the IRB today (irb@waldenu.edu) and talk to them about it. You, as the researcher, will not be allowed to do the intervention. So, some other group or institution will have to sponsor it. You are only allowed to collect the data on it. After you talk to the IRB about this, if it still makes sense to do an intervention, you will need to clearly describe what will be done and who will do it. Also, clearly indicate your role and what you will and will not be doing with it.

Archival (or secondary) data, using already collected data in your dissertation, is a great method to use. You will need to relate all the information about how the original study was done, including recruitment and data collection. Then you will need to describe the procedure used to access the data, including any permission letters you needed (include them in the appendix). If you are using historical or legal documents (this is uncommon), describe how you know they are accurate and why they are the best possible sources to use.

Next time we will switch to qualitative and mixed methods studies and look at Chapter 3: Participant Selection. Do you have an issue or a question that you would like me to discuss in a future post? Would you like to be a guest writer? Send me your ideas! leann.stadtlander@waldenu.edu

Student webinar on dissertation problem solving

Chapter 3: Recruitment- quantitative

In this section, you need to think details! It must be written in enough detail that someone else could replicate the study based on your description. This is always more difficult than you think it will be, you might find it helpful to talk through the section with someone, such as a classmate or spouse. Have them push you for as many details about it as possible.

The checklist is pretty vague in the specifics of this section. Let's start with how you will recruit- will you put up flyers? Send out emails? If you are sending out emails- where will you get the addresses? Make a note that you will need copies of all communications (flyers, emails) for the IRB and they will need to be in your paper's appendix. If you are planning to get emails from a company, institution, etc. you will need a letter of agreement from them spelling this out.

Once people contact you, what will you say to them? This needs to be written out. If you are sending them to a survey on a website, what will it include? What demographic information will you collect? Typically, you will want to know their sex, age, and maybe socio-economic status. This is collected so you will be able to describe who your sample is. There may be other information that you will want to know to confirm their eligibility to be in the study, perhaps marital status, part or full time work, etc.

Each participant will need to read and agree to a consent form (see the IRB website for a template). If they are completing an online survey, this is usually the first page of the survey. A copy will need to be sent to the IRB and be included in your paper's appendix.

Next, you need to describe how the data will be collected. Again, be very specific on how this will occur. Then you need to discuss if you will debrief your participants in some way (often done in in-person studies). This is where you would thank them for participating and give them an overview of the study, if it wouldn’t have been clear before. Again, a copy will need to be sent to the IRB of the debriefing and added to the appendix.

Finally describe any follow up procedures, such as their being contacted again in the future or having to return for any reason.

Next time we will continue with quantitative studies and look at Chapter 3: Additional Information. Do you have an issue or a question that you would like me to discuss in a future post? Would you like to be a guest writer? Send me your ideas! leann.stadtlander@waldenu.edu

Chapter 3: Population and Sampling Procedures from quantitative studies

In quantitative studies, it is particularly important to understand who your population and who it is not. Therefore, the first thing to address in this section is who is your target population. Again, the best way to think about this is who does the study generalize to? So if you were testing undergraduates, that is who your population really is (contrary to many of the old studies who make the case that they are similar to the population in general). Next, you want to state appropriately how large the population is that you are using. This may take some detective work to get at.

The next section is Sampling. You rarely can test the entire population; instead, you must strategically sample to make sure you get a representative group. How will you do that? There are many ways that have been suggested, the important thing is to realize the cost and benefit of each method and how they affect your study. Let's say that you decide that you will post flyers and have people contact you if they want to participate. There are a number of costs of doing it this way: you will only get people who frequent where you are posting flyers, you will only have people who volunteer, and only people who have a phone to contact you. None of these issues are fatal flaws, but you need to be aware of them.

What are your inclusion criteria (who can participate) and exclusion criteria (who cannot participate)? Think details. If you are doing an internet survey on survey monkey- inclusion criteria include: people who can read English, have access to a computer, as well as any issues related to your population (e.g., divorced for a year).

You will need to do a power analysis to determine how large your sample should be. Check with your methodologist, as to whether they have a preference as to how to do it. I generalize recommend using one of the power analysis calculators available online (do a search for power analysis calculator). You need to give the website that you used for the calculation and why you used the parameters that you entered. You may need to talk to your methodologist about this aspect. 

Next time we will continue with quantitative studies and look at Chapter 3: Recruitment. Do you have an issue or a question that you would like me to discuss in a future post? Would you like to be a guest writer? Send me your ideas! leann.stadtlander@waldenu.edu