I want now to outline the basic research design of my replication study of my dissertation research.
General Goals of Study:
I have to ask my self seriously whether this replication is really targeted at verification at this point. If so, it changes the whole nature of the study. Grounded theory is about theory generation, not verification, so if I tip over to verification I would be altering my original study to greatly. I must maintain a sense that I am still evolving and discovering this theory. If I want to "verify," I think that needs to be another study.
However, I think verification has a double-meaning within grounded theory methodology. Verification runs through out the method of constant comparison and what might be called the double-movement of grounded theory data analysis (or coding). Here is how it works. I observe the data closely, naming phenomena, and beginning to state understandings or meanings about what is going on. When I go from my sense of "what is going on" and then look again at the data to see if what I think is going on is really going on--that is a kind of verification. As I do this same checking by comparing one instance with another instance--that is verification too. Selective coding, at least as I understood it, was a form of verification around a core concept. You took that core concept and then checked to see if it fit and worked through various instances. Hence, I think I need not be afraid of verification because it is within the fiber of grounded theory methodology, but I don't think I should start there.
Outline of Research Design:
Sample:
Six "essay cycles" from my students between Essay #2 draft 1 and draft 2.
This might be followed with another set in either draft E3-2 to E3-3 or E4-3 to E4-3.
(I'm thinking for this round I will keep it smaller and more manageable).
Data includes: assignment, draft 1, peer responses, writing review, and next draft
Coding Procedure:
recode a fresh set of data following the procedure I used for slice 5
--initial coding and memoing
--perhaps chart the dynamic of rhetorical reflection (?)
--create problem-analysis chart of essay cycle
--further memoing and diagraming
--attempt to replicate selective coding around comparison, assessment, judgment made in terms of essay success
Member Checking:
Since I have access to students this time, I am able to talk to them about their writing, reflections, and thinking. Once I have my data and some conclusions, I will talk to them.
Final Memoing comparing my impressions with those thoughts stated by the students.
This replication can not be called "exact" and perhaps not even "approximate" or "adaptive." So my categories fall apart this quickly! This research is really an extension. If seeks replication but it also seeks to follow the initial trajectory of the original research--taking it to the next step or level that the original research did not reach.
Hmm... So is extension really replication? In this case I believe so because grounded theory is a set of repetitions and replications of constant comparative analysis over an over again.
One complicating factor that definitely makes this replication study approximate is that I am using my own students. I already feel some anxiety about who to choose to include in this study. I know these students. How do I pick this sample based on theoretical reasons and not any sort of rationale based on getting a "representative sample" as in experimental designs?
What will be the theoretical basis for my selection? Could I base this selection upon my theory of the spectrum of reflectiveness? Between reflections that predominantly tell/report vs. those that consider/evaluate? I would anticipate to see either a high or low degree of reflectiveness (in the form of comparison, assessment, and judgment made in terms of essay success) within these different populations. Am I setting up to great a "checking" with this hypothesis testing sample section rationale?
I think at this point I will have to ponder my sample section a bit more. Perhaps continue to review my previous research up to this point and see what I wanted for the next slice of data.
Monday, March 12, 2012
Sunday, March 11, 2012
Principles of Replication in Composition Studies
Becky Rickly spoke to me often about the importance of RAD research designs--replicable, aggregable, and data-driven. I believe this term goes back to a Richard Haswell article in Written Communication back in April 2005, but it likely has a wider circulation and history. I admit my philosophical acceptance of this approach to research as well as my high regard for Haswell's thinking and scholarship. But it has all been abstract for me--I've never replicated a study. I attempted to set up my own dissertation research as "RAD," but to date I have not replicated it.
So this post will begin the articulation and planning for my own replication study. I need to put up or shut up (even in my own mind) when it comes to replication. I tend to focus on replication because the other two elements of RAD seem to me to be basic parts of any legitimate research design. It will be emperical (or data-driven) and the sampling and analysis of this data should be presented in such a format that it could be compared to other similar studies. I have often wondered why composition studies has resisted replication so much. It gets no respect. I believe it is getting more respect now, but so far I have not seen it highlighted in any CCC articles.
Before I chart out my own replication study, I want to state a few principles of replication that occur to me. I have not as yet read any literature or scholarship on replication other than Haswell (and Johanek discusses it as well). I think for composition studies replication does not have to mean "exact reproduction" of a previous research study to be meaningful. Replication might exist along a spectrum that might run from exact to approximate reproduction to adapted reproduction.
Spectrum of Replication in Composition Research
Exact reproduction----Approximate----Adapted
Which place along this spectrum any replication study would inhabit would depend upon the type of study and the goals of the researcher. My impression of scientific research that is more experimental and positivistic is that exact reproduction is very important. The study can not be called a replication unless it is exactly done as the previous study. However, I am sure that compromises happen even in the most diligently pursued exact replication study. Composition though is not a hard science, but as Louis Wetherbee Phelps states a "human science." That means it is a highly interpretive, social, and contextual "science" that should not seek the same kinds of exact knowledge as that found in the physical sciences. Psychology, sociology and some education research certainly can serve as models for a more scientific approach to the study of writing, but I don't necessarily think we need to go back to the control group experimental studies of the 60s and 70s. Where that method seems warranted, yes. Let's experiment. The first principle of replication, then, for composition studies is that exact replication is not a prerequisite for engaging in a replication study. As long as the researcher is open about the new limits--and possible adaptations--to the study and can articulate a good rationale for the reduction of the scale of the study (approximate) or the changing of some of the research design (adapted), then I think this replication is allowable. Going too far in reducing the scale or changing the design will result in nullifying the link between the two studies, so there is a degree of change from the exact reproduction that can go too far. What that degree of change is obviously would be a subject for debate among researchers.
Can we replicate qualitative research studies?
Why not? Although qualitative research is founded (sorry for that word...) upon the subjectivity of the researcher and her interpretations, I think that we gain tremendous value by having another researcher bring her subjectivity to the same research question. Just because previous research was based upon another's individual and subjective interpretations, does not mean someone else could not consider the same phenomena or data.
So another principle of replication in composition studies is that ANY methodology (not just experimental designs) can be replicated.
But what about sampling?
Unless the replicating researcher has access to the same data, it will inevitably be the fact that replication studies will use different samples. Does that then invalidate the replication? I think it depends. For a replication study to be credible, it must find a sample that is similar enough to the original sample to be comparable. I should note that in some cases one of the most interesting part of a replication study might be the change of the sample taken. For instance, what if we replicated Emig's study of the composing process of 12th graders, but instead of middle class whites from a good school constituting the sample we used 12th graders at a poor predominantly hispanic population at a high school from the West side of San Antonio. That would be an example of an interesting "Adapted" replication study.
I think the third principle of replication for composition studies is the acceptance that sampling will diverge from the original study, but the researcher should seek comparative and approximate samples to the original (unless the study moves to adapt the sample).
How about methodology and methods?
Theoretically, I believe that replication should take special care to follow the methodology and methods of the previous research study. In practice, however, this may be quite difficult for two reasons: 1) the scale of the original study may be so large that following the exact methodology could be difficult (in that case, the replication study becomes "approximate" at a smaller scale), and 2) the original research design may have flaws in either its methodological approach or implementation of methods (an all too frequent occurrence in composition research). In this case, the researcher has some decisions to make. I would say that if a researchers did make minor changes to the methodological orientation and use of methods the replication could still be called "exact" or "approximate." However, if some significant adjustment to the methods is made then the study must make clear it is "adapting" the study to improve its research design. Perhaps subsequent replications would then benefit from these methodological fixes. The fourth principle then has to do with methodology.
How about results?
Aggregable is the term RAD uses to describe how results from various replicated studies could be compared and put together. I want to suggest that aggregation may be possible in a lot of cases, but too forced in others. We have to allow for a principle of "interpolation" between results of different studies. This interpolation between different replicated studies constitutes the most interesting reason why we would want to replicate in the first place. The principle of interpolation acknowledges various contexts and various subjectivities and seeks for meaningful convergences, divergences, and patterns between them.
Now that I have these principles in mind, my next post will articulate the replication study I am embarking on this Spring.
So this post will begin the articulation and planning for my own replication study. I need to put up or shut up (even in my own mind) when it comes to replication. I tend to focus on replication because the other two elements of RAD seem to me to be basic parts of any legitimate research design. It will be emperical (or data-driven) and the sampling and analysis of this data should be presented in such a format that it could be compared to other similar studies. I have often wondered why composition studies has resisted replication so much. It gets no respect. I believe it is getting more respect now, but so far I have not seen it highlighted in any CCC articles.
Before I chart out my own replication study, I want to state a few principles of replication that occur to me. I have not as yet read any literature or scholarship on replication other than Haswell (and Johanek discusses it as well). I think for composition studies replication does not have to mean "exact reproduction" of a previous research study to be meaningful. Replication might exist along a spectrum that might run from exact to approximate reproduction to adapted reproduction.
Spectrum of Replication in Composition Research
Exact reproduction----Approximate----Adapted
Which place along this spectrum any replication study would inhabit would depend upon the type of study and the goals of the researcher. My impression of scientific research that is more experimental and positivistic is that exact reproduction is very important. The study can not be called a replication unless it is exactly done as the previous study. However, I am sure that compromises happen even in the most diligently pursued exact replication study. Composition though is not a hard science, but as Louis Wetherbee Phelps states a "human science." That means it is a highly interpretive, social, and contextual "science" that should not seek the same kinds of exact knowledge as that found in the physical sciences. Psychology, sociology and some education research certainly can serve as models for a more scientific approach to the study of writing, but I don't necessarily think we need to go back to the control group experimental studies of the 60s and 70s. Where that method seems warranted, yes. Let's experiment. The first principle of replication, then, for composition studies is that exact replication is not a prerequisite for engaging in a replication study. As long as the researcher is open about the new limits--and possible adaptations--to the study and can articulate a good rationale for the reduction of the scale of the study (approximate) or the changing of some of the research design (adapted), then I think this replication is allowable. Going too far in reducing the scale or changing the design will result in nullifying the link between the two studies, so there is a degree of change from the exact reproduction that can go too far. What that degree of change is obviously would be a subject for debate among researchers.
Can we replicate qualitative research studies?
Why not? Although qualitative research is founded (sorry for that word...) upon the subjectivity of the researcher and her interpretations, I think that we gain tremendous value by having another researcher bring her subjectivity to the same research question. Just because previous research was based upon another's individual and subjective interpretations, does not mean someone else could not consider the same phenomena or data.
So another principle of replication in composition studies is that ANY methodology (not just experimental designs) can be replicated.
But what about sampling?
Unless the replicating researcher has access to the same data, it will inevitably be the fact that replication studies will use different samples. Does that then invalidate the replication? I think it depends. For a replication study to be credible, it must find a sample that is similar enough to the original sample to be comparable. I should note that in some cases one of the most interesting part of a replication study might be the change of the sample taken. For instance, what if we replicated Emig's study of the composing process of 12th graders, but instead of middle class whites from a good school constituting the sample we used 12th graders at a poor predominantly hispanic population at a high school from the West side of San Antonio. That would be an example of an interesting "Adapted" replication study.
I think the third principle of replication for composition studies is the acceptance that sampling will diverge from the original study, but the researcher should seek comparative and approximate samples to the original (unless the study moves to adapt the sample).
How about methodology and methods?
Theoretically, I believe that replication should take special care to follow the methodology and methods of the previous research study. In practice, however, this may be quite difficult for two reasons: 1) the scale of the original study may be so large that following the exact methodology could be difficult (in that case, the replication study becomes "approximate" at a smaller scale), and 2) the original research design may have flaws in either its methodological approach or implementation of methods (an all too frequent occurrence in composition research). In this case, the researcher has some decisions to make. I would say that if a researchers did make minor changes to the methodological orientation and use of methods the replication could still be called "exact" or "approximate." However, if some significant adjustment to the methods is made then the study must make clear it is "adapting" the study to improve its research design. Perhaps subsequent replications would then benefit from these methodological fixes. The fourth principle then has to do with methodology.
How about results?
Aggregable is the term RAD uses to describe how results from various replicated studies could be compared and put together. I want to suggest that aggregation may be possible in a lot of cases, but too forced in others. We have to allow for a principle of "interpolation" between results of different studies. This interpolation between different replicated studies constitutes the most interesting reason why we would want to replicate in the first place. The principle of interpolation acknowledges various contexts and various subjectivities and seeks for meaningful convergences, divergences, and patterns between them.
Now that I have these principles in mind, my next post will articulate the replication study I am embarking on this Spring.
Subscribe to:
Posts (Atom)