Principles of Replication in Composition Studies

Becky Rickly spoke to me often about the importance of RAD research designs--replicable, aggregable, and data-driven. I believe this term goes back to a Richard Haswell article in Written Communication back in April 2005, but it likely has a wider circulation and history. I admit my philosophical acceptance of this approach to research as well as my high regard for Haswell's thinking and scholarship. But it has all been abstract for me--I've never replicated a study. I attempted to set up my own dissertation research as "RAD," but to date I have not replicated it.

So this post will begin the articulation and planning for my own replication study. I need to put up or shut up (even in my own mind) when it comes to replication. I tend to focus on replication because the other two elements of RAD seem to me to be basic parts of any legitimate research design. It will be emperical (or data-driven) and the sampling and analysis of this data should be presented in such a format that it could be compared to other similar studies. I have often wondered why composition studies has resisted replication so much. It gets no respect. I believe it is getting more respect now, but so far I have not seen it highlighted in any CCC articles.

Before I chart out my own replication study, I want to state a few principles of replication that occur to me. I have not as yet read any literature or scholarship on replication other than Haswell (and Johanek discusses it as well). I think for composition studies replication does not have to mean "exact reproduction" of a previous research study to be meaningful. Replication might exist along a spectrum that might run from exact to approximate reproduction to adapted reproduction.

Spectrum of Replication in Composition Research

Exact reproduction----Approximate----Adapted

Which place along this spectrum any replication study would inhabit would depend upon the type of study and the goals of the researcher. My impression of scientific research  that is more experimental and positivistic is that exact reproduction is very important. The study can not be called a replication unless it is exactly done as the previous study. However, I am sure that compromises happen even in the most diligently pursued exact replication study. Composition though is not a hard science, but as Louis Wetherbee Phelps states a "human science." That means it is a highly interpretive, social, and contextual "science" that should not seek the same kinds of exact knowledge as that found in the physical sciences. Psychology, sociology and some education research certainly can serve as models for a more scientific approach to the study of writing, but I don't necessarily think we need to go back to the control group experimental studies of the 60s and 70s. Where that method seems warranted, yes. Let's experiment. The first principle of replication, then, for composition studies is that exact replication is not a prerequisite for engaging in a replication study. As long as the researcher is open about the new limits--and possible adaptations--to the study and can articulate a good rationale for the reduction of the scale of the study (approximate) or the changing of some of the research design (adapted), then I think this replication is allowable. Going too far in reducing the scale or changing the design will result in nullifying the link between the two studies, so there is a degree of change from the exact reproduction that can go too far. What that degree of change is obviously would be a subject for debate among researchers.

Can we replicate qualitative research studies?
Why not? Although qualitative research is founded (sorry for that word...) upon the subjectivity of the researcher and her interpretations, I think that we gain tremendous value by having another researcher bring her subjectivity to the same research question. Just because previous research was based upon another's individual and subjective interpretations, does not mean someone else could not consider the same phenomena or data.
So another principle of replication in composition studies is that ANY methodology (not just experimental designs) can be replicated.

But what about sampling?
Unless the replicating researcher has access to the same data, it will inevitably be the fact that replication studies will use different samples. Does that then invalidate the replication? I think it depends. For a replication study to be credible, it must find a sample that is similar enough to the original sample to be comparable. I should note that in some cases one of the most interesting part of a replication study might be the change of the sample taken. For instance, what if we replicated Emig's study of the composing process of 12th graders, but instead of middle class whites from a good school constituting the sample we used 12th graders at a poor predominantly hispanic population at a high school from the West side of San Antonio. That would be an example of an interesting "Adapted" replication study.
I think the third principle of replication for composition studies is the acceptance that sampling will diverge from the original study, but the researcher should seek comparative and approximate samples to the original (unless the study moves to adapt the sample).

How about methodology and methods?
Theoretically, I believe that replication should take special care to follow the methodology and methods of the previous research study. In practice, however, this may be quite difficult for two reasons: 1) the scale of the original study may be so large that following the exact methodology could be difficult (in that case, the replication study becomes "approximate" at a smaller scale), and 2) the original research design may have flaws in either its methodological approach or implementation of methods (an all too frequent occurrence in composition research). In this case, the researcher has some decisions to make. I would say that if a researchers did make minor changes to the methodological orientation and use of methods the replication could still be called "exact" or "approximate." However, if some significant adjustment to the methods is made then the study must make clear it is "adapting" the study to improve its research design. Perhaps subsequent replications would then benefit from these methodological fixes. The fourth principle then has to do with methodology.


How about results?
Aggregable is the term RAD uses to describe how results from various replicated studies could be compared and put together. I want to suggest that aggregation may be possible in a lot of cases, but too forced in others. We have to allow for a principle of "interpolation" between results of different studies. This interpolation between different replicated studies constitutes the most interesting reason why we would want to replicate in the first place. The principle of interpolation acknowledges various contexts and various subjectivities and seeks for meaningful convergences, divergences, and patterns between them.

Now that I have these principles in mind, my next post will articulate the replication study I am embarking on this Spring.