Thinking based on feedback v1.0beta

Two thoughts at this point:

1) Trios and Triangulation
Survey--Interview--Focus Group Trio
Administrator--Student--DocInstructor Trio
My original design (v1.0beta) outlines two sets of trios. One trio in methods and one trio in subjects.

The reason I had thought about doing this trio of methods and subjects was to get some level of triangulation. The question is how much triangulation do I need?
--Do impressions given by students in the survey correlate with the goals outlined by the administrators?
--Does the impact of reflections described by students in the survey match the observations given by document instructors?
--Do statements provided by students/document instructors in Focus Groups correlate well with the information provided via the survey (i.e. can the survey results be confirmed and perhaps fleshed out within focus groups. I suppose interviews could do this also--but how many would I need to do)

This design reflects my thinking from my inexperienced researcher's perspective. Actually, this design was modeled after what Chris Thaiss and Terry Zawacki recently did at George Mason University to study the place and development of writer's throughout their four years at college (see their book Engaged Writers & Dynamic Disciplines 2006). The surveyed students and faculty about their writing experience in college and then followed up with focus groups and interviews. (But then they performed their study over about a five year period!)

What I see you suggesting is that I may not need a cannon to get at what I want to explore. Use more focused and targeted methods (especially for a class project).

But would a survey be enough? Would I still need to do interviews? Would I do some interviews of administrators first before I put together the survey? Would I follow up with a few interviews of students and doc instructors? Would I survey both students and docinstructors?

2) Second Point--Genre Tracing and Activity Theory
As you probably know, my focus is on Comp/Rhet (despite my growing enfatuation with rhetoric and growing awareness of "compositions" problematic and unrhetorical nature), so I am reading Clay's book with something of a comp/rhet lens. I'm seeing the writing process as an activity system that could be analyzed along the three levels of scope he describes. How would these acts of reflection fit within the larger activity of writing (and education)? If I were to take this "integrated scope analysis" perspective of the writing activity inside freshman composition--with a specific focus/interest in where and how reflection fits--then I might need to also examine various artifacts to gain this broader perspective.

Let me rephrase things--
The freshman comp curriculum is a design intended to help teach students to "write" academically. Lee Ann Carroll in her book Rehearsing New Roles: How College Students Develop As Writers says that what we call "writing assignments" are really complex "literacy tasks" where they construct a complicated sequence of "literacy acts" (3-4). So college composition students are being initiated into a complex and specific genre that performs a particular role within a particular "activity system" (academia). Hence, we have Sharon Crowley's astute observation that composition is not rhetoric. Of course, composition is rhetoric but it is a specialized, focused rhetoric (that many people believes is too narrowly conceived--including me).

What I'm getting at hear is what Spinuzzi would call the macroscopic level of scope of freshman composition. At this level, writing program administrators have certain goals, functions, assumptions and ideologies that underpin their construction of this "system" we call the curriculum (the regularized production of learning across a writing program).

What this study might do, then, is observe and study the place and role reflection has within this "system."

My question, though, is if I would need to do artifact analysis to get at any notion of an "integrated scope" analysis? Would I need to do textual analysis of actual pieces of reflection and other artifacts of their writing process? A "microscopic" view would necessitate close textual analysis it seems to me.

One other thing to add--I get the impression from Rich that there are other Writing Programs that incorporate reflection into their standardized curriculum for freshman composition. Could my survey study for this course be a pilot for a survey that I could attempt to administer at some of these other programs? Just thinking...

Feedback on v1.0beta

Angela's Response

I've read your info (new blog and emails), and I think overall it's a promising study. However, you've done what most energetic and engaged students do when thinking about a study--you've outlined a good year or two of work. I recommend that you begin with a smaller portion. Try an interview with the ICON director, one assistant director, two graders, and then do a focus group or two with the students. Surveys usually work best when you know what some of the likely answers already are, and you can get that from interviews and focus groups. We can always talk about implementing a survey later on.

I think what you've chosen makes good sense for field methods. As a quant researcher, I'd recommend that later--after the class is over--you also consider a quant methodology to look at the same problem (if you're still interested in it, of course). For example, it's one thing for students to say in a focus group that they think the exercises are useless (or valuable, or boring, etc.). That's good stuff to know. However, it doesn't necessarily tell us if they're effective exercises. Are they having the effect on the writing that we are hoping for? To determine that, we can examing the writing reviews and subsequent drafts and see if there are any differences. Just something to think about.

Becky's Response

I think Angela's on target--that is, what you want to do is big (and, honestly, I don't mind that, because early on you realize it is, and you focus accordingly, and you get a sense of what you WILL be able to do for dissertation research in a safe situation). My only concern is this: I am wary of focus groups, especially when you're only doing one, because they tend to be "group think" unless someone very, very skillful leads them. That is, the loudest, most opiniated person becomes what the group thinks. Now--if youwere to conduct a series of focus groups, etc., but then, honestly, it's more trouble than a survey.

I would suggest going the survey route, noting that this would be a focused, preliminary survey--and you can use the information you gather from this one for a larger survey. I think that, in my experience, pilot surveys render more and more effective information than focus groups.

Rich also responded to the blog post itself

Proposal Version 1.0 (beta)

Field Methods—Spring 2007 Proposed Research Project (1.0)

Subject/Design Focus of Study: The use of reflective writing pieces within a freshman composition curriculum

Specific Focus: The use of reflection within the freshman composition curriculum at TTU (ICON).

Problem/Question: The freshman composition curriculum at TTU includes reflection into the curriculum for particular pedagogical reasons, but are reflections serving their intended purpose? Are reflections playing the intended role they are meant to play within the curriculum?

Among the intended pedagogical purposed for reflection are

• to find in-task solutions to assist them in making adaptations to their texts as they are composing
• to generate and construct knowledge about writing and their own writing process themselves
• to formulate post-task awarenesses and knowledge that can assist them when they encounter future writing situations

Research Methods

1) Separate surveys of both students and document instructors (who evaluate these reflective writings) to gain broad understanding about the "experienced curriculum" beliefs and attitudes about the role reflection plays in the curriculum.

2) Focus Group interviews with a group or groups of students and document instructors (in separate groups) to flesh out these beliefs and attitudes in more detail.

3) Possible interviews with the current and past Composition Program administrators to discover their view on the purpose of reflection in the curriculum

4) Additional "programmatic" data about reflections could be gained via datamining in the TOPIC system.

Purpose for Research: Kathleen Yancey talks about the difference between the "delivered curriculum" and the "experienced curriculum" in her book Reflection in the Writing Classroom. This study would seek to find if the intended purposes behind including reflection into the composition curriculum (the delivered curriculum) matches the experience that students are actually having (the experienced curriculum). This information should assist the Composition Director in future course designs of freshman composition at Texas Tech.

Possible Larger Implications: Writing Program Administrators at other writing programs may also include reflection into their course design for freshman composition courses. The information generated about reflection at Texas Tech may bare possible implications for their own context.

The Great SA Ice Storm of 2007

Here are a couple of pics from our backyard:

The bird feeder and a frozen pecan tree.

A Possible Study--Writer's Reviews in ICON

From deliberating over Becky's response to my list of possible research projects, she has gently been steering me toward a more "field methods" type of project that focuses on solving a problem, answering a question, or gathering information to address a situation/context/design. She suggests that I might study the "design" of using Writer's Reviews within ICON. This study would seek to answer the question of how writer's reviews are perceived by students, document instructors, and even the curriculum designers. How are these WRs done? It would study the place and role these writer's reviews have within this curriculum design and writing program. What would the writing program director (Susan Lang) need to know about how writer's reviews are actually being used by students and document instructors in order to better take advantage of their potential for the writing program? (Are they worth doing?)

This study would probably involve surveys of the different users and even follow up focus groups or interviews. It could possibly involve some data-mining, but I think that might begin to get out of hand (it depends how easy it is to do the data-mining).

This study looks very do-able and I would hope relevant to Texas Tech and ICON.

I'm still thinking...
It doesn't dig as deep as I had intended.

Returning to the Center

This post seeks to present the core of what I am interested in with my research--reflection. This quote from a workshop description captures the commonplace view of reflection:

"It is important to engage in the metacognitive process of reflection if you want to change and grow. Metacognition is thinking about thinking. It empowers you to know what you know and know what you don’t know. Once you engage in this type of reflection about your own thinking you can be deliberate and focused in your planning. The metacognitive process helps us move toward evaluating what we currently think and move beyond it. … It is an agent for deliberate and strategic change." (“Topic 7: Reflect on Learning.” Instructional Design Workshop. 2002. 2 February 2005 )

Does reflection really work as this "agent" for growth and change? For reflections done IN-TASK (within the midst of a writing process), what sort of growth and change do they encourage? What sort of knowledge does reflection "empower" the writer to know? What sort of planning does it help the writer be deliberate and focused about?

This nature of agency found within reflection for writers is what I am interested. My previous case study suggested that the most significant focus for productive deliberation for writers was reflection upon their "rhetorical stance." This deliberation represents a heuristic extension of invention throughout the writing act.

How do I study this agency in reflection?

NWP New Sites Institute

I'm off early tomorrow morning for the NWP New Sites Institute in Oakland. It should be a great time. I should have a good amount of time to think about my research on the flight there, and the big bonus is I'll get to stay with my brother.

I'm hoping to look through our class transcript and think about my past experience researching reflection. What did I do right? What did I do wrong? I also want to look again at overviews of various method between my MacNealy book and the article we read in class.

I want to think/rethink what my real question is? Becky mentioned a problem as being the starting point of research? What's my problem? It is interesting that Becky and Amy decided to do these type of "reflection-in-action" or what I'm tending to call in-task reflections (as opposed to post-task reflections) for our class. That's what I am continually doing in this blog. Maybe I could study our class!

Bad news from Susan Lang about the changes to the comp program at TTU. It sounds like they are only doing two drafts per essay cycle which means only one in-task reflection. I'm beginning to think that three drafts are needed for real development and growth. It looks like I may not be able to use data from TTU students. Maybe I could?

My goals for this semester

As I begin this semester, I want to think about what I'd like to accomplish. Ideally, I would like to do a research study that will serve as a pilot study for my eventual dissertation study. I'm still undecided (and need help in this decision) whether I should do a replication of my Spring 2005 Case Study. I see already that this class will provide me with a more sophisticated understanding of my methodology and my choice of exact methods. The question I have now is whether a Case Study will reveal anything meaningful about reflection and its role in the writing act.

Reading Patricia Goubil-Gambrell's article on research methods provided a good overview of the different methods. I see from my previous work that I have been far too fast to draw generalizable conclusions from the data I have so far analyzed. Heck, from my last class in Contemporary Rhetoric and my exposure to post-modern thinking, can we ever reach any generalizable conclusions. She says, "The purpose of qualitative research is to identify salient features or variables in the situation" (588). It is descriptive in nature. It makes me think a bit (just a comparison here) about the role teaching demonstrations have within the National Writing Project model of faculty development. They demonstrate some teaching strategy or assignment in action for that teacher's students. We watch how they accommodate their own situation and reach the students in that situation. IF we generalize from that teacher's demonstration it is in how we might adapt it to fit within our own teaching context. What is the first question to start discussion after an NWP demo? "How might you use this assignment in your classroom?" It is never a one-for-one match.

Thinking about the role my research might have in terms of the function of an NWP teaching demo is actually very helpful for me. The problem, however, is if I set up too rigid of perameters for when and how this reflection happens. The problem is if I believe that reflection done only in this way in this situation within the writing process will get these results. I'm still caught to a degree in desiring a cause-effect relationship which means that I would need to use a different methodology (experimental): "As Lauer and Asher have noted, co-occurrences are often misinterpreted as cause-and-effect relationships in qualitative research" (589. That is what I have tended to do with my observations of repeated results in my previous results. I have to ponder this weakness and see how I might avoid it this time (if I do another case study).

Burke and the Epistemic Power of Language/Reflection

I've been reading the section in Kenneth Burke's Rhetoric of Motives called "Traditional Principles of Rhetoric" because I am becoming more interested in his views on rhetoric, particularly how he interprets classical rhetoricians. The chapter is dense and full of excellent "motives" of rhetoric, but I was particularly struck by a passage he has on page 178:

"For the distinctive insight into human invention is not the use of tools, since animals use tools; it is in the use of tools for making tools. And this insight-at-once-remove, this reflexive pattern, is much like the insight of language itself, which is not merely speech about things (a dog's barking at a prowler could be called that), but speech about speech. This secondary stage, allowing for "thought about thought," is so integrally connected with the human power to invent tools for making tools, that we call such power linguistic in essence (as Carlyle did)."

This passage is a typical Burkian moment of brilliance that he casts out like a jewel in the road--and then he moves on. Here he observes the inventional (epistemic) nature within our linguistic capacity to talk about our talk, to write about our writing, to think about our own thinking (since even our thoughts are an internal dialogue). "Inventing tools" is an interesting comparison for what we do with language since a word is not literally a hammer or a caliper or a GPS device. I think Burke captures what is important about reflection as a "tool for making a tool"; something emerges from reflection that is like a tool to transform our perspective, our assumption, or direction for action or thinking. And is is done linguistically!

Conceptual Metaphors

My wife said an amazing thing to me just the other day. Since I think about reflection quite a bit, and she is quite reflective herself and thinks often in terms of psychology, we often will talk about reflection. She made the comment that the term "reflect" is a metaphor. Of course it is! I have been so focused on the notion of "stance" as a spacial metaphor to describe how we understand our thinking while shaping and negotiating all the aspects of writing that I had ignored the obvious and crucial point that even the term "to reflect" is a metaphor for this complex experience of "thinking about our thinking," of Janus-like looking back as we look forward. I shouldn't be surprised because all language is founded upon metaphor, but now that I am aware of the metaphor at the root of "reflect" I have to process and think about it more and what it means that we choose this visual image of mirroring to describe this mental operation/function/experience. How does this metaphor work as a "terministic screen" to shape how we understand and even experience reflection? How does is link with other metaphors to operate on the larger level of a conceptual metaphor? What are those other metaphors? At this point, I can only ask these questions, but now that I am aware of this metaphoric root within reflection I can begin to explore these answers.

Spring 2007 Field Methods Research ideas (#1)

Copy of email sent to Dr. Becky Rickly 1/3/07
*****************************************

Hi Becky,

I'm so glad to hear from you. I've been having a metaphysical crisis because my accd.edu email is not working and my dad even said a TTU email he sent to me got bounced, so I've been fretting horribly about not getting email. I've also noticed that I am signed up for the course (or so I think), but I don't have a balance to pay. I'll look into my registration situation today.

I'm writing you (and cc:ing Rich, my diss chair) because I'd like whatever research I do in this class to count in some way toward my eventual dissertation research. What follows will be a summary of what I am thinking of doing with a lot of messy ideas/possibilities. I will REALLY appreciate your help and feedback as I decide what to do this semester. Since what I plan to do may require me to set up certain conditions within my own classes for this semester, your response (at least in a broad sense) will help a lot.

Here it goes:
**************************

Lennie's diss research

Disciplinary Focus--Composition/Rhetoric

General Area of Research Interest--reflection, reflection in the writing process

Focused Target of Research Interest--post-draft reflections (called Writer's Reviews in TTU TOPIC)

Research Question: (rephrased in various ways)

What is happening in terms of the writer's "process" (activity and progress with writing a text) within writer's reviews?

What affect do writer's reviews have on a writer's process of writing and eventual end product?

Do writer's reviews have a significant impact on a writer's writing process and activities in writing?

Can we see evidence that writer's reviews are a form of invention for writers?

Who Gives A Rat's Ass (WGRA factor)

Reflection may not be "trendy," nor is research into the writing process (since post-process ideas have complicated our view of the writing process), BUT reflection is a powerful tool for developing rhetorical sensitivity and practical wisdom (phronesis) for writer's within the activity of writing. It parallels the practice of usability testing within the field of tech comm. for an iterative development process. I think, also, that rhet/comp's views toward reflection are too narrowly focused on portfolio reflection (or a final student self evaluation as in Service Learning) and not on the role reflection plays WITHIN a process/activity rather than AFTER it is completed. The other area of significance for this project is that it may lead us to look at invention differently because my feeling is that these writer's reviews constitute a heuristic extension of invention within the writing process. Invention within writing instruction too often gets pigeoned-holed into "pre-writing" and methods and means of identifying how invention continues throughout the activity of writing are too little defined within our view of the writing process. Post-draft reflections (writer's reviews) are one formalized activity prompting (or re-prompting) invention. Also, poststructuralist views of "the subject" have basically castrated our sense of invention—much to the detriment of our conception of rhetoric (in my view). This study (perhaps naively in the poststructuralist view) tries to recover our conception of the subject and its potentialities for invention.

Previous Work:

In Spring 2005 I did a Case Study on post-draft reflections for students in my Fresh Comp II classes. I examined the writer's reviews for four students through two writing cycles which constituted three writer's reviews for each cycle (three drafts, three writer's reviews—parallel to TTU). I used Marcia Baxter-Magolda's "Constructivist Interpretation Process" to guide my methodology and my methods. Here is a link to my paper:

http://www.accd.edu/sac/english/lirvin/TTech/5363LLIpaper.htm

Here is a summary of Baxter-Magolda's constructivist interpretation process:

http://www.accd.edu/sac/english/lirvin/TTech/5363paper/Appendix2.htm

--RESULTS of Case Study:

I found that the most significant thing happening within writer’s reviews is that students were working on their “rhetorical stance” and you could see signs of how they either transformed or confirmed their stance. “Rhetorical stance” is a term signifying a metaphor that describes all the constraints a writer balances and brings into alignment as they write (the writer’s situation, Kinnevey’s communication triangle—writer, subject, text, audience, situation). Reflection is where students in some respect seek to adapt general principles to specific situations (a la Kant’s Reflective Judgment). This result led me to conclude that these acts of reflection occurring “in and amongst the drafts” represented an extension of invention—testing, adapting, identifying and solving problems. Something is also happening because this is done in writing (i.e. discursively).

SUBSEQUENT WORK (leading to where I am now):

Summer 2005—Rhetorical Analysis class with Sean Zdenek. Learned more about discourse analysis; did paper on metaphor. Since the notion of “rhetorical stance” is itself a conceptual metaphor, I am interested in the conceptual metaphors used by students to represent their writing and writing activity.

Fall 2005—Theory’s of Invention class with Amy Koerber

I made my class paper project an exploration of what invention is and how it might or might not relate to invention. I wrote a paper identifying direct connections between invention and reflection, building the case that reflection represents a heuristic extension of invention.

Summer 2006—Classical Rhetoric with Rich Rice

Since my whole research interest presupposes a “writing process,” I felt I had to learn more about post-process theories which often are interpreted as a negation of the writing process. What they reject is a formulaic, generalizable writing process (the science of writing). Post-process is actually an affirmation of rhetorical principles in communication. My course paper debunked some of the more extreme post-modern views of communication by grounding and tempering some of its views with classical rhetoricians. Phronesis, practical wisdom applied in contingent situations involving uncertain knowledge, became a strong focus for my work. It is my belief that our ultimate goal as writing instructors is to develop this capacity for phronesis in our students.

Additional subsequent work:

I have gathered more data from students

10-12 sets of writers reviews from two writing cycles from four different sections of 1301 students. (I just have the data; I did not do the full constructivist interpretation process including interviews and checking my interpretations with this data. In fact, I haven’t closely analyzed it yet, just glanced at it.)

4 sets of writers reviews from two sections of Developmental English for on writing cycle. (unanalyzed at this point)

***********************************

RESEARCH PROJECT POSSIBILITIES

***********************************

General Goal:

Ideally, I’d like to get the core of my dissertation research done this semester. Hah! Realistically, I would like to do something that would in someway move me forward in doing the eventual research I will do for my dissertation.

Eventually, I really would like to tap into the TOPIC Vault because it offers such a vast amount of data and it is in a form that allows for some new and innovative approaches to research. My original thinking was to do qualitative research that might lead me to do something more quantitative within TOPIC. Last May, Susan Lang and I talked about some possibilities for data mining or even setting up some special treatments within TOPIC. That might mean that my eventual dissertation might include a mixture of methods???? I think I am still at the stage of doing qualitative research that might help inform whatever I do with TOPIC in a quantitative way.

What to do this semester?

Option A) Case Study Redo (either with my students or with TTU students—I assume that Fresh Comp at TTU is still using the three draft, three writer review writing cycle?)

This would involve getting two writing cycle’s worth of data for four to six writers. What might be adjusted and improved this time are my research methods. I could do a mix of SAC and TTU students?

My thinking for this option is that I should try to do TTU students in a 1301 class, but this makes it harder to do interviews. One big value I found in my other case study was interviewing my subjects and testing my own interpretations with their impressions. I could ask about specific things they had written in their reflections. I could do interviews via TTU MOO or I could fly up one day for interviews (if I could manage to get interviews with all my subjects in that time). This project might be more do-able because I would get writer’s reviews from the second and third essay or just the second essay; that way, I would have the time period of the first essay cycle to set up my research study before I began gathering data.

Option B) Survey/Focus Group Study

If I were to use TTU students, this study would again depend upon TTU still using writer’s reviews with its composition classes.

I could design a survey to gather student impressions about doing reflections. I could also survey document instructors and teachers about them. Then I could do a number of focus group discussions with these subjects to flesh out the surveys. This could be matched with some individual interviews perhaps. Focus group discussions could be held in the MOO!

Option C) Data-mining

I’m not sure this qualifies as “field methods” because I’m thinking our class will involve more qualitative methods, but this option involves identifying “key terms” (which may signify important conceptual metaphors used in reflections hammering out rhetorical stance) and doing a broad TOPIC data-mining for the prevalence of these terms. Analyzing the results sounds tricky to me.

Option D) Quasi-Experimental (or even Experimental Study)

This would be more work, and might not fit for this class, but I’ll outline this study too. This, again, presupposes TTU still using writer’s reviews. In this study, I could establish a “special treatment” for a class or cohort of students. Pre- and post-tests could be established in the form of writing samples, but this is labor intensive as far as holistically scoring these. The special treatment could be either a )one cohort not having to writer writer’s reviews (so you could compare them to the overall population of TOPIC students) or b) this cohort could get a different set of writer’s review prompts that might have been specially designed to illicit deliberation of rhetorical stance and this group could be compared to the general population. What we would compare would be key. As I mentioned, we could do the laborsome task of getting pre- and post- test writing samples as well as pre- and post- course surveys.

The cool thing about TOPIC is that this kind of study possibly could be turned into a true experimental study since TOPIC might allow for a random sample.

The problem with this kind of study is that it will take a long time to set up, and lots of labor to pursue and I don’t think I am positioned to pull all this together at this late date. The question remains, too, about whether this kind of method will truly show in a reliable and valid way what I am hoping it will show.

Blog Purpose

Speculum is Latin for "mirror." This blog will be a mirror and digest for my thinking related to my dissertation work on the role of post-draft reflections in the writing process. In it I will post kernels of information and jewels that I find a long the way. I will also brainstorm and push my thinking as I work on this extended project. Since the blog will be public--or at least public to those I invite--it will also be a place where others can collaborate with me in this work.

Lennie raises his champagne bottle and virtually smashes it against the side of the blog.

We're launched.