Magazine Article for Intercom Magazine
Supercharge the Results of Usability Testing with Formative Reflection
Having information is not enough? It's what we do with that information that matters. As we saw from our government's intelligence shortcomings pre-9/11, we might have a host of data, but this data is useless unless we "connect the dots" and pull together a comprehensive understanding of its meaning. As technical communicators who follow principles of User-Centered Design (UCD) and Information Development, we engage early and often in usability testing, generating substantial amounts of data and analysis. The iterative testing and design process requires us continually to make sense of these results of iterative tests to move our design efforts forward. This article proposes a technique called "Formative Reflection" to help the information design process.
Formative Reflection is a strategic technique for assessing the results from usability testing within an iterative design process. Useful for low-fidelity to high fidelity prototypes, Formative Reflection's purpose is to assist technical communicators and information designers make sense of iterative testing in order to formulate new directions for design improvements. This technique is most productively done as an act of writing, though it can be done verbally. It prompts technical communicators to pull together past, present, and future in their deliberations. It is helpful in conceptualizing problems and helping technical communicators and designers check their thinking so that they can more productively move forward in the iterative design process. Formative Reflection is particularly effective when applied to the assessment and design of what Janice Redish calls "complex information analysis" that involves open-ended, complex problems.
Whereas Formative Evaluation seeks to present data and provide analysis derived from usability testing or other forms of review, Formative Reflection goes beyond just analysis to consider the implications of testing results. Formative Evaluation is something like the ill-fated daily briefing presented to the President on
When Is It Best To Use Formative Reflection?
Formative Reflection borrows ideas about professional practice and learning theory from Donald Schon, David Kolb, and John Dewey. For each of these theorists, reflection plays a key role in learning and informing action.
Donald Schon (1983, 1987) is best known for his ideas of "reflection-in-action" and "reflection-on-action" in professional practice. He points out the shortcomings of what he calls "technical rationality" and the narrow application of scientific rules and principles in our work. He contrasts the high ground of how things ought to be (based upon research-based theory and technique) and the swampy lowlands of how things really are—messy, confused, and unique. When a problematic situation presents itself as a unique case, it often falls outside the professional's current knowledge and techniques. It's not "in the book." Schon calls the inability to deal with the gap between this form of textbook, scientific knowledge and our understanding of particular situations a crisis in professional knowledge. He says that if the professional practitioner is to deal with a unique situation competently, "she must do so by a kind of improvisation, inventing and testing in the situation strategies of her own devising." Reflection, Schon thinks, helps bridge this gap through a kind of translation process between general rules and particular situations that makes technical problem-solving possible.
The iterative design process has many similarities to what David Kolb has called the "Experiential Learning Cycle." The cycle of User-Centered Design can be represented broadly in this way:
Figure 1: The iterative usability testing cycle.
Just as in the UCD iterative testing process the improvements in design depend upon the frequency and quality of these user tests, in the Experiential Learning cycle Kolb believes that the progress in learning from experience depends upon the quality of the reflective observation. It is from this analytic and detached position that a learner creates a broader understanding of the experience and formulates a new way of approaching it for another attempt.
Translating Kolb's view on the cycle into UCD terms, the cycle might look something like this:
Figure 3: A revised version of the Experiential Learning Cycle for User-Centered Design
Each usability test represents an experience from which the designer must learn. It is the application of Formative Reflection that assists the technical communicator and information designer form the important insights and understandings necessary to move the design process forward successfully.
Formative Reflection, however, may not be necessary in every iterative design cycle. If Schon points out how reflection helps us apply general principles within particular contexts, and Kolb describes how reflection helps us stand back and learn from experience, John Dewey highlights the special purpose of reflection—to address perplexity. It is within a situation of doubt, uncertainty, or difficulty where reflection's special role comes into play. Dewey (1933) described reflection as a special form of thinking where we consider a belief or knowledge, but also evaluate the basis for that belief or knowledge and the conclusions to which it leads. A sense of doubt or perplexity is what triggers the reflection process and drives it. Through this focused examination of where they have been, where they are, and where they might or need to go, reflective thinkers are led to insights that can transform or confirm their stance within the development process.
For technical communicators and information designers that means Formative Reflection can be a methodology employed when what to do based upon usability testing is unclear. If we think of the iterative design process as a spiral-shaped road that leads forward from concept to final product or document, Formative Reflection is applied within instances when a tree has blocked that road or the road suddenly comes to a place where it branches in five directions. Formative Reflection can be a way of problem-solving out of set-backs when usability testing has produced particularly disappointing results. It can help us make the right choice when we have multiple possible paths to pursue in development.
How to Do Formative Reflection
Applying Formative Reflection should be a flexible procedure that adapts to the particular needs of a situation where technical communicators face this perplexity in their design process. However, I would like to outline some general guidelines for how you might use this methodology in your design practice.
Do it in writing!
Many companies may employ a verbal approach to evaluating the implications of usability testing by gathering in meetings to discuss findings, but Formative Reflection should be done in writing (at least at first). But why writing? Many theorists of reflection, particularly Max Van Manen, see the particular value of reflection coming from its power to distance a person from events so that they can be viewed in a more objective manner. Writing by its nature gives this kind of distance. Elliot Eisner also sees writing as important for our learning because how we represent what we know actually shapes what we know. The idea is that we come to insights and understandings as we write. It may be awkward within a work environment not used to this kind of silent activity, but Formative Reflection should be seen as a technique applied when needed just as Card Stacking or Affinity Diagramming is used when necessary. Just as these are strategic activities conducted in particular ways to help achieve a particular end, Formative Reflection works the same way—in writing.
Open-Ended Formative Reflection
One approach to Formative Reflection is to conduct an open-ended reflection without a strict prompt. Particularly in instances early in the iterative development cycle, this open form of reflection will promote innovative thinking and problem-solving. Below are two examples of open-ended Formative Reflection prompts:
What are your thoughts on the usability test report and where we need to go from here?
What's working? What's not? Why? Where do we go from here?
Structured Formative Reflection
Structured Reflection has been used in Nursing to help nurses learn from incidents they encounter as they work. It entails filling out a fairly detailed form asking them to evaluate and think about a clinical incident. The usefulness of a Structured Formative Reflection is that you can include specific questions pertaining to the particular development process you are engaged within.
I would like, however, to offer two general frameworks for a more structured Formative Reflection. The first comes from David Boud, a theorists on reflective learning in the workplace. He presents four key elements to consider when re-evaluating experience:
1. association—How does the new data relate to that which is already known?
2. integration—What relationships are apparent within the data?
3. validation—What is authentic? What can be confirmed and what can be negated?
4. appropriation—What understandings and insights do I have?
Another structure for a more detailed form of Formative Reflection comes from the ideas of Jack Mezirow, a theorist in Adult Learning. He sees reflection as taking three different forms: content reflection, process reflection, and premise reflection. Content reflection would be similar to the focus on the data and what its implications are found in Boud's four elements. Process reflection, however, adds the important element of design activity and sequence. The affect of product design can be as impacted by process as by bad design, and the entire design enterprise entails a large scale coordination of personnel and resources. Premise reflection can be the hardest, but most important focus to have while doing reflection. Within premise reflection, you play the devils advocate and question beliefs or procedures that you take for granted. Mezirow believes that it is only through this deeper form of premise reflection that transformations occur. Arriving at such a badly needed change or perspective or course of action could make a significant difference for the entire information design process.
Collaborative Formative Reflection
Most technical communicators work within a team, so we need to consider how this kind of Formative Reflection could work within a collaborative context. One scenario has the entire team in a meeting room discussing a usability report or other form of review. At either a pre-determined point (say after the report has been fully discussed), or at a point where there is particular confusion and lack of consensus within the team, Formative Reflection is used as a technique to help the group come to conclusions and consensus. After the group writes, they share the results of their written reflection out loud. Another interesting approach would be to share the usability testing report with the entire team electronically before a meeting. Team members would do a Formative Reflection and post this writing piece electronically to the entire group. The team could then consider each individual member's Formative Reflection before the meeting, thereby making the team meeting more productive. Through this technique of reviewing the Formative Reflections of others, the benefits of reflection become magnified even further.
Breaking Out of the Straightjacket of Rules
As technical communicators working within contexts dominated by scientific and technical thinking, we may have a tendency to latch on to technical rules of information design development that become formulaic. If it worked once, it will work the same way the next time. If it is described and used by So and So Expert, we seek to imitate them exactly. We all need this kind of guidance for our work. However, viewing these practices as technical rules that predetermine our design can have dangerous results, particularly in instances when our design process faces problems and uncertainty. If we are not able to problem-solve creatively and fully comprehend the implications of usability test results, we could be drawn into creating flawed products. Formative Reflection is one technique that can help technical communicators address perplexity and mediate between the technical rules and universals that guide their practice and the requirements of the particular communication context. Only with this nimbleness of mind and willingness to go beyond the rules when necessary can we produce truly innovative and successful designs and products—Formative Reflection helps us accomplish this goal.
References for further inquiry
Boud, David. Productive Reflection at Work: Learning for Changing Organizations.
No comments:
Post a Comment