Monday, August 17, 2015

How Can We Have Good Assessment in College?

Assessment is too often a dirty word in education, and for good reason. It's usually bad, done to people, and force upon faculty. It often serves no other purpose than because it is is required by accreditation agencies. Below, I hope to make the beginnings of an argument for why it may very well be necessary for good student learning. 

Recently, Erik Gilbert, an Associate Dean and Professor of History at Arkansas State University, wrote an opinion column in the Chronicle of Higher Education called, "Does Assessment Make Colleges Better? Who Knows?" It's a very good question, one I've asked myself several times during my career; however, I've usually asked it in a different way: "HOW can assessment make my college (or program) better? How can I know?" It seems, part of the answer would have to be student participation -- but I'm getting ahead of myself. 
Gilbert's column in the Chronicle

Gilbert begins his piece with his own search for colleges for his sons. The features of any college worth considering to him and his sons did not take them to college assessments or their results -- not that a parent or prospective student would be able to readily find such results on many web sites. When I think about how my sons and I will pick colleges in the near future, we'll likely consider the same kinds of things Gilbert describes: faculty interests, students' abilities to work with faculty on projects, extracurricular amenities like rock climbing walls and rec centers. We likely won't look at any program assessment results from, say the writing program, or a major of interest. Then again, because I'm in this field, I know where many of the good writing programs are. I know their directors, and I've read their published results.

Invisible Good vs. Visible Bad Writing Assessment Still, Gilbert has a point. What good is assessment? Does it do any good for teachers or students? How would we know? Recently on the WPA-L listserv, Galen Leonhardy asked about the role of the writing assessment theorist, which got a number of responses. Rich Haswell, a long-time writing assessment researcher (now retired), responded with a funny answer that has a kernel of truth to it: 
The role of good writing-assessment theorists is to regard history, ideas, and findings and come to conclusions that are disregarded by everyone else. 
The role of bad writing-assessment theorists is to shape history, ideas and findings to fit the druthers of their bosses. 
While Haswell's cynically humorous response doesn't answer the question of whether writing assessment (or any assessment) is good or not, it does point to why educators often see so much bad assessment. The good writing assessment is disregarded or ignored by everyone. We never see it, or it's off everyone's radar. The bad assessments are just supplications to higher ups, and they are the ones most likely visible, even if they don't factor into many students' college application decisions. 

Yes, Haswell isn't meaning his answer to be an actual answer, and I know he'd say in reality good and bad assessment is not so cut and dry. We often must do assessments with which we do not agree because we are required to. What is good and bad assessment is contextual and based on the unique place, people, and purposes of any school. What is important to hear in Haswell's response is the kernel of truth, that good writing assessment is often ignored or disregarded because it is assessment. Is it a case of everyone, including well-meaning teachers, parents, and students, throwing out the good assessments with all the bad ones? Guilt by association? 

Are We Using the Wrong Criteria To Assess?  Gilbert sees another reason to ask the question about assessments' abilities to make colleges better, which may help us develop one answer for why good assessments are so often disregarded. He says near the end of his article: 
And most troubling of all is that the fundamental premise of assessment is that the problems we need to test for and try to fix are found in the classroom and the curriculum. So while we are agonizing about whether we need to change how we present the unit on cyclohexane because 45 percent of the students did not meet the learning outcome, budgets are being cut, students are working full-time jobs, and debt loads are growing.
So we spend money on assessment, which only exacerbates the problems of cost in higher ed, where money can often be hard to come by for good programs and scholarships. We focus too much on matters inside the classroom, when factors outside of it matter just as much or more to students' present and future success. For Gilbert, what makes good assessment, then, is assessment that uses external criteria for validating its decisions. His examples are of "changes in a college’s reputation, ranking, or employment prospects for its students." These are surely good things to know, things worth measuring in some way, things parents and students care about and would find to be good measures of value and worth in schools. Of course, they are not the only reasons to go to school. They do seem to assume that the existence of a college (and the reason to go to college) is to prepare students for their roles in a corporate Capitalist economy. 

Still, to carry Gilbert's argument a bit further, assessing for academic outcomes won't tell you whether students have employment prospects after graduation (we all need to work and earn money) since the external factors that control those prospects are mostly outside of the students' learning and higher education generally. Beyond the fluctuations of Capitalist labor markets controlled by political decisions and other things, how can, for instance, a history or philosophy department control the socio-cultural narratives and values in our society today that devalue such worthwhile degrees, that see them as worthless? Is it not important to have a good number of citizens who have thought and can think carefully about history and ethics in just about every position possible? Of course, it is. But what job is there for a philosophy major or a history major today? 

Thus, I think good assessment in higher ed doesn't have to placate to the whims of the Capitalist marketplace, or of society, or current cultural trends. I'm not saying that education ignores such things, but it must offer a corrective, a counter voice to society, to the Capitalist marketplace. If we don't, who will? So good assessment might provide students with experiences to participate in learning and its assessment, to participate in judgment, which means they participate in meaning-making. Put another way: good assessment might ask students to make judgments on learning and have those judgments count in the classroom in real ways. This kind of good assessment can make colleges better because the criterion we are validating our decisions by are students themselves. 

What's Good Writing Assessment? 
So let me focus on good writing assessment, since that is what I do, but I don't think my ideas are exclusive to writing assessment. Writing assessment should not be primarily about measurement. That is, I don't think educators should design assessments in order to simply measure student value, proficiency, or accomplishments. These things require academic standards and yardsticks that by their nature are problematic when used with/against diverse student populations. I've discussed in various places this phenomenon, so I won't go into it here (for my most thorough argument about this question, you can read Antiracist Writing Assessment Ecologies: Teaching and Assessing for a Socially Just Future). But beyond this problem, assessments that measure, that look to find out who meets levels of proficiency and who do not, usually make students feel poked at, like lab rats being tested. In these instances, assessment is usually done last, an add on to the class or lesson/unit. It is often in the way, or something to get out of the way, so that the real work of the class can continue. Students feel this, and feel that they are not trusted to learn without assessments -- tests. This kind of bad assessment constructs an adversarial relationship between teacher and students. It assumes students will do what they can not to learn, unless they are made accountable through tests. 

On the other hand, good writing assessment is an entire ecology that involves students in processes of inquiry that build agency and stake in their learning. In this kind of classroom, students don't need to be tested, instead they test themselves through their learning processes. Assessment becomes central to the course and its pedagogy, to what students are doing on a weekly basis. It becomes the main reason students are there -- and students can see and feel it. This kind of good assessment doesn't even need to produce grades or rankings or numbers. What it produces is student learning, questions, dialogue, exchanges, problems, and contingent solutions. Learning is always contingent and complex, so why do we think its measurement will be anything but contingent and complex? In a simplistic way, I'm taking the old adage that we learn most when we teach others, and modifying that adage: we learn most when we assess each other. But don't misunderstand me. When I say assessment, I don't mean grade. Grading, by its nature as a ranking mechanism, is usually bad, harmful (see Alfie Kohn for a good argument against grading).

Testing An Assessment
A simple way to test or validate an assessment in a classroom, for instance, might be to ask: what does this assessment do for students and their learning? Or How does this assessment engage students in learning? Better answers for this validation question will suggest the centrality of students in all learning/assessing processes, and a deep, compassionate trust in them to do the work asked, which ironically results in higher degrees of fairness in the assessment system (a discussion of fairness is for another day). For instance, some answers to this validation question might run along the following lines and should have evidence to support them: 
  • engages students in processes of inquiry and exchange of judgments based on the content of the course and the students' interests or personal connections to course content 
  • helps students build/construct authority or confidence in what they are learning and its usefulness to them now and in their foreseeable futures
  • constructs power and ownership of the classroom's assessment ecology, which includes power to affect directly their course grades 
I could name other things, but the idea is that students, their processes of learning and assessing, and the power and participation that they construct in assessment processes are central to the classroom. These kinds of assessments make colleges better because they make students more critical or critically conscious (in the Freirean sense, see Pedagogy of the Oppressed) of themselves, of their learning, of how they judge, and how that all that learning happens for them. This kind of assessment, one that stays away from grading and ranking, provides students with better, intrinsic motivation, rather than less effective extrinsic motivation (grades). I like to say to my students: "we'll try to substitute your usual striving to earn grades for a striving to learn about writing and judgment." 

What would this kind of classroom writing assessment look like? That's more complex and dependent on the school, students, teacher, constraints, etc. But I've offered a few ways to see how I've done it in my classroom in my recent book. I'll offer just this one bit of advice here. Good writing assessment gets students to help teachers do it in significant ways. It doesn't produce grades or numbers. It produces student agency, dialogues and exchanges that are about the learning, and intrinsic motivation to learn. 

And to do this kind of assessment, it typically requires that a course be rethought so that assessment comes first in its design, not last, not something to get out of the way, but something to dwell on. I use a grading contract in all my courses to determine course grades. This gets rid of all the other grades on individual papers, and opens up the classroom for more critical discussions of judgment. I design processes of assessment with my students that they do, and that we go over afterwards. I ask them to continually reflect each week on these processes. And we think a lot about the contract and what it asks of us. To see my class in detail, see "A Grade-less Writing Course that Focuses on Labor and Assessing." 

So how can we have good assessment in college? I think, it starts and ends with us (teachers and administrators) getting out of the way, and letting students lead, engage in it, and showing us how it's all done. 

Thursday, August 6, 2015

Using Lectio Divina as Assessment

Teachers of all stripes who assign peer review or peer assessments often complain that students don't know how to give helpful feedback, if they give much at all. Even when we structure feedback by providing rubrics or other heuristics to guide student assessors and what they offer their colleagues, student feedback is often less than helpful, or perceived as such by writers. To complicate matters further, many students just don't trust their peers for a variety of good reasons, (e.g., their peers aren't grading them, their peers aren't experts in writing or the topic of the class, etc.). I have found, however, that there are practices that students can do that provide helpful feedback, and that can encourage a deeper attention to that feedback by writers.

Good Feedback Starts with Good Reading
Every act of judgment must proceed from an act of reading or listening. One measure of the thoughtfulness and helpfulness of any judgment of a text is the quality of the practice of reading and listening that produced that judgment. So to ask students to make judgments, or provide feedback, on their colleagues' drafts, no matter the kinds of judgments, means that we should provide students with ways to read.

I don't mean that we help students develop a purpose for reading a colleague's text when offering feedback. Yes, this should be an important first step in developing a feedback activity with students. It helps determine the goal of the reading (on the part of the assessor) and the application of any feedback (on the part of the writer). Feedback might be geared toward generating and revising ideas, or reconsidering key dimensions of writing (see my last post on using dimensions vs. standards in writing), or arranging and structuring ideas (organization), or proofing. But these are not ways of reading. They are stances readers take when reading texts, not methods to do so. Good reading starts with good methods for reading, so good feedback really starts with good methods for reading.

One Method for Reading: Lectio Divina
Too often, especially in the college classroom, we simply assign reading or assign feedback activities. We ask students to read and give their colleagues feedback on their drafts. If we are thoughtful, we give them some guidelines for what we expect the product of their feedback to take (such as questions to answer, issues to discuss, and minimum word counts of the text of the feedback), but we don't always offer many methods for reading the text. We simply assume that students will know how to read their colleagues' drafts for the purposes of the feedback we want them to provide, that they have a repertoire of reading practices available to them. We assume they know how to read carefully. For instance, two very good and robust online technologies for peer review, Eli Review and My Reviewers, focus most of their attention on the feedback students provide -- that is, the form and content of the feedback. You can tell this by where the technologies place their attention, particularly in the data available to teachers and students. Neither can, for instance, keep track of students' reading time. How long are students spending reading a colleague's draft? Do they spend more time reading later drafts in the semester? Time can be an indicator of the quality of reading method.

We also assume that the most difficult part of any feedback activity is the articulation of the feedback itself, the ideas and opinions expressed and their expression in words. So if we spend any time on helping students with feedback methods, it's usually (in my experience) with the nature and form of the feedback. Do we want them to ask questions or respond to ideas or make only observations? Do we want them to spend some time saying where the good stuff is, then say where the things that need improving are? But we don't spend much time, if any, on how students actually read a draft.

What if students changed their focus in feedback activities to the labor of reading or the practice of reading? This is what I suggest.

One of the most ancient forms of careful, close, and meaningFUL reading practices is the practice of lectio divina, which means "divine reading." Originally in monastic and other Christian traditions, it was a practice of slow reading of short bits of scripture, with pauses and time to reflect  and meditate, allowing God's Word to touch the reader, or not. Typically, there are four parts or aspects to lectio divina:

  • lectio: reading or listening
  • meditatio: meditation or reflection
  • oratio: prayer
  • contemplatio: contemplation
Each part of the practice is saturated in silence in order to allow God's Word to come to the person, since often, it is said, that God speaks to people in a "still, small voice" (I Kings 19:12). There's a good description of this practice offered by St. Andrew's Abbey in Valerymo, CA, as well as a short description of it by the Order of the Carmelites, which offers two short video explanations of the practice as well. And for a discussion of it as an academic practice and not a spiritual one, Daniel Barbezat and Mirabai Bush's chapter 6 in Contemplative Practices in Higher Education is a good resource. I don't want to focus on the spiritual dimensions of this practice for obvious reasons, but I do think it is important to know that the practice was and still is a spiritual one, one that was meant to allow an experience with God without placing one's own purpose or intentions onto the text of the Bible. God's Word dictates to us meaning and next steps, not the other way around, goes the logic. This purposeless reading can be helpful to students in feedback activities, if translated correctly.

Reading without intention -- or rather, reading that allows intention to come from the text, not the reader -- which seems contrary to what I mentioned above about determining a purpose for feedback activities, is what makes lectio divina potentially powerful and useful to students as a method for reading to give feedback to peers. Why? Because if we let our judgments come from the intentions of the text, or at least our perceived intentions of the text, then students have a better chance at reading colleagues' drafts on their own terms, not placing too many expectations and values of their own onto it, and realizing that they have such expectations and values that automatically get placed onto texts when they read. This latter distinction is most useful and a good way to help students reflect later upon the feedback they receive from peers.

But lectio divina also offers a method for how to get students to experience diverse voices and ideas from colleagues, resisting the need or impulse to judge first, or to correct. This leaves only one purpose left that makes sense (in my view): to read for your colleague's sake. That is, I want to argue that if done right, lectio divina can give students a way to read with more compassion, from a stance of listening, not of judging, which ironically, I believe, is a more powerful and helpful stance for offering feedback. When we know we are being listen to carefully, we listen back carefully. When we think and feel that the other person just wants to hear himself talk or just wants to tell you what he thinks, then we are less inclined to listen carefully.

A Sample Feedback Activity
One way to use lectio divina as a method for students' readings of their peers' drafts is to give them the process for reading, making reading the first step in the feedback activity, giving it priority (not the feedback text itself), maybe even asking students to only read first without writing any feedback. Pausing between reading a peer's draft and writing feedback to the peer about it is critical. Remember, lectio divina required lots of pauses, silence, and reflection between steps or parts of it. This takes time, and so allow for the time, even if it is over a night or weekend. Here's one method that uses lectio divina that could be modified for use in a classroom, written as directions to students:

  • Read your colleague's entire paper quickly (do not mark or take any notes).
  • After you finish, pause for 2-3 minutes. Just let the paper's ideas sink in or drift in your head. Don't worry about order or what you want to say about it or to the writer. Just let the paper's impression sit with you. 
  • Take 3-4 minutes and write down your first impressions of the paper. What does this paper make you think about? Why?
  • Now, read the paper again more carefully, but this time, pause during your reading at the end of every page (so if the paper is 3 pages, you will pause a total of 3 times). 
  • When you pause at the end of each page, take 3-4 minutes and write your thoughts to the writer. This will be a short paragraph (maybe 100-150 words) in which you freewrite what you hear and see the writer saying on that page. Try to address these two questions each time: 
    • What is the main idea you hear the writer expressing on this page?
    • What are you expecting as a reader at this point for the writer to discuss? Why?
  • Repeat this reading, pausing, and writing until you finish the draft. 
  • Compile your paragraphs into one document, addressed to the writer. 

This process or method of reading can be modified or changed, but the important parts are the frequent pauses and writings that help the reader slow down and pay attention to the text. Notice also that the questions begin with observations. What does this reader hear or see on the page? When I give such instructions to my students, I make sure that we talk carefully about why we don't want to jump quickly to judgment, that we first want to make observations, and that often just doing that can be very helpful to a writer, if it is done in detailed ways. We also talk about how often time gives us answers or insights, that often we cannot and should not rush our thoughts or ideas, so the pauses are important. Just sit between papers for 30-60 seconds before you begin writing. It is amazing what 60 seconds can offer you.

Other versions of this practice can be even simpler. For instance, I've had students read a draft or a text in class, then we pause after each paragraph, and I asks students to write down one word or phrase that pops up into their heads at that moment. It might be a word that characterizes what they just heard/read, or a word from the text itself. We then quickly go around and just say our word or phrase aloud. No comments. No questions. We just listen to the bare words with lots of silence around them. Then we read another paragraph, and do it again. When we finish, I ask them to reflect upon their list of words, the text, the impression of other words they heard, etc. It's a pretty enlightening practice that uses the silence to call attention to words, which then lead to a discussion that can be judgment- or assessment-based.

Similar reading methods can be used in writing groups, when students are to get together and talk about their feedback to writers. In those activities, I usually focus students' reading on their assessment documents, the feedback as text. So it is the feedback texts or letters to writers that get the lectio divina treatment in writing groups. Since everyone in the group has read the draft begin discussed, it seems what's really at stake is how each reader read the text. Lectio divina offers a method for writing groups to focus on the text of feedback as an entry into the practices of reading by readers. The discussion becomes one about how we read and came to judgments, not which judgments are right about a text, or what to do next in the text, or what the writer should do in revisions. Those are questions the writer must decide, not readers. So discussions about feedback end up being ones about reading and reader assumptions that can deeply inform writers when they take home the feedback and begin processing next steps and revision.

No matter how you use lectio divina, it calls attention to the labor of reading. It doesn't let students take for granted the practice of reading, instead it helps students listen for the still quiet voice within their colleagues' drafts, and reflect back to writers that voice in compassionate ways, ways that say, first and foremost, "I am listening to you. What are you trying to tell me?"