Wrapping up Assessment & Evaluation
This semester, I had the privilege of teaching a graduate course in Assessment, Evaluation and Criteria of quality for the second time. The learners were all art education MAT students. Early in the semester, we established a need on their part for a heavy integration of research skills into the course. From my perspective, this is a perfect match, since very similar skills are required for assessment and evaluation, as are required for pure research.
The only complaint I have about the course is that I find it very difficult to include both assessment and evaluation into only one course. On the assessment part, students expect to spend a lot of time discussing classroom assessment and developing skills in the classroom. It takes a lot of time and discussion to help students develop a vocabularly to discuss assessment in the terms that educational researchers and assessment-savvy educators use. Then suddenly, bam-- let's switch to program evaluation. Yes, in one sense it's a micro to macro perspective, but I always feel like the evaluation portion is rushed.
The best part of the class is that it was very hands-on. Most of the students currently teach, so the project for formative assessment was for them to conduct action research whereby they thoughtfully integrated a formative assessment strategy into their classroom practice and reflected on the results and noted how it would impact their future practice. For the students who are not teachers, they did an inquiry into the assessment practices of someone who is currently teaching through interviews and observation.
Instructors should have aha moments too. My aha moment over the course of the semester and in tandem with my own work which is increasingly heavily assessment-oriented was that feedback is at the center of instructional practice. While some instructors understand this intuitively, others are not aware of how crucial it is. One of the students in the class, who teaches in an informal setting (thus making assessment that much more challenging) simply changed up the introduction to the lesson in a way that required the kids she was working with to reflect on what they were creating. Simply adding a question can radically change an entire lesson and the way someone else is thinking about their own work. That's powerful.
Crowdsourcing: intensifying a “culture of assessment”
I understand why the call to create a "culture of assessment" has spread like wildfire in the title of workshops and in the rhetoric of the outcomes assessment movement. Just look here, and here, and here. I understand the concept is originating in the need for assessment in the process of considering the effectiveness of any assignments, courses, programs, and so on. But why the presumption that we don't have an existing "culture of assessment?"
Isn't what distinguishes the educational experience, or what makes it an educational experience is formal feedback and review from an expert/instructor and/or peers? And while I was drafting the beginnings of this lame and derivative discussion of what constitutes a culture of assessment, my brother emails me this idea:
I was thinking that it would be really neat to bring self-education and adult education into the 21st century a bit... to have a website which lays out an entire secondary and post-secondary education, delineated by courses, similar to high school and college work. So you have the syllabi, maybe videotaped lectures, problems, assignments, and tools for working through the material with others that have done it before in addition to volunteers that want to stick around and help with it. I think that the format for working through the material is something that will be evolving very rapidly, with people like you to blame for the innovations.
I think that its value would still be relatively small without some rigorous or at least semi-rigorous assessment. So, I'm thinking, you sit down and do a test... maybe it's too hard to check if someone is cheating. That's one problem to look at. But I think it might be possible to have others (multiple people) go through and correct/score the quiz/test/assignment. This is more straightforward, I think, with science/math, where it's just a matter of getting the correct answers (though there's the technical problem of how do you load it onto the website - do you require scanning it? faxing it in?). You have multiple people go through and score the test, and then you weight the scores depending on how reliable the scorer is. So the scorers themselves are scored.
Yes, the scorers would have to initially be scored by experts - perhaps real professors or students, either volunteer or working under a grant. The point of crowdsourcing the assessment is to allow the site to scale to serve millions without the need to add thousands or tens of thousands of paid experts.
First, I'm flattered he thinks I would be to blame for innovation. He's even considered inter- and intra- rater reliability in the model. OpenCourseWare movement, I would like to introduce you to Crowdsourced assessment and then ask the question, why will college still cost $50k/year? Content + assessment will not be enough to create a holistic college-level education for the average user. What it's missing behind it is the community of practice that accepts it as acceptable preparation. In other words, we still need a sponsor, a reputable one, to own the process and assure its rigor. But stepping back from the logistics of the whole process presented, and just on the assessment piece-- why not some degree of crowdsourced assessment for the existing college classroom? We do it for screening for plagiarism (well, kind of, if we consider the artifacts of each student to be the "crowd").
And it would tie in nicely with Downes' Open Source Assessment model. State the problem or have students state the problem. Have a number of raters assess the success of the solution. People will definitely learn this way...but can they be credentialed (and should we care if they can't ? )?
Assessment in the wiki world
Wow, now that I have my blog home all set up again, I'm so inspired to comment on the massive comments in my aggregator. Through a chain of reading, I see Nate talking about Clarence's post, which discusses the concept of how students should be assessed in a world where the classroom walls can extend globally and where collaboration can be made transparent through wikis and other web 2.0 tools.
In the original post to which they refer, the author relays an exchange that highlights a huge issue not just with the web 2.0 world of assessment, but with group assessment in general:
Jeff Utecht: How would you assess a student who changed a single word?
Ryan Bretag: Think about contributing one word from a poetry standpoint, how critical is one word? Writing in a hypertext society makes that one word critical.- From the Strength of Weak Ties.
If a group of students is working together and one student serves as the student who conceptualizes an idea, and the other students work diligently to carry out the idea at the direction of the first student, did the self-selected/group-elected leader contribute more? And now, when collaboration takes place through a more transparent record, for example, if you use a wiki, how do you ascertain individual student's contributions. One of the references on The Strength of Weak Ties is to a wiki that uses a framework of significant contribution versus constructive modification. The creator of that rubric, David Kuropatwa noted previously on his own blog the distinction between these two varieties of contribution and the special challenge of "constructive modification" as a critical thinking process is addressed. In determining assessment strategies, one could also do as Ben Wilkoff did and turn it over to the students to discuss.
I encounter more often the issue raised by another blogger who noted failure to integrate wikis in a way that led to student creation of content.  Even the earlier example of the math wiki is very structured, with questions I assume were determined by the teacher for students to answer.
Not in these posts and discussions but in society at large, I see a lot of confusion about the purpose of a wiki. And for that matter, a lot of suspicion about externalized collaboration in education. Though a wiki is in written form, it is, for me, at least, an externalization of thought process and collaboration. At its most elaborate (for example wikipedia), it can be a valuable resource, but most of the time it will reflect information that is targeted for a specific community. In some cases, the information gathered might have outlived its purpose (not unlike, by the way, many websites!). The wiki exists not to document forever and ever, but to provide an opportunity for collaboration as those collaborating make connections in their own minds and developed their own questions, ideas, and analyses. That too is the power of the wiki, that it can change and grow as its purpose and function expand. Wikis offer an opportunity to externalize the work of a collective mind, which directly contradicts the typical education model which requires us to assess each student purely for their individual contribution (most of the time). And there are a lot of students and teachers experimenting with how to make it work.
