Crowdsourcing: intensifying a “culture of assessment”

I understand why the call to create a "culture of assessment" has spread like wildfire in the title of workshops and in the rhetoric of the outcomes assessment movement. Just look here, and here, and here. I understand the concept is originating in the need for assessment in the process of considering the effectiveness of any assignments, courses, programs, and so on. But why the presumption that we don't have an existing "culture of assessment?"

Isn't what distinguishes the educational experience, or what makes it an educational experience is formal feedback and review from an expert/instructor and/or peers? And while I was drafting the beginnings of this lame and derivative discussion of what constitutes a culture of assessment, my brother emails me this idea:

I was thinking that it would be really neat to bring self-education and adult education into the 21st century a bit... to have a website which lays out an entire secondary and post-secondary education, delineated by courses, similar to high school and college work. So you have the syllabi, maybe videotaped lectures, problems, assignments, and tools for working through the material with others that have done it before in addition to volunteers that want to stick around and help with it. I think that the format for working through the material is something that will be evolving very rapidly, with people like you to blame for the innovations.

I think that its value would still be relatively small without some rigorous or at least semi-rigorous assessment. So, I'm thinking, you sit down and do a test... maybe it's too hard to check if someone is cheating. That's one problem to look at. But I think it might be possible to have others (multiple people) go through and correct/score the quiz/test/assignment. This is more straightforward, I think, with science/math, where it's just a matter of getting the correct answers (though there's the technical problem of how do you load it onto the website - do you require scanning it? faxing it in?). You have multiple people go through and score the test, and then you weight the scores depending on how reliable the scorer is. So the scorers themselves are scored.

Yes, the scorers would have to initially be scored by experts - perhaps real professors or students, either volunteer or working under a grant. The point of crowdsourcing the assessment is to allow the site to scale to serve millions without the need to add thousands or tens of thousands of paid experts.

First, I'm flattered he thinks I would be to blame for innovation. He's even considered inter- and intra- rater reliability in the model. OpenCourseWare movement, I would like to introduce you to Crowdsourced assessment and then ask the question, why will college still cost $50k/year? Content + assessment will not be enough to create a holistic college-level education for the average user. What it's missing behind it is the community of practice that accepts it as acceptable preparation. In other words, we still need a sponsor, a reputable one, to own the process and assure its rigor. But stepping back from the logistics of the whole process presented, and just on the assessment piece-- why not some degree of crowdsourced assessment for the existing college classroom? We do it for screening for plagiarism (well, kind of, if we consider the artifacts of each student to be the "crowd").

And it would tie in nicely with Downes' Open Source Assessment model. State the problem or have students state the problem. Have a number of raters assess the success of the solution. People will definitely learn this way...but can they be credentialed (and should we care if they can't ? )?