The Challenge of Organizational Knowledge

The New Opportunity

Faculty-Driven Assessment

Authentic Assessment

Making Achievement Visible

Our Mission and Vision

Who We Are

Seminars and Events

Our Clients

eLumen Achievement: A Summary of the Software

How eLumen Achievement Models the Essential Process

Consulting Services

The Collaborative

Contact Us | Home

Making Achievement Visible

Angela closes her book and thinks new thoughts. Until now she had understood interpretation as something that academic disciplines add to ordinary, uninterpreted experience. Today, however, identifying with the persons vividly described in this historical novel, she nevertheless realizes that life looked very different viewed from within the ordinary life of their community. She recalls details of her own community, as though they were also being described in a book. Explanations learned in last term’s anthropology course apply to them as well! It is as though she had stepped through a mirror, turned around, and was now looking back, past herself, at her own surroundings. She sees that the “common sense” of a community, whether the novel’s or hers, is not neutral at all and is itself already an interpretation. She resolves to consciously reconsider some of her own long-held assumptions. Her perspective on herself and the world has shifted again, irrevocably, and she returns to reading with a renewed sense of quiet excitement.

As Angela reads, her professors and others meet to reluctantly discuss the dean’s last memo on assessment of student learning outcomes. The question posed is how to provide numerical data showing that the college’s courses are preparing students in the skills that business and industry leaders want for employees. The meeting is contentious, and the decision is made to reply that it is not at all clear how this is to be done. The faculty know that the dean, given this response, is likely to recommend the purchase of a standardized test to measure “critical thinking” that, they hope, will be given to students outside of class time.

Of course, the sharp contrast between Angela’s interior learning and the faculty and dean’s discussion about assessment is hypothetical and overdrawn. Yet I believe it reflects what is now occurring across the country. Throughout the 1990’s there have been growing expectations from regional accreditation agencies that colleges and universities produce data on actual student learning outcomes and use that data to make decisions for improving learning. Business and industry leaders, often communicating through boards, committees, or professional associations, expect curricular changes to incorporate the skills they are hoping to see in graduates. Legislatures devise accountability indicators for public higher education. In response to all these, educational administrators have been trying to raise the profile of outcomes assessment. Some faculty have eagerly accepted this, but most have not. One of my colleagues said recently that, for faculty, assessment of student learning outcomes is “the elephant in the room that no one wishes to talk about.”

Viewed nationally, this large project has been going on long enough that we should be able to examine what works well and what does not. But what meaning should “well” have? With many higher education institutions seeking only easily implemented solutions to satisfy accreditation teams or federal grant requirements, there has been little reason to substantively discuss this question. In contrast, I propose two criteria by which we may re-assess assessment. The first is the extent to which assessment efforts genuinely respond to the new challenge that higher education is facing -- to answer the question “how well prepared are students for the world they are entering?” empirically, with data that has a clarity, specificity, and objectivity to which we are unaccustomed. The second is the extent to which they acknowledge the breadth, depth, richness, complexity, and variety of academic environments. With these in mind, it is clear that some approaches to assessment are dead-ends. Others are distractions. A few may contribute to re-vitalizing post-secondary education.

Many institutions have incorrectly assumed that student learning outcomes are attributes of the organization (e.g., course, program, or college). The questions have either been course-focused (what are the outcomes of this course? what difference does this course make? what will all students who take this course, or set of courses, know or be able to do?) or, using standardized tests, program- or college-focused (what scores do our students achieve?) This approach places all of the accountability on the faculty who, it is assumed, can make learning happen. Although there is a problem with grades – they are increasingly recognized to be lacking sufficient clarity and specificity -- they are right in being data that aggregates on a student record. There is only one sense in which the phrase “course outcomes” makes sense: as the specific set of learning opportunities that a teacher offers students in a given course. The “enacted curriculum” – the actual achievement of those outcomes – depends, like grades, on each student’s efforts and abilities and should be seen as attributes of individual students. Approaches to assessment that assume that student achievements are attributes of an organization are dead ends, because they cannot handle the actuality and specificity of individual variation. They would be better seen as demonstrated achievements that trace personal, professional and intellectual development.

By emphasizing certain learning outcomes and disregarding others, many assessment efforts have been distracted by becoming unnecessarily entangled in disagreements between advocates of workforce preparation and advocates of traditional academics. How we were led into this can be explained if we examine four distinct types of learning. Understanding -- a student’s demonstration of what he or she knows and understands within a specific context (discipline or profession) -- is best assessed by examination or student paper. Performance -- a student’s demonstration of what he or she can do that is specific to a context (discipline or profession) -- is best assessed by student demonstration according to faculty standards. Capability -- a student’s demonstration of what he or she can do that can be carried from one context to another -- is also best assessed by student demonstration according to faculty standards. The difference is not in the student achievement (which is still a contextual performance) but in the process by which faculty from different contexts have broadly agreed to assessment standards for any case in point of a specific capability. Perspective -- a student’s demonstration of what he or she knows and understands that can be carried from one context to another -- is surely the most difficult to assess. Yet this too could be assessed by a faculty member, if assessment characteristics were broadly agreed to and if the college were a place within which a student (like Angela) has reason to choose to come forward and demonstrate what she has learned.

In order to balance the four types, a larger overview is needed, for few faculty are concerned with more than two. Traditional academics emphasize understanding and perspective. Those whose primary allegiance is to their discipline or profession focus on understanding and performance. Those committed to broad liberal education emphasize perspective and capability. Proponents for preparation for employment emphasize performance and capability. One of the political problems for assessment in the 1990’s was that, by responding directly and only to external voices, institutions addressed only the last pairing, thereby excluding all or some of everyone else’s concerns.

Many assessment efforts have been caught in an either/or decision, either creating data for institutional purposes that has little or no meaning for students or facilitating student evidence that provides little or no data for the institution. Both are needed and can be simultaneously achieved if a college or university chooses to organize its outcomes data carefully enough. If done well, data per student both can be aggregated in whatever way the institution decides and can create a document that communicates a student’s demonstrated learning. If a college uses student portfolios, it can help organize their contents without limiting the communication of results to only the types of outcomes that can be shown there.

By choosing not to develop internal standards and processes, many institutions are unwittingly choosing to outsource assessment to the for-profit testing industry. The long-term effects of this are unknown. This too will eventually be a dead end, because the numerical data generated by standardized tests fails both criteria. It does not provide either the needed clarity or specificity. Nor can it handle the richness and complexity of higher education. To do this well, we will need to own assessment as a crucial function too integral to our work to be given to anyone else.

Increasingly assessment efforts are being hindered by the organizational incapacity to document student learning outcomes. Small groups of faculty have produced great examples of student learning and assessment within their classrooms, but the work has been manual and laborious and is entirely dependent on individual commitment and energy. Administrators need to see that they are responsible for maintaining the institution’s reliance on administrative systems that make documentation of clear and specific learning outcomes impossible. Choosing to work together to create a new type of administrative system that makes such documentation easy and that permits a college faculty to make its chosen outcomes and standards visible appears now to be a necessity.

Perhaps we should view this last decade as one of necessary experimentation in assessment of student learning outcomes. It is beginning to be clear that higher education must itself meet the new challenge, choosing the best of what has been developed and letting go of some approaches we have tried. Once we are able, as organizations, to acknowledge what Angela learned, we will know we have succeeded.

Dr. David A. Shupe is System Director for Academic Accountability for Minnesota State Colleges and Universities.

Assessment Update (September-October 2001), pp. 6-7.