THE PITFALLS OF COMPREHENSIVE ASSESSMENT

Jack Citrin
Professor of Political Science
University of California at Berkeley
October 17, 200

[Presented to the UC Board of Regents, Laurel Heights, San Francisco,
October 17, 2001]

I want to thank Regent Johnson for giving a rank-and-file faculty member the
rare privilege of speaking to you today. Although these comments are my own,
the point of view they express are, I assure you, widely shared on every
campus.

Admissions is the life blood of universities. Since there are more
applicants than available slots, it also is a zero-sum game affecting the
life chances of young people. At the highly selective University of
California, the admissions process should be founded on the following
principles:

I. The primacy of academic excellence.

II. Fairness, meaning that all applicants are judged by the same rigorous
standards.

III. Openness, meaning that applicants and their parents know the criteria
of admissions and understand how they are applied.

BOARS proposes the Regents to approve new policies that eliminate the
requirement that at least half of each freshman class be admitted on the
basis of academic achievement alone and to admits students on the basis of a
comprehensive, holistic assessment of all applicants.

These proposals are defective in terms of each of the principles just
stated. If adopted, they will lower the aggregate academic quality of
students at the University.

They will increase subjectivity and bias in the selection process. And they
will leave applicants and their parents confused and uncertain about what is
expected of them, further reducing the transparency and legitimacy of UC's
admissions processes.

BOARS' proposal is modeled on what Berkeley has been doing for four years.
Yet, remarkably, there has been no critical assessment of that process
bearing on the decision to eliminate rather than maintain or enlarge the
academic tier of admissions.

RATIONALE

According to data provided by UCOP, the current policies imposed by the
Regents and the electorate have improved the academic quality of the student
body at every campus without a reduction in the overall number of minority
students admitted. The recent freshman classes at most campuses are more
homogeneous in their academic ability than in the past, which is both
pedagogically and psychologically a superior outcome.

Yet BOARS wants to reverse this trend because of disappointment about the
social composition of the students admitted to at least some of the
campuses. The objective of comprehensive review is to effect a different
racial, ethnic, and socioeconomic distribution of students at UC.

It is proposed in full knowledge that there will be some diminution of
academic quality as a result. Indeed, if not, why propose the removal of the
academic tier. The 1999 report of the UC Davis Admissions Committee is quite
forthright in stating:

"The inescapable effect of such a policy (increasing the role of contextual,
non-academic factors) is to reduce the quality of the admitted class."

GUIDING PRINCIPLES OF COMPREHENSIVE REVIEW

According to BOARS, comprehensive review is guided by the application of
these principles:

1. the ongoing "refinement and redefinition of the concept of merit to make
it more inclusive,"

2. "evaluating all achievements and performances in the context of the
opportunities and experiences available to the applicant;"

3. "the search for applicants who can contribute to the life of the campus
and emerge as state and national leaders"

The principles articulated above are flabby, vague, and impossible to apply
in a rigorous fashion. For example, I have no idea of what a refined concept
of merit for a political science or electrical engineering major might be?
Evaluating in context means that a 300 point difference in the SAT scores of
a graduate of Poway High School in San Diego and Dorsey High School in Los
Angeles is dismissed as irrelevant when we are assessing achievement or
likelihood of future academic success. In a word, context preferences
substitute as intended proxies for ethnic preferences; quality depends on
who you are not what you are. I believe that poverty should not be a
handicap to attending UC and support generous student aid from all sources.
But I do not believe that such background factors should be a positive
advantage, outweighing factors more directly linked to academic success.
Moreover, putting someone in a situation where the majors they can take are
limited by their academic preparation or where they cannot compete
effectively does no one much good.

Comprehensive review is implemented by the decisions of readers who have no
experience teaching at the University and who lack valuable information such
as senior year grades or personal references from teachers, counselors or
ministers.

Whatever the good intentions of these readers, their instructions make it
inevitable that over time reliance on objective indicators of merit will be
set aside in favor of achieving political targets and goals, much as the
erosion of the limited Bakke decision led to de facto quotas.

The desire to identify future leaders and men and women of fine character is
admirable. Of course there is no evidence that we must sacrifice academic
quality to achieve this goal or that we know how to do so. A May 1999 note
to Professor Moore from a senior admissions officer stated that Berkeley's
readers expressed concern about "the ability to discern who might become a
future state or national letter," about what the AEPE committee had in mind
when it spoke of "alternative metrics of achievement," and about how using
the essay to discern the applicant's "ability to articulate thoughtfully and
creatively their own life experiences" would work against non-native
speakers. So much for the clarity of the criteria of judgment in
comprehensive review.

Decades of research comparing the validity of actuarial (quantititative and
formulaic) versus clinical (qualitative) consistently show the superiority
of the former. Comprehensive review opens the door to subjective biases of
both the individual and collective or what Berkeley calls normed variety.

The fact that Stanford, Harvard, Yale, Brown embrace comprehensive review is
hardly relevant. These institutions have different missions from UC and
operate in a different legal environment. More significantly, they are
choose among a far narrower band of students at a higher overall level of
academic achievement. They have more information about each applicant, have
fewer to evaluate, and devote more time to each one. And who knows if they
are doing the right thing.

STUDIES OF COMPREHENSIVE REVIEW

What is most astonishing and indeed disturbing about BOARS proposal is that
those recommending changes in the admissions procedures have not done any
critical analysis of the likely impact of this and alternative procedures.

In their letter to President Atkinson and Professor Cowan requesting
permission to abandon the academic tier, Chancellor Berdahl and Professor
Moore conclude "we would be happy to provide a comprehensive review of the
new process after five years experience." This strikes me as putting things
backward. Berkeley has had ample time to conduct such an assessment of the
version of comprehensive review it has been employing for the past four
years. Yet despite the ease and relevance of a study designed to compare the
academic performance and campus contributions of tier 1 and tier 2 admits
and the capacity of simulating the impact of moving to a Unitary Score, the
minutes of Berkeley's AEPE committee repeatedly state that an assessment of
the post-1997 admissions process is "in abeyance." I choose not to speculate
about why. But surely the burden of proof should be on those who seek change
that will inescapably diminish the role of academic excellence in
admissions.

CONCLUSIONS

In the light of the above, I respectfully recommend that the Regents refuse
permission to abandon the academic tier that functions as a mandatory
insurance policy safeguarding the academic quality of the University and
that you table proposals to introduce the unitary mode of assessment and
other redefinitions of academic merit until you receive a comprehensive
study of the impact of such reforms on grades, graduation rates and the
like. This study should be conducted by an independent panel composed of UC
faculty and outside experts. Thank you again for your attention.