Casual Empiricism in Law School Grades

-- By TaylorKelson - 10 Mar 2017

Performance

Good grades don’t make good lawyers. There is little reason to believe that law school grades are a meaningful measure of the skills and characteristics that result in aptitude or satisfaction in the practice. When I first approached this topic, I did so under the assumption that measuring such skills and characteristics was the purpose of the grading system. Thus, the reason the grading system wasn’t very good at that must be because the graders (professors) and their supervisors (the law school) weren’t trying very hard to build a good metric, or weren’t trying very hard to get their sums right, or both. I still believe that they’re not trying very hard but I see now that they aren’t going to start, because that isn’t the point.

Grades are deployed for the sake of performative stratification of law students. Certain employers find value in hiring (so they can advertise that they have hired) the “top” students from Columbia Law School. With 20% of the USNWR ranking metric based on “Placement Success,” the administration sees itself in no position to imperil that value.

Prediction

There is nothing wrong with grading as a concept. I would prefer that the military continue to evaluate its pilot trainees on their mastery of relevant knowledge and skills before giving them control over jet aircraft, and that it use an appropriate grading system to quantify those evaluations. The military has a strong vested interest in those grades reflecting objective reality about who is likely to be a competent pilot, and it will design its evaluations accordingly. The problem is that law students aren’t pilots. The law school creates a valuable good when it identifies the “top” students with academic honors. But when it comes to the empirical utility of those rankings, there isn’t much on the line, so the lack of investment in developing an empirically useful metric should come as no surprise. Tellingly, the curve is structured such that the rest of the class can all say that they are approximately at the median. There is no sense in destroying value by publicly outing the bottom students, because they’re not going to crash any airplanes. It is almost self-evident that law school exams are not a direct measure of good lawyering. Ability to produce generic, topic-level analysis within a few hours, with extensive notes and study time but with no research resources or opportunity to incorporate feedback, is not a useful job skill. The problem, then, isn’t that we can’t learn much about a lawyer-in-training from her transcript. The problem is that everyone, including the law school itself, pretends we can.

Law school grades do predict some things. For example, they predict career earnings and other marks of “career success,” such as partnership at major firms, with moderate statistical significance. (Causation doesn’t imply causation, of course—only a double blind controlled study would demonstrate how much of that effect is due to the actual predictive value of grades and how much is due to self-fulfilling prophecy.) However, the little research that does exist suggests that traditional measures of career success among lawyers have nearly zero, and perhaps even negative, correlation with happiness. Good grades don’t predict happy lawyers so much as they predict rich lawyers, and it is not clear how one would even measure whether they predict good lawyers.

The Fire Has Gone Out

In my experience so far as a law student, the predictive uselessness of grades beyond that of a self-fulfilling prophecy is received community knowledge. It is commonplace to say “you’re going to get a great job because you did well on your exams,” but it would sound quite odd to say “you’re going to be a great lawyer because you did well on your exams.” The Wiki from 2009-2010 reveals that Columbia Law students used to be mad about this. In my experience at Columbia so far, that anger has largely subsided. Grades are a convenient punching bag sometimes, but no one seems to mind much on the whole, and the message that “B’s get jobs” prevails.

This sentiment is true today, but it was also true then. There were 8 unemployed Columbia Law graduates in the Class of 2015, and 10 in the class of 2010. This was not the trend nationally, of course. Only 55% of the national law school Class of 2011 were employed. But even through the worst of that legal job market, Columbia Law graduates never particularly suffered. One explanation for those years on the Wiki is the sentiment that “grades are bad because they might hurt me.” But the commentary I read seems deeper than that. It seems that the climate surrounding those students motivated them to examine the institution of grades more skeptically than my peers and I tend to. That fire has gone out, and I think we should light it again.

The Path Forward

There are two potential paths beyond the status quo. The first is to follow several of our peer institutions in dispensing with the charade and instituting a true pass/fail grading system. This change may not alter much the marketable good that is the top of the class. Anecdotes from Yale law students, for example, suggest designation of the top simply shifts to measurement by recommendations from the most prestigious professors, participation in the most selective extracurricular activities, etc. But such measures may actually reflect more meaningful characteristics like networking and persistence, and this path at least has the virtue of dispensing with the officially sanctioned hocus pocus. The second path is to embrace the grading system by attempting to reform it to predict something useful. It is not obvious what that something is, or how it should be measured. It is also not obvious that this goal is possible—it may be that the social forces which have led us to Columbia plus the admissions selection process have resulted in a class about which no more fine-grained prediction is possible at this stage. But whatever came of such an attempt, at least we would be trying.

• #Set ALLOWTOPICVIEW = TWikiAdminGroup, TaylorKelson
• #Set DENYTOPICVIEW = TWikiGuest

Casual Empiricism

The Fire Has Gone Out

In reading old material on the Wiki, I was struck by the urgency and zeal of the debate about grading in 2009-2010. I knew in a general sense that debates about and changes to grading systems were happening in top law schools in those post-crash years, and I knew that Columbia had opted to retain a traditional grading system (the top-ranked law school to do so), but I didn’t realize how passionate and personal the debate had been. In my experience at Columbia so far, that debate has largely subsided. The grading system is a convenient punching bag sometimes, but no one seems to mind much on the whole, and the message that “B’s get jobs” prevails. Based on the empirical data that we do have, this sentiment is true today, but it was also true then. The American Bar Association reports that 8 of the 413 Columbia Law graduates in the Class of 2015 were unemployed nine months after graduation. In the Class of 2010, 10 of 430 Columbia Law graduates were unemployed by the same metric. This was not the trend nationally, of course. Only 55% of the nationwide law school Class of 2011 were employed. But even through the worst of that legal job market, Columbia Law graduates never particularly suffered. It is possible, then, the skepticism about Columbia’s grading system that I see from those years on the Wiki was overreaction to an employment climate which turned out not to be much of a problem, at least for Columbians. But the commentary I read seems deeper than “grades are bad because they might hurt me.” It seems instead that the discourse surrounding those students motivated them to examine the institution of grades more skeptically than my peers and I tend to. The fire has gone out, and I think we should light it again.

What Are We Measuring?

It is almost self-evident that law school exam grades are not a direct measure of anything useful. Ability to produce generic, topic-level analysis within a few hours, with extensive notes and study time but with no research resources or opportunity to incorporate feedback, is not a useful job skill. The law school must, then, be asserting that they are a proxy measurement for characteristics we care about: effort, commitment, rhetorical ability, artful drafting, research skills, etc. They offer no evidence to support this claim. We should demand it.

It’s not that law school grades don’t predict anything. They do predict career earnings and other marks of “career success,” such as partnership at major firms, with moderate statistical significance. First, the answers provided by such research are far from complete. Given law schools’ extensive internal reliance on their grading systems, they should be at pains to empirically demonstrate that the systems represent the best metrics they can muster to predict whatever it is that they are purporting to predict. Second, and far more seriously in my view, this type of research asks the wrong question. “Career success” is not self-defining. Financial earnings are not the complete definition of career success for almost anyone; for some, earnings are completely irrelevant as a measure of success once they surpass some basic livable income. For example, career satisfaction, personal happiness, and health may be much more meaningful metrics for at least some new lawyers. The little research that does exist suggests that traditional measures of career success among lawyers have nearly zero, and perhaps even negative, correlation with happiness. It should be the responsibility of the institution assigning grades to further research like this and adjust their practices in response to the findings.

Proposals

The law school should publish annual reports of anonymized, aggregate grade distributions. It should undertake its own analysis in order to convince itself and its students that grades do not capture characteristics thought to be irrelevant to the exercise, such as race or gender after control for other measures of aptitude and prior educational achievement.

Further, the law school should actively undertake experiments with the grading methodology by grading with experimental systems on a test basis, side-by-side with the traditional system. It should examine the consequences of a less discretized scale, of more emphasis on subjective feedback, and of giving more transcript weight to research, writing, and advocacy activities, to name a few. It should examine the efficacy of these and other alternative metrics, both on promoting learning in law school and predicting success afterwards.

This confuses methods of grading with methods of evaluation. You want to be more precise about that, because grading isn't very interesting or important, but how learning is evaluated has very important effects on learning indeed.

It doesn't matter in context, however, because "the law school" won't experiment in any way at all. Individual teachers have power to experiment, which by and large they don't use.

Conclusion