Mr. James Fallows
Editor
U.S. News & World Report
2400 N Street NW
Washington, DC 20037

Dear Mr. Fallows:

      I appreciate that, as the new editor of U.S. News & World Report, you have much to do at this moment. However, it is precisely because you are the new editor that I write to you, personally.

      I emphasize you, because of your demonstrated willingness to examine journalism in the same way that journalism examines all other facets of society. And I say personally because my letter is for your consideration, and not a letter to the editor for publication.

      My timing also is related to the recent appearance of the annual U.S. News "America's Best Colleges" rankings. As the president of a university that is among the top-ranked universities, I hope I have the standing to persuade you that much about these rankings - particularly their specious formulas and spurious precision - is utterly misleading. I wish I could forego this letter since, after all, the rankings are only another newspaper story. Alas, alumni, foreign newspapers, and many others do not bring a sense of perspective to the matter.

      I am extremely skeptical that the quality of a university - any more than the quality of a magazine - can be measured statistically. However, even if it can, the producers of the U.S. News rankings remain far from discovering the method. Let me offer as prima facie evidence two great public universities: the University of Michigan-Ann Arbor and the University of California-Berkeley. These clearly are among the very best universities in America - one could make a strong argument for either in the top half-dozen. Yet, in the last three years, the U.S. News formula has assigned them ranks that lead many readers to infer that they are second rate: Michigan 21-24-24, and Berkeley 23-26-27.

      Such movement itself - while perhaps good for generating attention and sales - corrodes the credibility of these rankings and your magazine itself. Universities change very slowly - in many ways more slowly than even I would like. Yet, the people behind the U.S. News rankings lead readers to believe either that university quality pops up and down like politicians in polls, or that last year's rankings were wrong but this year's are right (until, of course, next year's prove them wrong). What else is one to make of Harvard's being #1 one year and #3 the next, or Northwestern's leaping in a single bound from #13 to #9? And it is not just this year. Could Johns Hopkins be the 22nd best national university two years ago, the 10th best last year, and the 15th best this year? Which is correct, that Columbia is #9 (two years ago), #15 (last year) or #11 (this year)?

      Knowing that universities - and, in most cases, the statistics they submit - change little from one year to the next, I can only conclude that what are changing are the formulas the magazine's number massagers employ. And, indeed, there is marked evidence of that this year.

      In the category "Faculty resources," even though few of us had significant changes in our faculty or student numbers, our class sizes, or our finances, the rankings' producers created a mad scramble in rank order, for example:

 

      One component of this category, "Student/faculty ratio," changed equally sharply, and not just in rank order but in what the magazine has presented as absolute numbers. Again, this is with very little change in our student or faculty counts:

 

      Then there is "Financial resources," where Stanford dropped from #6 to #9, Harvard from #5 to #7. Our resources did not fall; did other institutions' rise so sharply?

      I infer that, in each case, the formulas were simply changed, with notification to no one, not even your readers, who are left to assume that some schools have suddenly soared, others precipitously plummeted.

      One place where a change was made openly was, perhaps, the most openly absurd. This is the new category "Value added." I quote the magazine:

Researchers have long sought ways to measure the educational value added by individual colleges. We believe that we have created such an indicator. Developed in consultation with academic experts, it focuses on the difference between a school's predicted graduation rate - based upon the median or average SAT or ACT scores of its students and its educational expenditures per student - and its actual graduation rate.

      This passage is correct that such a measure has long been sought. However, like the Holy Grail, no one has found it, certainly not the "we" of this passage. The method employed here is, indeed, the apotheosis of the errors of the creators of these ratings: valid questions are answered with invalid formulas and numbers.

      Let me examine an example in "Value added": The California Institute of Technology offers a rigorous and demanding curriculum that undeniably adds great value to its students. Yet, Caltech is crucified for having a "predicted" graduation rate of 99% and an actual graduation rate of 85%. Did it ever occur to the people who created this "measure" that many students do not graduate from Caltech precisely because they find Caltech too rigorous and demanding - that is, adding too much value - for them? Caltech could easily meet the "predicted" graduation rate of 99% by offering a cream-puff curriculum and automatic A's. Would that be adding value? How can the people who came up with this formula defend graduation rate as a measure of value added? And even if they could, precisely how do they manage to combine test scores and "education expenditures" - itself a suspect statistic - to predict a graduation rate?

      Were U.S. News, under your leadership, to walk away from these misleading rankings, it would be a powerful display of common sense. I fear, however, that these rankings and their byproducts have become too attention-catching for that to happen.

      Could there not, though, at least be a move toward greater honesty with, and service to, your readers by moving away from the false precision? Could you not do away with rank ordering and overall scores, thus admitting that the method is not nearly that precise and that the difference between #1 and #2 - indeed, between #1 and #10 - may be statistically insignificant? Could you not, instead of tinkering to "perfect" the weightings and formulas, question the basic premise? Could you not admit that quality may not be truly quantifiable, and that some of the data you use are not even truly available (e.g., many high schools do not report whether their graduates are in the top 10% of their class)?

      Parents are confused and looking for guidance on the best choice for their particular child and the best investment of their hard-earned money. Your demonstrated record gives me hope that you can begin to lead the way away from football-ranking mentality and toward helping to inform, rather than mislead, your readers.

                                              Sincerely,

                                              Gerhard Casper