Compared to What?

It is not uncommon for a request for a letter of reference -- for a faculty position or as part of an evaluation for tenure and/or promotion -- to include a specific appeal for a comparison of the candidate with others

Here are some examples of how this request is worded in instructions to external letter-writers:

Standard model/general: How does Dr. X compare with other individuals in his/her field at similar career stages?

Standard model/more specific: How does Dr. X compare in terms of research achievements, standing, and potential with other individuals in his/her field at similar career stages?

Advanced model/domestic: How does Dr. X compare with the best people in the US at similar stages in their careers?

Advanced model/global: How does Dr. X compare with the best people in the world at similar stages in their careers?

Advanced model/comprehensive: How does Dr. X compare with leaders in his/her field with respect to both current and potential future standing in the field, nationally and internationally?

I have worded these as questions, but the same thing may also be worded as a statement: "It would be very helpful to our evaluation if you would compare Dr. X with ..."

A common variant requested of current or former advisors of an applicant for a faculty position is for the advisor to compare the applicant with other advisees.

I don't have any particular compelling reason to argue against this practice, but, when writing letters, I ignore this request.

Many people comply with the request, and that is fine, but as a reader of many letters containing comparisons, I can say that I rarely find the comparisons useful, even if I know all the people involved (applicant, letter-writer, noted peers).

In my previous experiences on committees evaluating candidates for tenure, promotion, or awards in disciplines other than my own, the names of compared-peers are typically meaningless to me, and are used only for the departmental evaluation stage. I can maybe get something out of the peer's institution, but even here, I'd have to know the field fairly well because there are universities that have top-ranked departments in some fields but low-ranked departments in others. This applies to all types of universities, of varying levels of prestige and ranking overall.

One thing I find fascinating when I read these letters is that some candidates/applicants are compared to a completely different set of peers in each letter. For tenure/promotion, there might be 6 or 10 or more letters containing these comparisons for one candidate, each one listing different people as peers. That means that there is little agreement about what constitutes a peer and what constitutes an individual's "field".

This is particularly true in science and engineering fields that are highly interdisciplinary and that may involve very large numbers of researchers. Letter writers may be selected for their expertise in different aspects of an applicant/candidate's research interests, and so the letter writers may be familiar with different sets of peers.

Some letter-writers who make comparisons are quite general in their response: "Dr. X's research accomplishments are comparable to W (University of A), Y (University of B), and Z (C University)."

Others are more detailed: "Dr. X's research accomplishments are greater than those of W at the University of A, but s/he has not been as productive as Y at the University of B. Nevertheless, I rate his/her creativity as superior to that of Z at C University, although all of these are second-tier compared to T at the University of R." It is rarely clear what the basis of such detailed comparisons is.

Except when it is very clear, such as when the letter-writer specifies that s/he used a specific metric, such as number of citations, papers, or the h-index.

I am usually quite skeptical about claims that someone is "the next [insert name of genius-successful-famous person]". If the claim is based on the types of problems or methods used, or is at least backed up by some achievements to date, OK.. maybe. If, however, the letter is a vague prose-poem to someone's awesomeness, these comparisons are as believable as book blurbs.

Dr. X's last article in Nature could have been written by a young Charles Dickens!

Dr. X really knows how to write a proposal -- like Marcel Proust if restricted to a 15-page limit!


Dr. X's CV is an electrifying page-turner! The 2008 article in The Journal of Rigorous Science could have been written by Stieg Larsson!


Dr. X is that rare anomaly: an evocative and inventive scientist and prose magician, like Nikola Tesla and Virginia Woolf rolled into one!