Are Mean Reviewers More Competent?

By Tom Kando

 Here is the research question: Is there a relationship between competence and meanness?  Now you are supposed to laugh. This is called humor, although I have seen a lot more nonsensical sociological projects!

Most of us in the academic world are familiar with the peer review process. We submit a manuscript for a paper or a book for publication, or for conference presentation. It is reviewed and either rejected, or accepted, or accepted provisionally. In any of these three scenarios, the author usually receives feedback from the referees.  During the late1990s, I peddled a manuscript for a new textbook in Social Psychology (a re-write of my Social Interaction, which was first published in 1977, St. Louis: Times-Mosby). The various publishers obtained a total of 17 reviews over three rounds of reviews. Incidentally, I did finally get a contract and a modest advance payment, from Holt-Rinehart.

After studiously reading all the reviews, I came to the conclusion that they can be classified along two dimensions or axes: (1) from courteous to mean-spirited and (2) from good to incompetent. Now I realize that I am violating all the canons of validity and reliability when I propose to use this sample of reviews for any test. But this is only exploratory. I just want to conceptualize types of reviews and reviewers, and formulate some hypotheses about the review process. Table One shows the four categories generated by the combination of the two dimensions.

Table I. Types of Reviews

		Review  is Mean-spirited
		Yes	no
Review is Competent	Yes	1	2
	No	3	4

Definition and Operationalization of Terms: By “mean-spirited,” I mean a review by someone  who would be colloquially called an asshole. Some reviews exude the negative qualities of their writers. There are four prominent negative qualities: (1) mean-spiritedness, (2) prejudice and bigotry, (3) self-serving parochialism, and (4) temporary bad mood.

            A few examples:

(1): “this author needs to go back to high school and learn how to spell.” (Use of insult).  “This author is merely trying to cash in on the current humanistic fad.” (Imputation of evil motives).

            (2): “Clearly, this author is a xenophobic redneck who has never experienced foreign cultures.” Or, “These Symbolic Interactionists are all the same - they theorize and prove nothing.”

            (3): “This book will never fly. Symbolic Interactionism is passé. Certainly most psychologists are now aware of that” (this reviewer is no doubt a psychologist, not a sociologist). Or, “The author fails to recognize that the fundamental condition underlying all human relationships consists of power and class-conflict.” (Reviewer is obviously a Marxist).

            (4): “I have no time for this nonsense right now.”

            Now for the operational definition of what constitutes a good or a bad review (and it isn’t clear which of the two dimensions in table One is the independent variable and which one is the dependent variable - a point to which I shall return in a moment): A good review has three characteristics: (1) it makes specific criticisms; (2) it makes valid criticisms; (3) it makes important criticisms.

            Some examples:

(1):Good criticism: “Chapter Two, section 3, should also cover Ethnomethodology.”(Specif­ic and helpful).  Bad criticism: “The manuscript should be re-organized.” (Vague and unhelpful).

            (2): Good criticism: “You are misreading Howard Becker’s article.” (Could be true).  Bad criticism: George Herbert Mead coined the term “Symbolic Interactionism.” (False).

            (3): Good criticism: “The labeling theory of emotions overlooks the importance of hormonal chemistry.” (Good point). Bad criticism: “you misspelled  definately,” or  “pain  means bread in French.” ­(Petty and unimportant).

            The Four Types:

            Type 1: A well-done but mean review

            Type 2: A well-done and nice review

            Type 3: A poorly done and mean review

            Type 4: A poorly done but nice review

            The question is whether there is a correlation between the two variables, or are the cell frequencies distributed randomly? Table Two gives the null hypothesis - no correlation. Table Three provides numbers that would support the hypothesis that mean reviews tend to be more competent reviews.

Table Two: No Relationship between Reviewer’s       Competence and his/her Meanness

		Review  is Mean-spirited
		Yes	No
Review is Competent	Yes	25%	25%
	No	25%	25%

            Table Three: Mean Reviews Tend to be

Done more Competently

		Review  is Mean-spirited
		Yes	No
Review is Competent	yes	35%	15%
	no	15%	35%

So the research question is whether there is a correlation between the meanness of reviews, and how competently they are done. The hypothesis is that there is one, namely a positive one. Why?

Well, one rationale could be this: The more knowledgeable or simply the better a scholar is, the more difficulty he/she probably has with reading flawed material. Unlike the mediocre academician (say, one who teaches at a community college), the excellent professional (e.g. the Berkeley professor) does not have patience with the many inevitable flaws found in any manuscript. So his/her review is not only correct, but it also expresses his/her displeasure. So you see,  I end up treating competence as the independent variable.

Now remember, this research question leaves entirely aside the quality of the “stimulus,” i.e. the manuscript under review. Obviously, good work can be expected to receive kinder feedback than bad work. But that is a different question.

My rather grim hypothesis is inspired by personal experience, common-sense and popular culture: We can all remember the frightening teachers we had in junior high school, the witch-like old ladies who still give us an occasional nightmare. They were a menace, but they sure taught us something, didn’t they? They certainly taught us more than those “cool” permissive teachers who just wanted to be our friends. The frightening teachers taught us more, because they were both mean and competent. Same with coaches. Would Bobby Knight have won more basketball games, NCAA championships and Olympic Gold than any other man in history with love, rather than with the fierce discipline and outright meanness for which he is famous?

But this is an empirical question. I would be delighted to see my hypothesis disproved. What could be better than a world in which quality performance is associated with love and kindness, rather than the opposite? leave comment here