A manuscript that was submitted earlier this summer was recently returned from review, and lucky for us there were many positive comments about the work. There were also some very negative comments from one particular reviewer.
This reviewer, who indicated his identity in the review and recommended that we cite more ofpapers his , didn't really care about the cosmic implications of our work. He focused intensely on one particular method. He wrote in detail about the assumptions of this method, giving us a little tutorial about the method we used. Apparently his research is an excellent example of the appropriate uses of this method. Apparently, our research is deeply flawed.
We disagreed with every one of his comments, and wrote a detailed response letter to the editor, who accepted the paper.
By complete coincidence, as part of an entirely different project, some colleagues and I found ourselves revisiting the research of this reviewer, including the specific subject of the paper he held up as an awesome example of the appropriate use of the method in question. We had not intended to check into his research so closely, and the timing of his review relative to our delving into his research was unintentional.
Nevertheless, the term 'house of cards' kept coming to mind as we delved. Many of the issues that the reviewer raised as problems with our application of the method were present at similar or more serious levels in his own work.
I think that his work still has a lot of merit. The study he did was important and the resulting publication is interesting. Nevertheless, it was extremely hypocritical of him to take our manuscript apart as he did, and hold us to a higher (impossible) level of purity -- a level he himself did not even approach in his own work.
I don't know this reviewer well, so I have no insight into the psychological aspects of his review. Does he really think that his paper lacks the flaws he sees in ours or does he know this but prefers to pretend otherwise?
Is it ever reasonable for a reviewer to hold others to a higher standard?
In fact, in some cases it is. For example, early studies to explore a new method or a question may be a bit rough; later work should improve on this if possible.
That wasn't the case with the review in question. In this review, the researcher was being pedantic and hypocritical. We were fortunate to have an editor who weighed the reviews, positive and negative, and decided that the negative comments were not reasonable. Hooray for thoughtful editors.