I think about peer review a lot. This is probably because much of my ultimate success in academia will be defined by the peer review process. My fellowship director had lots of good advice (especially in retrospect) and while searching for my first job he told me not to go anywhere that they couldn’t clearly describe how they measured success…but that is a different post altogether. Anyway, the mantra is teaching, research, and service. And while teaching (especially) and service are important, if you ever want to advance – you have to produce scholarly works. Research typically requires money (or at minimum, the currency of time) and there is a prevalent school of thought that believes research does not even *exist* until it has been published in a peer reviewed journal. Every step of the way, you are dependent on peer review. Your grant and research proposals undergo peer review. Your abstract and papers for conferences undergo peer review. Your journal manuscripts undergo peer review. Now, I happen to love the process of research – so this suits me fine. I love the process of identifying and prioritizing which question is worth answering, figuring out a methodologically appropriate way to answer the question, actually answering the question (the step with the most mystery), and then converting the answer to some form where it can make a difference.
Being Vexed and Trends in Transparency
I love research. I appreciate the peer review process. But sometime peer review vexes me. I think it is because I have seen how arbitrary and uneven it can be. This isn’t sour grapes. I have largely been blessed with great collaborators and decent results (decent results = stuff gets published). Rather, it is from taking part in panels that review grant proposals and serving as a peer reviewer (and editorial board member) for a number of journals. Most of the journals I peer review for are the usual suspects in pharmacy practice and informatics, but I have also had the opportunity to contribute in the review process for others like Journal of Immigrant and Minority Health, The Lancet Oncology, and Patient Education and Counseling. I think serving as a peer reviewer is absolutely fantastic for ongoing professional development and I always try to encourage new faculty to seek out those opportunities. It is a good way to get exposed to new research in the pipeline, it forces you to truly critically evaluate literature, and it enhances your abilities as a mentor. Increasingly, due to trends in transparency, you (as a peer reviewer) will sometimes see the other peer reviewers’ comments and the final disposition of the paper after the fact. This is invaluable. You get to see how your peer group thinks and reacts to the same information you examined! Did they miss something? Did you miss something? Did they prioritize something differently than you did? Why? Why not? As an intellectual exercise, it is tough to beat. The whole process is really time consuming if done correctly, but it is an investment.
Revenge of the Throwaway
When I review anything, the basic formula is some version of: rigor+relevance+clarity. How those items are weighted can vary tremendously. I think an interesting example is a JAMA study in which a half-dozen physicians evaluated clinical relevance in review articles from peer reviewed journals compared to review articles from “throwaway” journals . Seriously. Throwaway. Look it up. (Yes, I know it is common parlance…but I still feel bad for their editors). Anyway, their findings were that the throwaway journal articles had less methodological rigor, but significantly higher relevance (p<.001). A bit of the ‘beauty in the eye of the beholder’, eh?
There are plenty of issues and controversy surrounding peer review as it is currently practiced. Some of my favorite topics involving peer review include: open vs blinded, ethical quandaries, pre and post publication, training and consistency, and ‘peer review 2.0’. The best overview of the topic I have read is by a former editor of BMJ . It has a light enough tone to make it an easy read, but is complete enough to make it worthwhile. As a bonus, it is available for free, full-text.
 Rochon PA, Bero LA, Bay AM, et al. Comparison of review articles published in peer-reviewed and throwaway journals. JAMA. 2002;287(21):2853-6.
 Smith R. Peer review: a flawed process at the heart of science and journals. J R Soc Med. 2006;99(4): 178–82.