Incomplete instructions May 10, 2012
Posted by mareserinitatis in papers, research, Uncategorized.Tags: papers, peer review, research
trackback
I know I’ve been talking a lot about reviewing papers, but I figured one more post on the topic couldn’t hurt.
I was very lucky that my MS advisor started us reviewing papers as soon as we took a class from him. In all of his grad level classes, we were usually required to select 2-3 potential published papers for review and then to write up a critical analysis about 4-6 pages in length on one of them. It was a good experience, but I don’t think I would’ve made a very good reviewer my first year or two into grad school.
My real critiquing skills came when I started getting into some of my MS projects and I had to reproduce some of the work already done in papers. The first three I came across, it became very clear that the reviewers hadn’t done the best job: all three were missing critical details that required me to write the authors and ask how they had done certain things. In other words, there wasn’t enough information to replicate the work. That, therefore, became one of the first things that I look at with a paper. It would be nice if, when reviewing, one actually had time to sit down and try to replicate the experiment. Unfortunately, that’s not realistic…although I’ve also had papers with blatant errors that I’ve been trying to reproduce. I hope it’s just an oops that is the result of last minute writing, but I am beginning to think there are a lot of careless authors out there.
I’m not sure why this is the case, other than the fact that maybe people get too far into their experiment and fail to realize that there are many things they do automatically that one cannot take for granted. Even though most of the work I do is in simulations, there are a lot of things that appear superficially minor but can really change your results.
While there are other things one should look at it, I think the quality of most papers I’ve read follows along with the detail presented in laying out the process. If the process is not clearly spelled out, then chances are the other aspects of the paper are going to need some work, too.
So, for those who review papers, do you have things that you really look for in a paper and, if so, why?
I also blog at Engineer Blogs, home away from home to some of the best engineering blogs.
I haven’t had to write an actual review yet (although my advisor keeps threatening to make me do her next one), but there were a few things I looked at in particular when writing the rubric for my class. It pretty clearly showed my biases and the gaps in my knowledge. My advisor also contributed to the rubric, and then we also added in the points that the students had brought up that we both missed. We’re expected to give a casual, yet critical, evaluation of papers we present at lit meetings.
Bias #1 is that my heart belongs to electron microscopy, so I look for good images with legible scalebars. This is also a place where I know exactly what information should be included in the methods section, so it’s easy to tell what’s missing. We are formatting snobs, so I also look for clear figures (with good captions).
I look for a lot of details in the experimental section, especially for the techniques that I am very familiar with (mostly ones that I use myself). This can be an area where I am weak if it’s a technique that I haven’t used before, but things like missing concentrations and reaction temperatures are pretty easy to spot. We do a fair amount of statistical analysis (limited sample sizes), so I look for even simple things like # of significant figures and presence/lack of standard deviations or margins of error.
Assessing grammar comes pretty naturally to me, so I usually have an opinion on that in the first paragraph. :) Then I try to turn that off so I can look at the rest of the paper without obsessing over the linguistic errors.
That’s the easiest stuff for me. The next part is making sure that their discussion and conclusions are actually supported by the data they present. Sometimes obvious, sometimes not. This is my big weak area, and it’s a pretty big one, but (I think) I’m getting better at it.