I think you have the wrong engineer December 12, 2013
Posted by mareserinitatis in engineering, papers.Tags: electrical engineering, engineering, peer review, reviewer comments, skills
add a comment
Every few weeks, I get a request to review papers for a conference. (For those who are sciencey types, a lot of engineering conferences require full, peer-reviewed papers rather than abstracts.) At first, this was rather cool and accepted the first half dozen that came to me. Then I started realizing that it was a bad idea, but not simply because, as you expect, it required a decent amount of time.
I started realizing I had no business reviewing some of those papers. The reviews often request that you assess your own knowledge and expertise in the area. Unfortunately, many of them didn’t have an option that was similar to, “Ignorant dolt.” The best I could do was say I had a passing knowledge and try to make constructive comments on the lack of legible text in the legend and poor grammar here and there. Oh…and finding out that half of their text was copied and pasted from another document.
I have to wonder why they aren’t more careful about screening potential reviewers given most of my requests come from a service which describes my qualifications. After all, there are several subdisciplines within electrical engineering, and I don’t imagine too many people are knowledgeable about the state of the art for all of them. Beyond that, my undergrad is in physics, so my knowledge of EE is probably even more limited than your standard engineer.
I guess I’m probably a decent reviewer as long as as you’re only looking for someone who can point out when something is undecipherable. Maybe I should add that to my skills: unenlightening critiques.
self scrutiny March 25, 2013
Posted by mareserinitatis in engineering, papers, research.Tags: conference, papers, peer review, reviewer comments
add a comment
After spending a considerable amount of time griping about other people’s papers, I feel I would be remiss if I didn’t complain about my own once in a while, as well. I’m currently revising a paper that I and a coauthor submitted to a conference. It was accepted, but there were changes requested. I started to work through some of them, but then realized that some of the comments didn’t make sense.
I sat down with a couple people, including the coauthor, and we started trying to figure out what was going on. After reading through each comment with a fine-tooth comb, we came to the realization that the problem was that we took for granted the method we were using and gave a very succinct explanation. It obviously wasn’t enough: we gathered from the comments that they resulted in a complete misunderstanding of what we were showing.
In other words, I screwed up because I didn’t explain clearly enough what we were doing. This lead to some huge misunderstandings by the reviewers, and some of the more…ummm…cynical? Yes, cynical is a good euphemism. Anyway, this explains some of the more cynical comments we got from reviewers.
The good news is that, with more explanation, I think we’ll have a much better paper when we’re done. However, this has made me realize that I really can’t take for granted what my reviewers may or may not know. It’s best to be as explicit and detailed as possible.
Incomplete instructions May 10, 2012
Posted by mareserinitatis in papers, research, Uncategorized.Tags: papers, peer review, research
1 comment so far
I know I’ve been talking a lot about reviewing papers, but I figured one more post on the topic couldn’t hurt.
I was very lucky that my MS advisor started us reviewing papers as soon as we took a class from him. In all of his grad level classes, we were usually required to select 2-3 potential published papers for review and then to write up a critical analysis about 4-6 pages in length on one of them. It was a good experience, but I don’t think I would’ve made a very good reviewer my first year or two into grad school.
My real critiquing skills came when I started getting into some of my MS projects and I had to reproduce some of the work already done in papers. The first three I came across, it became very clear that the reviewers hadn’t done the best job: all three were missing critical details that required me to write the authors and ask how they had done certain things. In other words, there wasn’t enough information to replicate the work. That, therefore, became one of the first things that I look at with a paper. It would be nice if, when reviewing, one actually had time to sit down and try to replicate the experiment. Unfortunately, that’s not realistic…although I’ve also had papers with blatant errors that I’ve been trying to reproduce. I hope it’s just an oops that is the result of last minute writing, but I am beginning to think there are a lot of careless authors out there.
I’m not sure why this is the case, other than the fact that maybe people get too far into their experiment and fail to realize that there are many things they do automatically that one cannot take for granted. Even though most of the work I do is in simulations, there are a lot of things that appear superficially minor but can really change your results.
While there are other things one should look at it, I think the quality of most papers I’ve read follows along with the detail presented in laying out the process. If the process is not clearly spelled out, then chances are the other aspects of the paper are going to need some work, too.
So, for those who review papers, do you have things that you really look for in a paper and, if so, why?
Review season May 7, 2012
Posted by mareserinitatis in engineering, papers, research, younger son.Tags: engineering, engineering research, papers, peer review, research, reviewer comments
4 comments
Both Mike and I have been getting requests to review papers, and this has led to a lot of foul language around the house…along with frequent reminders from the younger son that our language is inappropriate.
It’s really hard to restrain yourself, however. As we’re sitting at the dining room table, occasionally one of us will turn our laptop toward the other and ask something like, “What does this look like to you?” or, “What do you think this means?” or, “What the hell were they thinking?”
I have to admit that I appreciate having a second pair of eyes to catch the things that I miss. I’m sure the authors of the papers we’re reviewing probably will not appreciate it. Not only do they have the third reviewer going over their papers, they have two of them. I hope this will result in double the hair pulling and teeth gnashing on their end…because it sure has for us.
Was blind, but now I see… May 6, 2012
Posted by mareserinitatis in engineering, papers, research.Tags: double blind, papers, peer review, single blind
3 comments
I was recently asked to review a paper for a fairly large conference in one of the engineering subfields I’m involved in. This particular conference is one which I’ve not attended, so I had no familiarity with the procedures. As a side note for non-engineers, I discussed before (on my old blog) that many (most?) engineering conferences take full, peer-reviewed papers.
When I received the paper and looked it over, I nearly fell out of my chair. I could see the freakin’ authors!
In most of the conferences where I’ve submitted papers, the peer-review was double blind. One conference in particular was this way because it’s such a small area of research that they wanted to make doubly sure that people are as objective as possible. (In reality, there’s a good chance that you could tell who it was just by what they were doing, but I applaud the effort.) It seems like a very straight-forward thing to do: you submit the paper without any names on it. The session chair knows who it is but picks people to review who will be none the wiser. If the paper is accepted, a revision is submitted with names on it. Easy-peasy.
I have to say that this was very disconcerting for me. I don’t WANT to know whose paper I’m reviewing. I spent the whole time writing this review terrified that knowing who they were, where they were from, how many authors were on the paper, etc. was affecting my perceptions of the paper and destroying my objectivity. I was amazed at all the stupid things I found myself questioning in terms of my reaction. Was I making a mountain out of a molehill? Was I overly impressed by something which shouldn’t have impressed me?
It really isn’t all that hard to keep reviews double-blind when using an automated submission system such as the one used for this conference (and most IEEE conferences), and as a reviewer, I would have been far more comfortable.
I’m curious about other fields, though. Is single-blind review the norm? (When I stumble across these things, I feel like I’ve been living under a rock.)
I also blog at Engineer Blogs, home away from home to some of the best engineering blogs.
When is it plagiarism? June 6, 2014
Posted by mareserinitatis in engineering, papers, research, work.Tags: engineering, IEEE, journals, papers, peer review, plagiarism, reviewer comments
3 comments
When I sit down to write a paper, I usually try to start from scratch. I type up an outline and try to fill it in, and then I begin work on all the different parts of the paper. While it has been tempting to reuse sections of previous papers, particularly the introductory material, I try not to do that. If you work in a specialized field, people are going to notice that they’re reading the same thing over. My writing may be fine, but after the 3rd or 4th time, it’s going to bore even me.
The issue came to the fore as I’ve been reviewing papers for a conference. While it’s not one I think I will be able to go to (it’s usually in Asia), I have reviewed for this conference the past couple years and really get some interesting things to examine. However, this year, all of the papers I reviewed has issues with self-plagiarism: that is, they copied verbatim materials from their own previous papers. Many of the papers I review are now being checked automatically for similarity to other papers, and while the process is supposed to be double-blind (that is, they don’t know I’m reviewing their paper, and I’m not supposed to know that I reviewed theirs), it makes it very easy to figure out who wrote the paper I’m reviewing: it’s the one with huge tracts of text that are identical but never referenced.
As I mentioned, I try to write papers from scratch, but I started to wonder if this was an ethical issue. After all, if I wrote a paper, shouldn’t I be allowed to copy it? It turns out that it’s not a good idea. In particular, most of the papers I’m dealing with will fall under IEEE copyright rules (that is, the authors transfer over copyright of their written materials should the IEEE publish those materials). Therefore, if you wrote the paper and it was published by IEEE, it’s simply not a matter of copying your own writing but plagiarism of IEEE materials. In fact, the IEEE communications society has an explicit policy that says,
I know people who do this regularly. All you have to do is read enough of their papers, and it becomes obvious that the intro section is commonly recycled by several authors. I really don’t like the practice because it also drives up index values for papers that are simply examples of related work while not being foundational. On the other hand, it is a pain to rewrite those sections every time.
I’m very glad the Com Soc is being very explicit about their policy. However, other places are not as explicit, and this is honestly something that no one has ever mentioned to me. It’s something I would like to see delineated more clearly by all publications as I think it would draw more attention to using ethical practices in paper writing and submission.
When reviewing, I can’t be certain that the person writing the paper is aware of the policies on self-citation, if there even is one for a given organization or venue, so I generally mention that it’s a good idea to change the text. I’m always curious what the editors/session chairs do with this feedback, though. Do they take it seriously? Finally, it reinforces to me that it’s never a good idea to reuse previous writing unless it’s properly attributed, even if it is my own.