tag:blogger.com,1999:blog-8216930037889709643.post3711995202136207233..comments2023-06-11T00:31:55.052-07:00Comments on You Infinite Snake: What's wrong with computer science reviewing?Unknownnoreply@blogger.comBlogger8125tag:blogger.com,1999:blog-8216930037889709643.post-59597931431411057442012-04-26T10:17:57.913-07:002012-04-26T10:17:57.913-07:00@Suresh: Have people noticed a change in the qual...@Suresh: Have people noticed a change in the quality or carefulness of the reviews since that change? <br /><br />I submitted my first VLDB paper last year (albeit on something related to distributed query data structures for virtual worlds, something a bit outside the norm), and was quite unimpressed in the brevity and superficialness of our reviews (compared to my normal SIGOPS venues).<br /><br />And yes, I realize a single data point is a really bad indicator of the norm :)Mike Freedmanhttp://www.cs.princeton.edu/~mfreed/noreply@blogger.comtag:blogger.com,1999:blog-8216930037889709643.post-83055693589532142232012-04-25T21:53:38.314-07:002012-04-25T21:53:38.314-07:00I don't have any direct experience with it, bu...I don't have any direct experience with it, but conceptually, I like that model a lot.Anonymoushttps://www.blogger.com/profile/13177925339061636642noreply@blogger.comtag:blogger.com,1999:blog-8216930037889709643.post-30282064293175052392012-04-25T20:55:11.612-07:002012-04-25T20:55:11.612-07:00One option you didn't mention is jettisoning t...One option you didn't mention is jettisoning the conference entirely, since much of the sins of reviewing can be laid at the feet of the compressed timeline forced by conferences.<br /><br />While this would seem overly radical, it's what VLDB (one of the flagship DB conferences) has essentially done, with its rolling monthly deadlines for paper submission throughout the year. All papers submitted within a fixed 12 month window are eligible for the next year's conference if they are accepted, and there's a proper journal review process that goes on. <br /><br />It is hard to measure the effect of this, because VLDB is only one of three/four major conferences in DB, but I think it's a model that provides a gentle transition to a more deliberative journal-like review process.Suresh Venkatasubramanianhttps://www.blogger.com/profile/15898357513326041822noreply@blogger.comtag:blogger.com,1999:blog-8216930037889709643.post-55825021531397298082011-08-03T14:50:07.395-07:002011-08-03T14:50:07.395-07:00That has been my experience as well. The author re...<em>That has been my experience as well. The author response mechanism makes authors feel better, but I'm not sure it does much more than that.</em><br /><br />I've heard that too, but I don't completely buy it. Author response can help by motivating reviewers to write better reviews and perhaps even be more open-minded in the first place. That benefit would be invisible if you just look at whether rebuttals change anyone's mind.<br /><br />Also, it's possible that authors and reviewers are just not used to the mechanism yet and are not using it as well as it could be. I've never served on a PC with author response, so I can't say. But other fields publish in journals use author responses as the normal case and apparently find them useful.Anonymoushttps://www.blogger.com/profile/13177925339061636642noreply@blogger.comtag:blogger.com,1999:blog-8216930037889709643.post-8824190643467209862011-08-03T13:58:59.583-07:002011-08-03T13:58:59.583-07:00However, it is not so clear if the rebuttals are r...<i>However, it is not so clear if the rebuttals are really having some influence in the decision making</i><br /><br />That has been my experience as well. The author response mechanism makes authors feel better, but I'm not sure it does much more than that.<br /><br />On the other hand I am very encouraged by the fact that double-blind reviewing has been making a minor comeback in the last year or so.<br /><br />At the moment I think the best hope for improving the situation is some sort of authors-rate-the-reviewers'-understanding mechanism <i>so that lazy and/or malicious reviewers feel like they have something to lose</i>.<br /><br />For this to be meaningful, reviewer ratings would need to persist in some way across conferences (even ACM-wide?), which raises anonymity concerns. The solution is not obvious, but I'm sure one of the cryptographers in the CS community could come up with something solid; reviewers already trust the program chairs to shield their identity, so continuing to rely on that assumption should make the problem easier.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8216930037889709643.post-50809998565684250762011-08-03T11:16:02.601-07:002011-08-03T11:16:02.601-07:00Very interesting post! I really like the idea of a...Very interesting post! I really like the idea of author response to reviewers - another conference that has started to do this in the last couple of years is CCS. However, it is not so clear if the rebuttals are really having some influence in the decision making ...Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8216930037889709643.post-48349528459900755502011-08-03T02:21:52.838-07:002011-08-03T02:21:52.838-07:00That's interesting to know. At a minimum, the...That's interesting to know. At a minimum, the memory proposals seem logistically complicated, and don't fulfill the goal of author response.Anonymoushttps://www.blogger.com/profile/13177925339061636642noreply@blogger.comtag:blogger.com,1999:blog-8216930037889709643.post-90037606973295322032011-08-03T01:52:36.505-07:002011-08-03T01:52:36.505-07:00Thanks for writing this post.
maintaining memory ...Thanks for writing this post.<br /><br /><i>maintaining memory across conferences (so resubmissions are associated with old reviews)</i><br /><br />ICFP'11 just tried this and it was a total catastrophe.<br /><br />Several papers marked as "resubmissions" received only one or two reviews, and were summarily rejected due to lack of reviewing (or on the basis of a *single* review!).<br /><br />I understand and support the reasoning behind the "memory" idea, but unfortunately researchers -- especially in CS -- fetishize novelty. Any sort of indication that the paper has been peer-reviewed in the past can only be negative. Authors are foolish to take advantage of any such mechanism.Anonymousnoreply@blogger.com