Is computer vision peer-review less progressive than others?

Context: I was discussing with some people. A handful mentioned sad things about the ongoing peer-review process for CVPR’21.

Main issue: Lack of transparency. Discussion between reviewers, ACs, and PCs are hidden. Who holds them accountable for their job?

Facts

  • Most (all?) conferences prefer using CMT3.

  • AFAIK, only ECCV’20 used open-review, but without transparency. By that, I mean that the discussion was confidential.

What can we do?

  • First, let’s discuss here.

  • Shall we track the quantifiable nuisance of the reviewing process that we observed?

  • Perhaps, we can raise a motion to use Open-Review in the next conferences.

  • Shall we demand that discussions between the Reviewer and ACs are released?
    One can make better decisions with more information. The discussion can tell us if our paper was properly assessed or rejected due to randomness. For example, reviewers and AC are sometimes sloppy with borderline papers.

1 Like

I will vote for releasing discussions. Open-review has drawbacks too. Until we find a way to somehow get the best of both worlds, we can settle with the transparency in closed reviewing procedure.

1 Like

Pros & Cons of Open-review

Pros
:heavy_plus_sign: They are quite active in Twitter
:heavy_plus_sign: I got the impression that they got stuff implemented quickly
:heavy_plus_sign: I think that they fixed the multiple account issue of CMT3.
:heavy_plus_sign: It’s highly configurable. The fact that ICLR and ECCV review process were different means that one can tailor to multiple the needs.

@justanhduc Could you please elaborate on the drawbacks of open-review?

@escorciav I have seen some drawbacks of open-review

  • The first, and most major, is that reviewers will start to be biased once people start to comment, especially reviewers who are not confident about the paper. Some comments are honest, but some might not.
  • I have seen some dramas in open review that people trading words and insults directly in the comment section. While it does not directly affect the decision on the paper, obviously it is not what open review is for, and it may affect the authors of the paper.

I think these two cons are completely solved by making it closed, but we wont know why the decision on our paper is made that way. Hence, the publish of the discussions after the decision.

1 Like

Those might be simple things to implement. If one needs them, we can reach out to them on Twitter. I have the impression that they operate under the principle of “where there is a will, there is a how”.