double-blind.org: double-blind reviewing status of computer science conferences
This document tracks the status of major computer science conferences (currently defined as those listed in CSrankings) with respect to how they maintain anonymity during the reviewing process; it is curated by Emery Berger. For additions or updates, please post an issue or pull request on its GitHub repository.
For the purposes of this document:
Single-blind means that author identities and affiliations are visible to reviewers at all points during the reviewing process, while reviewer identities are hidden from authors.
At least partially double-blind means that author identities and affiliations are not revealed to reviewers for papers prior to the submission of initial paper reviews, and reviewer identities are hidden from authors.
Fully double-blind means that "double-blind to accept" is employed. That is, author identities and affiliations are not revealed to reviewers until the conclusion of the PC meeting, and only for accepted papers (including conditionally-accepted papers, pending modifications due to shepherding).
arXiv restricted means that submissions to arXiv or other pre-print servers are restricted for at least a limited period before and during the review process. When appropriate, there is a link to the CFP or other relevant document.
Each conference is linked to its call for papers (CFP) or similar. In cases where the CFP does not explicitly state the use of blind reviewing (or whether it is fully double-blind or not), this status has been verified directly with the program chair(s).
|Conference||At least partially double-blind?||Fully double-blind (blind to accept)?||arXiv restricted?|
|fully double-blind conferences|
|Oakland (IEEE S&P)||Y||Y||N|
|partially double-blind conferences|
- (*): ISCA 2020's industrial track (not research track) papers are single-blind.
- (**): CHI and UIST's research paper tracks have one or more meta-reviewers who know the identities of all authors.
- (1): Authors are discouraged from posting the submitted work on arXiv or a similar site immediately before or after submitting to the conference.
- (2): "We do not discourage authors to put their submission on arXiv, but we strongly encourage authors to not put the work on arXiv around (within 1 week) or shortly after (within 1 month) the submission deadline, because potential reviewers may be subscribed to receive updates on recently posted papers."
- (3): "You should not [...] post your work on ArXiV or a similar site just before or after submitting to the conference."
- (4): Information about arXiv submission must be provided on the submission form – the PC chair will make this information available to reviewers if it becomes necessary to ensure a fair review.
- (5): "You may post to mailing lists, arxiv, social media, or another publicity channel about your work, but do not mention where the paper is submitted and do not use the exact, as-submitted title in the posting."
- (6): "While authors can upload to institutional or other preprint repositories such as arXiv.org before reviewing is complete, we generally discourage this since it places anonymity at risk (which could result in a negative outcome of the reviewing process)."
- (7): "If available online (e.g., via arXiv) and not anonymous, their titles and abstract must be sufficiently different from the submission"
- (8): PODS will be fully double-blind as of 2022.
Resources related to double-blind reviewing
Reviewer bias in single- versus double-blind peer review, Tomkins, Zhang, and Heavlin. PNAS November 28, 2017 114 (48) 12708-12713; first published November 14, 2017. (a.k.a., the WSDM Experiment)
Effectiveness of Anonymization in Double-Blind Reviewing, Le Goues, Brun, Appel, Berger, Khurshid, and Smaragdakis; Communications of the ACM, May 2018. ArXiv link
Unblinding Double-blind Reviewing Mike Hicks, from "The PL Enthusiast", June 2016.
Improving Publication Quality by Reducing Bias with Double-Blind Reviewing and Author Response; McKinley, ACM SIGPLAN Notices, 43(8):5--9, 2008.