double-blind.org: double-blind reviewing status of computer science conferences
This document tracks the status of major computer science conferences (currently defined as those listed in CSrankings) with respect to how they maintain anonymity during the reviewing process; it is curated by Emery Berger. For additions or updates, please post an issue or pull request on its GitHub repository.
For the purposes of this document:
-
Single-blind means that author identities and affiliations are visible to reviewers at all points during the reviewing process, while reviewer identities are hidden from authors.
-
At least partially double-blind means that author identities and affiliations are not revealed to reviewers for papers prior to the submission of initial paper reviews, and reviewer identities are hidden from authors.
-
Fully double-blind means that "double-blind to accept" is employed. That is, author identities and affiliations are not revealed to reviewers until the conclusion of the PC meeting, and only for accepted papers (including conditionally-accepted papers, pending modifications due to shepherding).
-
arXiv restricted means that submissions to arXiv or other pre-print servers are restricted for at least a limited period before and during the review process. When appropriate, there is a link to the CFP or other relevant document.
Each conference is linked to its call for papers (CFP) or similar. In cases where the CFP does not explicitly state the use of blind reviewing (or whether it is fully double-blind or not), this status has been verified directly with the program chair(s).
| Conference | At least partially double-blind? | Fully double-blind (blind to accept)? | arXiv restricted? |
|---|---|---|---|
| fully double-blind conferences | |||
| AAAI | Y | Y | |
| ACL | Y | Y | Y |
| ASE | Y | Y | N(1) |
| ASPLOS | Y | Y | N |
| CAV | Y | Y | N(2) |
| CCS | Y | Y | |
| CRYPTO | Y | Y | N |
| CVPR | Y | Y | N |
| ECCV | Y | Y | N |
| EMNLP | Y | Y | Y |
| EuroCrypt | Y | Y | N |
| EuroSys | Y | Y | |
| FAST | Y | Y | |
| FSE | Y | Y | N |
| HPCA | Y | Y | |
| ICML | Y | Y | |
| ICSE | Y | Y | Y(3) |
| IJCAI | Y | Y | N |
| ISCA | Y | Y | N(4) |
| KDD | Y | Y | N |
| MICRO | Y | Y | N(5) |
| MobiCom | Y | Y | |
| NDSS | Y | Y | |
| NeurIPS | Y | Y | N |
| NSDI | Y | Y | N |
| Oakland (IEEE S&P) | Y | Y | N |
| OOPSLA | Y | Y | N(6) |
| OSDI | Y | Y | |
| PLDI | Y | Y | N |
| PODS | Y | Y | |
| SenSys | Y | Y | |
| SIGCOMM | Y | Y | N |
| SIGGRAPH | Y | Y | N |
| SIGGRAPH Asia | Y | Y | N |
| SIGIR | Y | Y | N(7) |
| SIGMETRICS | Y | Y | N |
| SIGMOD | Y | Y | |
| SOSP | Y | Y | N |
| USENIX ATC | Y | Y | N |
| USENIX Security | Y | Y | N |
| WWW | Y | Y | N(8) |
| partially double-blind conferences | |||
| CHI | Y | N | N |
| UIST | Y | N | N |
| ICFP | Y | N | N |
| ISSTA | Y | N | |
| LICS | Y | N | |
| MobiSys | Y | ||
| POPL | Y | N | |
| SODA | Y | N | N |
| single-blind conferences | |||
| FOCS | |||
| ICDE | |||
| ICRA | |||
| IROS | |||
| VLDB |
- (1): [ASE] Authors are discouraged from posting the submitted work on arXiv or a similar site immediately before or after submitting to the conference.
- (2): [CAV] We do not discourage authors to put their submission on arXiv, but we strongly encourage authors to not put the work on arXiv around (within 1 week) or shortly after (within 1 month) the submission deadline, because potential reviewers may be subscribed to receive updates on recently posted papers.
- (3): [ICSE] You should not [...] post your work on ArXiV or a similar site just before or after submitting to the conference.
- (4): [ISCA] Information about arXiv submission must be provided on the submission form – the PC chair will make this information available to reviewers if it becomes necessary to ensure a fair review.
- (5): [MICRO] Information about arXiv submission must be provided on the submission form – the PC chair will make this information available to reviewers if it becomes necessary to ensure a fair review.
- (6): [OOPSLA] You may post to mailing lists, arxiv, social media, or another publicity channel about your work, but do not mention where the paper is submitted and do not use the exact, as-submitted title in the posting.
- (7): [SIGIR] While authors can upload to institutional or other preprint repositories such as arXiv.org before reviewing is complete, we generally discourage this since it places anonymity at risk (which could result in a negative outcome of the reviewing process).
- (8): [WWW] If available online (e.g., via arXiv) and not anonymous, their titles and abstract must be sufficiently different from the submission
Notes
- ISCA 2020's industrial track (not research track) papers are single-blind.
- CHI and UIST's research paper tracks have one or more meta-reviewers who know the identities of all authors.
Updates
- PODS will be fully double-blind as of 2022.
- SODA will be partial double-blind as of 2022.
Historical notes
- ASPLOS was partially double-blind since 2004, fully double-blind since at least 2020 (see https://github.com/double-blind-reviewing/double-blind-reviewing.github.io/issues/14).
- NSDI was single-blind in 2016, double-blind as of 2017 (see https://github.com/double-blind-reviewing/double-blind-reviewing.github.io/issues/18).
- OSDI has been double-blind since 2012 (see https://github.com/double-blind-reviewing/double-blind-reviewing.github.io/issues/16).
- PLDI was partially double-blind since 2014, fully double-blind since 2016 (see https://github.com/double-blind-reviewing/double-blind-reviewing.github.io/issues/10).
- PODS will be fully double-blind as of 2022.
- POPL has been partially double-blind since 2012 (see https://github.com/double-blind-reviewing/double-blind-reviewing.github.io/issues/11).
- SC has been fully double-blind since 2018 (see https://github.com/double-blind-reviewing/double-blind-reviewing.github.io/issues/15).
- SIGMOD has been double-blind since 2001 (see https://github.com/double-blind-reviewing/double-blind-reviewing.github.io/issues/17).
- SODA will be partial double-blind as of 2022.
- SOSP has been double-blind since at least 1999 (see https://github.com/double-blind-reviewing/double-blind-reviewing.github.io/issues/13).
- USENIX ATC has been double-blind since 2019 (see https://github.com/double-blind-reviewing/double-blind-reviewing.github.io/issues/12).
- USENIX Security was fully double-blind from 2011, partially double-blind in 2014, and back to fully double-blind as of 2017 (see https://github.com/double-blind-reviewing/double-blind-reviewing.github.io/issues/19).
Resources related to double-blind reviewing
-
Reviewer bias in single- versus double-blind peer review, Tomkins, Zhang, and Heavlin. PNAS November 28, 2017 114 (48) 12708-12713; first published November 14, 2017. (a.k.a., the WSDM Experiment)
-
Effectiveness of Anonymization in Double-Blind Reviewing, Le Goues, Brun, Appel, Berger, Khurshid, and Smaragdakis; Communications of the ACM, May 2018. ArXiv link
-
Unblinding Double-blind Reviewing Mike Hicks, from "The PL Enthusiast", June 2016.
-
Improving Publication Quality by Reducing Bias with Double-Blind Reviewing and Author Response; McKinley, ACM SIGPLAN Notices, 43(8):5--9, 2008.
-
More on Improving Reviewing Quality with Double-Blind Reviewing, External Review Committees, Author Response, and in Person Program Committee Meetings, McKinley, 2015.
-
A case for double-blind reviewing in software engineering, Brun, 2015.