Proposal: Policy for Papers Review at SIGCHI Conferences
Hey SIGCHI people-
This Tumblr is trying out a broader mission of being a common place to post news and summarize information about SIGCHI, on top of news about the annual CHI conference. Not everyone is on Facebook or Twitter, or even YikYak, so weāre trying to make this theĀ āhome baseā and then collect feedback through a variety of channels.
The text below is a proposed new guideline for the treatment of reviews received through SIGCHI conference submissions. This is not set in stone, we are explicitly presenting this policy to get wide discussion from the SIGCHI publishing community. This draft copy is also posted on the SIGCHI website.Ā
Feel free to comment here, or to email me. I may not be able to respond right away, but if you do opt for the private message I will reflect your thoughts in the EC discussion on this issue. I will also amend this post with comments drawn from other social media channels, like CHI Meta on Facebook, or the Twitter handle.
Thanks, Cliff Lampe VP of Publications - SIGCHI Executive Committee
UPDATE: People have asked for aĀ ādiffā to be able to see what the changes are. Those are now in bold in the text. ------------------------------
The purpose of the SIGCHI Submissions and Reviewing Policy is to provide a framework within which the conference Venue Chair(s) will design their submissions and reviewing processes for submissions that appear in the conference proceedings. This document is intended to be a policy document, not a process document, and it supplements the ACM Rights and Responsibility Policy and the ACM Author Rights and Publishing Policy (Version 9).
Outside this policy, we encourage all chairs to regularly revisit the review process, identifying opportunities for improvement, and using an experimental approach that includes gathering data to evaluate the effects of any changes based on this policy.
Submission
All submitters must accept the following agreement:
Making a submission to a SIGCHI conference is a serious matter. Submissions require time and effort by SIGCHI volunteers to organize and manage the reviewing process, and, if the submission is accepted, the publication and presentation process. Thus, we expect anyone submitting to a SIGCHI conference to confirm the following:
I confirm that this submission is the work of myself and my co-authors.
I confirm that I or my co-authors hold copyright to the content, and have obtained appropriate permissions for any portions of the content that are copyrighted by others.
I confirm that any research reported in this submission involving human subjects has been conducted in an ethical manner and has, where available, gone through an appropriate approval process (eg. at my institution).
I confirm that if my submission is accepted, I or one of my co-authors will attend the conference.
I confirm that I and my co-authors will treat the anonymous reviews of our submission as confidential and not for public dissemination unless specifically authorized to do so by the reviewer. Reviews can be shared within the confines of my institution, with individual colleagues, for the purposes of responding to reviewers, or for the mentoring and education of students.
Just as reviewers agree to treat the submissions they review as confidential, so authors are expected to treat reviews they receive as confidential and they should not be made public. Among the implications of this are that neither reviews, nor excerpts of reviews, may be published either in paper form or on social media. The intention here is to ensure reviewers can provide candid reviews which wonāt be misinterpreted outside the confines of the review process. Ā Under certain circumstances conferences might allow reviewers to specifically authorize the public use of their reviews. Ā This must be done as an āopt-inā process, or made clear as part of reviewer agreement.
SIGCHI is committed to running a fair and effective review process, and to working to improve the review process over time. We recognize that authors may sometimes feel a review is unfair, inappropriate, or problematic in other ways. SIGCHI reviewing processes include a number of mechanisms for addressing this situation. No conference has every mechanism, but all have several mechanisms. These are, where available and in the order that they should be pursued, as follows:
Where available, any rebuttal process, which is intended to flag factual errors in reviews
Any other form of response to reviewers, accompanying an updated paper, during a revision process
Appeal to the Associate Chair in charge of the paper
Appeal to Chair of the Subcommittee reviewing the paper
Appeal to the Venue Chair(s) (eg. papers chair(s))
Appeal to the Technical Program Chair(s) or Program Chair(s)
Appeal to the General Chair(s)
Appeal to the Conference Steering Committee Chair
Appeal to the SIGCHI Conference Management Committee
Appeal to the SIGCHI Executive Committee
Guidance on Reviewing
Reviewing is vital to the success of SIGCHI conferences. It is critical to the production of high quality publications, and it provides valuable feedback that assists authors in developing their skills as researchers and writers. Reviewing also represents a vast amount of volunteer time, and it is important that SIGCHI make effective use of this resource.
The SIGCHI approach to reviewing has three aims:
Quality in accepted submissions. SIGCHI is committed to strengthening the discipline of HCI and its constituent sub-disciplines by maintaining a high standard of quality in the submissions it accepts. Therefore SIGCHI looks for submissions each of which makes its own significant and potentially valuable contribution to the relevant sub-discipline's body of published work The quality of execution of the reported work must be sufficient to gain the trust of readers and ensure that they can extract from the paper the value and benefits it claims to offer.
Fairness in reviewing. SIGCHI has a policy of reviewing submissions on a fair and even-handed basis across all sub-disciplines, and takes this into account in recruiting and instructing reviewers. Each reviewer prepares his or her review of a submission independently of other reviewers. A committee member may then organize an anonymous discussion of the paper, but only between reviewers who have already submitted a full review in draft. The review criteria applied are derived from SIGCHI's policy on quality and therefore emphasize significance of contribution and benefit to be gained by readers. Reviewers are expected to assess significance in terms of relevant past work. The criteria also ask that work be carried out to sufficient standards of validity or trustworthiness that readers can have belief Ā in the results and apply them with confidence. Presentations should be of high standard, including the written presentations. These criteria apply equally to all submissions. They are spelled out in each conferenceās Call for Participation and in the review form used by SIGCHI reviewers. They provide the basis on which the venue committee selects the submissions to be included in the program.
Continuity and evaluation in the review process. SIGCHI is committed to maintaining an appropriate level of continuity in reviewing across conferences and to ensuring that process changes are adequately evaluated. The introduction of process changes should be viewed as part of the long-term, incremental development of the review process, and steps should be taken to minimize undesired outcomes. Thus every effort should be made to verify perceived problems before addressing them with process changes. Equally, effort should be devoted to identifying process problems that might go undetected and therefore unaddressed. All proposed solutions should be tested before their introduction, as thoroughly and as early as possible. The outcome of those changes that are introduced to the process should be evaluated and the results delivered in time to inform those responsible for the next conference. Conferences should also communicate results more widely in a timely way to the community. Detection, verification, testing and evaluation must be based on review process data, and will involve storing these data, making them accessible and addressing related security and privacy issues.
----- THIS IS THE CURRENT VERSION
The purpose of the CHI Papers Review Policy is to provide a framework within which the CHI Papers (Co-)Chairs will design their review process. It is intended to be a policy document, not a process document. The SIGCHI Publications Board is responsible for policy, while the Papers (Co-)Chairs are responsible for the review process.
Quality in published papers. CHI is committed to strengthening the discipline of HCI and its constituent sub-disciplines by maintaining a high standard of quality in the papers it publishes. Therefore CHI looks for papers each of which makes its own significant and potentially valuable contribution to the relevant sub-discipline's body of published work The quality of execution of the reported work must be sufficient to gain the trust of readers and ensure that they can extract from the paper the value and benefits it claims to offer. CHI sets these standards of quality in the belief that they are similar to, and therefore consistent with, the quality measures applied by other HCI conferences and journals.
Fairness in reviewing. CHI has a policy of reviewing papers on a fair and even-handed basis across all sub-disciplines, and takes this into account in recruiting and instructing reviewers. Each reviewer prepares his or her review of a paper independently of other reviewers. A committee member may then organize an anonymous discussion of the paper, but only between reviewers who have already submitted a full review in draft. The review criteria applied are derived from CHI's policy on quality and therefore emphasize significance of contribution and benefit to be gained by readers. Reviewers are expected to assess significance in terms of relevant past work. The criteria also ask that work be carried out to sufficient standards of validity that readers can place trust in the results and apply them with confidence. A high standard of written presentation is also expected. These criteria apply equally to all submitted papers. They are spelled out in the CHI Call for Participation and in the review form used by CHI reviewers. They provide the basis on which the papers committee selects the papers to be included in the program.
Continuity and evaluation in the review process. CHI is committed to maintaining an appropriate level of continuity across conferences and to ensuring that process changes are adequately evaluated. The introduction of process changes should be viewed as part of the long-term, incremental development of the review process, and steps should be taken to minimize undesired outcomes. Thus every effort should be made to verify perceived problems before addressing them with process changes. Equally, effort should be devoted to identifying process problems that might go undetected and therefore unaddressed. All proposed solutions should be tested before their introduction, as thoroughly and as early as possible. The outcome of those changes that are introduced to the process should be evaluated and the results delivered in time to inform those responsible for the next conference. Detection, verification, testing and evaluation must be based on review process data, and will involve storing these data, making them accessible and addressing related security and privacy issues.















