SIGCHI

Hey SIGCHI people-

This Tumblr is trying out a broader mission of being a common place to post news and summarize information about SIGCHI, on top of news about the annual CHI conference. Not everyone is on Facebook or Twitter, or even YikYak, so we’re trying to make this the “home base” and then collect feedback through a variety of channels.

The text below is a proposed new guideline for the treatment of reviews received through SIGCHI conference submissions. This is not set in stone, we are explicitly presenting this policy to get wide discussion from the SIGCHI publishing community. This draft copy is also posted on the SIGCHI website. 

Feel free to comment here, or to email me. I may not be able to respond right away, but if you do opt for the private message I will reflect your thoughts in the EC discussion on this issue. I will also amend this post with comments drawn from other social media channels, like CHI Meta on Facebook, or the Twitter handle.

Thanks, Cliff Lampe VP of Publications - SIGCHI Executive Committee

UPDATE: People have asked for a ‘diff’ to be able to see what the changes are. Those are now in bold in the text. ------------------------------

The purpose of the SIGCHI Submissions and Reviewing Policy is to provide a framework within which the conference Venue Chair(s) will design their submissions and reviewing processes for submissions that appear in the conference proceedings. This document is intended to be a policy document, not a process document, and it supplements the ACM Rights and Responsibility Policy and the ACM Author Rights and Publishing Policy (Version 9).

Outside this policy, we encourage all chairs to regularly revisit the review process, identifying opportunities for improvement, and using an experimental approach that includes gathering data to evaluate the effects of any changes based on this policy.

Submission

All submitters must accept the following agreement:

Making a submission to a SIGCHI conference is a serious matter. Submissions require time and effort by SIGCHI volunteers to organize and manage the reviewing process, and, if the submission is accepted, the publication and presentation process. Thus, we expect anyone submitting to a SIGCHI conference to confirm the following:

I confirm that this submission is the work of myself and my co-authors.

I confirm that I or my co-authors hold copyright to the content, and have obtained appropriate permissions for any portions of the content that are copyrighted by others.

I confirm that any research reported in this submission involving human subjects has been conducted in an ethical manner and has, where available, gone through an appropriate approval process (eg. at my institution).

I confirm that if my submission is accepted, I or one of my co-authors will attend the conference.

I confirm that I and my co-authors will treat the anonymous reviews of our submission as confidential and not for public dissemination unless specifically authorized to do so by the reviewer. Reviews can be shared within the confines of my institution, with individual colleagues, for the purposes of responding to reviewers, or for the mentoring and education of students.

Just as reviewers agree to treat the submissions they review as confidential, so authors are expected to treat reviews they receive as confidential and they should not be made public. Among the implications of this are that neither reviews, nor excerpts of reviews, may be published either in paper form or on social media. The intention here is to ensure reviewers can provide candid reviews which won’t be misinterpreted outside the confines of the review process.  Under certain circumstances conferences might allow reviewers to specifically authorize the public use of their reviews.  This must be done as an “opt-in” process, or made clear as part of reviewer agreement.

SIGCHI is committed to running a fair and effective review process, and to working to improve the review process over time. We recognize that authors may sometimes feel a review is unfair, inappropriate, or problematic in other ways. SIGCHI reviewing processes include a number of mechanisms for addressing this situation. No conference has every mechanism, but all have several mechanisms. These are, where available and in the order that they should be pursued, as follows:

Where available, any rebuttal process, which is intended to flag factual errors in reviews

Any other form of response to reviewers, accompanying an updated paper, during a revision process

Appeal to the Associate Chair in charge of the paper

Appeal to Chair of the Subcommittee reviewing the paper

Appeal to the Venue Chair(s) (eg. papers chair(s))

Appeal to the Technical Program Chair(s) or Program Chair(s)

Appeal to the General Chair(s)

Appeal to the Conference Steering Committee Chair

Appeal to the SIGCHI Conference Management Committee

Appeal to the SIGCHI Executive Committee

Guidance on Reviewing

Reviewing is vital to the success of SIGCHI conferences. It is critical to the production of high quality publications, and it provides valuable feedback that assists authors in developing their skills as researchers and writers. Reviewing also represents a vast amount of volunteer time, and it is important that SIGCHI make effective use of this resource.

The SIGCHI approach to reviewing has three aims:

Quality in accepted submissions. SIGCHI is committed to strengthening the discipline of HCI and its constituent sub-disciplines by maintaining a high standard of quality in the submissions it accepts. Therefore SIGCHI looks for submissions each of which makes its own significant and potentially valuable contribution to the relevant sub-discipline's body of published work The quality of execution of the reported work must be sufficient to gain the trust of readers and ensure that they can extract from the paper the value and benefits it claims to offer.

Fairness in reviewing. SIGCHI has a policy of reviewing submissions on a fair and even-handed basis across all sub-disciplines, and takes this into account in recruiting and instructing reviewers. Each reviewer prepares his or her review of a submission independently of other reviewers. A committee member may then organize an anonymous discussion of the paper, but only between reviewers who have already submitted a full review in draft. The review criteria applied are derived from SIGCHI's policy on quality and therefore emphasize significance of contribution and benefit to be gained by readers. Reviewers are expected to assess significance in terms of relevant past work. The criteria also ask that work be carried out to sufficient standards of validity or trustworthiness that readers can have belief  in the results and apply them with confidence. Presentations should be of high standard, including the written presentations. These criteria apply equally to all submissions. They are spelled out in each conference’s Call for Participation and in the review form used by SIGCHI reviewers. They provide the basis on which the venue committee selects the submissions to be included in the program.

Continuity and evaluation in the review process. SIGCHI is committed to maintaining an appropriate level of continuity in reviewing across conferences and to ensuring that process changes are adequately evaluated. The introduction of process changes should be viewed as part of the long-term, incremental development of the review process, and steps should be taken to minimize undesired outcomes. Thus every effort should be made to verify perceived problems before addressing them with process changes. Equally, effort should be devoted to identifying process problems that might go undetected and therefore unaddressed. All proposed solutions should be tested before their introduction, as thoroughly and as early as possible. The outcome of those changes that are introduced to the process should be evaluated and the results delivered in time to inform those responsible for the next conference. Conferences should also communicate results more widely in a timely way to the community. Detection, verification, testing and evaluation must be based on review process data, and will involve storing these data, making them accessible and addressing related security and privacy issues.

----- THIS IS THE CURRENT VERSION

The purpose of the CHI Papers Review Policy is to provide a framework within which the CHI Papers (Co-)Chairs will design their review process. It is intended to be a policy document, not a process document. The SIGCHI Publications Board is responsible for policy, while the Papers (Co-)Chairs are responsible for the review process.

Quality in published papers. CHI is committed to strengthening the discipline of HCI and its constituent sub-disciplines by maintaining a high standard of quality in the papers it publishes. Therefore CHI looks for papers each of which makes its own significant and potentially valuable contribution to the relevant sub-discipline's body of published work The quality of execution of the reported work must be sufficient to gain the trust of readers and ensure that they can extract from the paper the value and benefits it claims to offer. CHI sets these standards of quality in the belief that they are similar to, and therefore consistent with, the quality measures applied by other HCI conferences and journals.

Fairness in reviewing. CHI has a policy of reviewing papers on a fair and even-handed basis across all sub-disciplines, and takes this into account in recruiting and instructing reviewers. Each reviewer prepares his or her review of a paper independently of other reviewers. A committee member may then organize an anonymous discussion of the paper, but only between reviewers who have already submitted a full review in draft. The review criteria applied are derived from CHI's policy on quality and therefore emphasize significance of contribution and benefit to be gained by readers. Reviewers are expected to assess significance in terms of relevant past work. The criteria also ask that work be carried out to sufficient standards of validity that readers can place trust in the results and apply them with confidence. A high standard of written presentation is also expected. These criteria apply equally to all submitted papers. They are spelled out in the CHI Call for Participation and in the review form used by CHI reviewers. They provide the basis on which the papers committee selects the papers to be included in the program.

This year at CHI we won’t be including a USB key in your conference bag. We surveyed attendees and found that 40% of you used the USB precisely once, to copy the contents to the paper; 30% of you never used the key at all or accessing the contents; 20% told us you accessed the USB occasionally during the conference.

Each submission in the Proceedings and Extended Abstracts will be available starting the week before the conference on the ACM Digital Library, but for your convenience and to replace the utility of having the USB key, here’s a link to a file containing all the proceedings and extended abstracts.

[Link removed by request of the ACM]

We’ve left off the videos, which will all be available through the Digital Library, so it’s a manageable (hopefully!) 3.33GB, and uncompresses to 4.5GB. 

With CHI’16 just around the corner, we thought it was a good time to share some of the data and experiences from the Games & Play virtual subcommittee. For a discussion on why this decision was made, see Jofish’s post on the subject: Introducing the virtual subcommittee.

What was the process?

The committee then met virtually at the same time as the in-person meeting via Bluejeans, an online video-conferencing software. Prior to the meeting, a designated technology support person tested the software with committee members to ensure that it worked properly. The committee attended all plenaries virtually to ensure, as much as possible, that they had the same experience as the in-person PC members. The objective of the virtual meeting was to quickly review all easy accepts and rejects, providing a sanity check, while using the bulk of the meeting to discuss more controversial (or borderline) papers.

The Games & Play subcommittee followed the same timelines as the other subcommittees. The biggest difference was that they did not have an in-person meeting. No face-to-face contact meant that they could leverage the strengths of distributed collaboration and front-load some of the work, such as starting discussions sooner.

The 1AC and 2AC were encouraged to discuss their papers and come to convergence following the rebuttal period and before the meeting. A 3AC was also added at this stage in instances where the 1AC and 2AC were not able to come to agreement, the paper had received borderline scores, or the ACs felt that they needed expertise in a certain area. This front-loading allowed the committee to focus greater attention on the borderline papers at the meeting.

What worked well?

Perhaps most importantly, the subcommittee chairs were very effective facilitators. They kept the meeting moving and on-topic. Consequently, there were very few instances of grandstanding and limited-to-no tangents. This resulted in quick decisions for non-controversial papers, and focused and in-depth discussion for borderline papers.

Surprisingly, the software worked extremely well. It did not break in the ways that we anticipated it would and it was able to support multiple people simultaneously. Having a designated technology support person helped to ensure that technical glitches didn’t happen and that everyone had the proper plug-ins to make the software work. Pre-testing the software meant that people weren’t scrambling at the beginning of the meeting to figure out why the system wasn’t working (which is what happened in a few cases for the Late-Breaking-Work virtual program committee).

The committee quickly established processural conventions. For each paper, the 1AC provided a summary of the paper and then gave their decision, followed by the 2AC. A final decision was then made on the paper or it was tabled for further discussion. In cases of conflict, those members who were conflicted were asked to mute their sound and cover their screens and then pay attention to the chat to see when they should return to the meeting.

What didn’t work as well?

The process for determining best papers also did not work well. This is an already challenging process for the in-person PC and was made more complicated by the virtual nature of the meeting. Not all papers that should have been considered were considered. And in order to maintain anonymity, the committee had to rely on voting via email, which was a slow and painful process. One way this could be improved is through the creation of an anonymous voting system (which could also be useful for the in-person meeting).

Although the software worked, aspects of the technology did not adequately support the specific needs of the meeting. For instance, there was no built-in functionality that supported “hallway” conversations. This meant that members who needed to talk about a specific paper had to move off the software and talk via email or Skype. (The built-in chat function does not allow for private, anonymous conversations).

The software also does not adequately support conflicts. Although the committee established a convention (mentioned above), it relied on an honor-system -- there was no way to guarantee that members followed the guidelines. Functionality that allows the chairs to mute and unshare their screen for a period of time could resolve this issue.

What were the positives of the virtual meeting?

The social aspect did not suffer as much as was anticipated. For many, the virtual meeting was actually a fun experience.

The biggest positive of the virtual meeting was accessibility. It allowed people who wouldn’t have been able to travel (or who didn’t want to travel) to sit on the committee. About half of the people who served on Games & Play were in this boat. This was due either to financial limitations (i.e., no available funding) or time constraints. The result was a new pool of ACs.

The review process did not suffer. In the end, only a small number of papers get reviewed at the in-person meeting. By front-loading the discussions, the virtual committee was able to focus on those borderline papers, which resulted in quality discussions. The ACs also came well-prepared for the meeting and were able to provide insightful comments. This resulted in a very efficient meeting, ending well before the in-person PC.

What were the drawbacks of the virtual meeting?

Finally, just as holding the meeting virtually meant that the chairs were able to recruit a new pool of ACs, it also meant that some community members chose not to serve on the Games & Play subcommittee, in favor of a committee that would be held in-person.

The drawbacks of the virtual meeting were not surprising. The biggest negative, particularly since the meeting was held synchronously, were different timezones. This meant that some committee members were forced to stay up all night. (It’s hard to argue that a benefit of being virtual is spending time with family when you have to stay up all night). Not travelling to the meeting also meant that members had to deal with divided responsibilities. It is hard for people, such as employers and family, to perceive you as away when you’re still physically present. In some cases, this meant that members were not completely focused on the task.  

Holding the meeting virtually also meant that members were not able to benefit from the networking and mentorship that happens at the physical meeting. There was also less closure since everyone just hung up and went on with their day once the meeting was over. To compensate for this, there will be a social event at CHI for those committee members who are attending the conference. This could also be improved by building-in networking activities prior to the meeting.

What are some lessons for future iterations?

Pre-test software prior to the meeting, and have a designated technology support person to help with this.

Take advantage of the strengths of distributed collaboration by front-loading most of the work and only discussing the borderline papers in-depth at the meeting.

Find ways to build-in networking activities or other social components.

Chose a chair with strong facilitation skills.

Identify software solutions for unique needs of program meeting (i.e., conflicts, hallways conversations, best papers).

Open questions

With all the positives, it is important to keep in mind that the games community is tightly-knit, and that most of the ACs knew each other prior to the meeting. Games researchers also tend to be more comfortable in virtual environments. It will be interesting see how well other communities fair within the virtual space. For instance, how well will the virtual translate to subcommittees who rely on visual and/or tangible decision-making processes, such as watching accompanying videos or using affinity diagramming?

It is also unclear how much of a difference the chairs made to the success of the meeting. They were extremely effective facilitators, which made the meeting run smoothly and efficiently. Different chairing styles may not translate as well to a virtual environment.

Finally, there are questions around the best way to balance individual privacy with socialization. While using video affords greater opportunity to “get to know” the other committee members since you get to see them, it also potentially impacts one’s privacy. Sharing the same physical space at the in-person meeting differs from opening up your private space to committee members via video.

You might ask yourself, why robots?  We feel that in addition to all of the great talks, a large part of what makes CHI great are the interactions.  Whether wandering through the poster session and chatting with the presenter, bumping into someone you know in the hall, or chumming with new friends at the reception, those interactions are what turns a good conference experience into a great one.  Each year, over 2500 people travel from all over the world to share that experience.  However, some are unable to make the trip due to difficulties with travel accommodations, disabilities, or other obstacles.  Telepresence robots are a first step to providing these interactions for those who are unable to be physically present.  By offering the ability to attend via telepresence robots, we hope to increase access so that more people are able to share the CHI-ness of the conference as active participants rather than as passive watchers.

Two Beam robots chat at CSCW’16. Photo (cc) jofish on Flickr.

Although what we’re doing this year is on the small scale, especially when considering the size of CHI, we look at this is a pilot test of how this may work in the future.  This means that in addition to enabling people to remotely attend, we’ll also be keeping track of how it’s going and lessons learned.  During CHI, we’ll be looking for your feedback on whether it’s working out and after the conference, we’ll be letting you know how it went and what we’ve found so that we can make improvements in the future!

If you are unable to attend CHI this year in-person and this sounds like something you’d be interested in, please fill out this application form [http://goo.gl/forms/G6zECqYVkW] by April 20, 2016.  If you are selected to participate, we will contact you by April 22.  Registration cost will be $75/day and remote attendees will be able to register for the days of their choice, availability permitting.  

A post by CHI 2016 Late Breaking Work Co-Chair Helena Mentis.

There have been some inquiries as to why we changed the name of the poster category for CHI 2016 from “Work in Progress” to “Late Breaking Work”. Below we outline the motivation behind the name change as well as the thought process that went into it. But first, a little history...

History of the CHI Poster Submission Category

The posters category has had a long a varied past toggling between a number of name changes as well as redefinitions as to what was a good poster abstract versus a full paper. Twenty years ago at CHI 1995 this submission category was called Interactive Posters. The ‘interactive’ bit referring to the fact that the “posters are on display at the conference to educate, inform, and inspire conference participants.” More importantly, the type of work that was highlighted as being suitable for a poster was “late-breaking results and work-in-progress.” These two categories of work are what has, in essence, been at the crux of the tension over the posters category at CHI ever since.

For CHI 1997, the category was renamed “Late Breaking Results and Informal Presentations”, which encompassed work where “preliminary results have been obtained and the project members seek feedback from their colleagues in the HCI community” - much like a work-in-progress. A note of interest is that the submission category call highlighted the later deadline in order to “elicit presentations of work that is at once timely and still in a state to be influenced.” CHI 1998 then dropped the “Informal Presentations” bit and just left it as “Late Breaking Results”. However, that year’s call described it as a venue “to report late-breaking results, well-formulated but untested ideas, ongoing work or tightly-focused research or application results” - thus we moved back to the focus on late-breaking work. And then CHI 1999 reverted again to describing it as a venue for “presenting exciting new findings, as well as work that has demonstrated special promise, but that is still ongoing” - in other words, back to work-in-progress.

CHI 2000 saw the name itself change back to the previous version, “Short Talks and Interactive Poster” and the scope of work was also expanded to “exciting new findings, ongoing work that has demonstrated special promise, preliminary results, work that is at once timely and still in a state to be influenced by questions and discussions, or tightly argued essays or opinion pieces.” And even more interesting, the website went on to state that the “goal for this category is to provide a short, concise report of work that is important and likely to have high impact on the CHI community.” Many of us would probably say that that statement is not what we consider to be a goal for the posters presented at CHI and yet this description (and submission category title) stayed on until the next name change for CHI 2004. And even then, although the name changed back again to “Late Breaking Results”, the overall story remained the same.

“Late Breaking Results” was the name of the posters category until CHI 2006 when a number of changes to the conference were instantiated including the increase to a 4-day conference and the first time the posters category’s name was changed to “Work-in-Progress”. It was billed as “an exciting new venue at CHI.” In other words, it was meant to be different from prior instantiations of posters. The chairs went on to “encourage researchers and practitioners to submit work that is underway but still in the early stages and in a state to be influenced by informal discussion in a featured poster session.” Ok, well, not really all that new then, just a different name.

And it has been like this since CHI 2006 - a decade now.

But what really is the point of the Work-in-Progress venue for CHI?

Starting with CHI 2011, it was explained that “the difference between Works-in-Progress and other venues (e.g., papers, notes) is that the work submitted to the Work-in-Progress venue represents work that has not reach a level of completion that would warrant the full refereed selection process. That said, appropriate submissions should make some contribution to the body of HCI knowledge, whether realized or promised.”  

So, the mechanism of review as well as contribution is one of the defining factors of what is a poster and what is a full paper at CHI. Full papers and notes are refereed, which means that the submission is “rigorously reviewed by members of the program committee and peer experts. The process includes an opportunity for authors to respond to referees’ critiques. The program committee may ask authors for specific changes as a condition of publication.” 

The extended abstracts that are presented as posters at CHI are juried, which mean that “content is also reviewed by a committee and external experts in a less rigorous process that does not include an author’s response or conditional acceptance. Juried content is generally not required to make the same level of lasting and significant contribution to our knowledge and understanding as refereed content.” The mechanism of review, then, also influences the magnitude of contribution that a poster or extended abstract could make on the CHI community. In addition, CHI 2011 was also the first time the venue de-emphasized the necessity of it being “early-stage” work.

Our Motivation to Change the Name

Although this history lesson is interesting, our initial motivation to change the name was a reaction to the connotations invoked by the title “Work in Progress”. We felt it is not an entirely accurate representation of the work that has been submitted and presented, nor does it accurately represent the type of work we wish to see submitted. As CHI 2011 and subsequent years explained it, the submissions should make “some contribution to the body of HCI knowledge”. On reflection, most of us would agree that work that is not yet complete is unable to realize its contribution to knowledge. It may be an early finding, but it, in itself, has to make a contribution albeit a small one.

As the CHI 2016 co-chair, Jofish, explained to the incoming WiP/LBW co-chairs: 

for nearly every publication ever published in WiP, ***that’s its final resting place***. We’re fooling ourselves if we claim it’s in progress: it’s the thing, it’s published, it’s done. They’re not passing things en route to a full paper, they’re the real thing. Smaller than Notes and Papers, but fully formed unto themselves. ... **they’re not works in progress***.” 

The work presented may be repackaged or integrated into another larger contribution that is suitable for the impact a CHI paper/note or journal article should have. But that does not mean that that singular contribution that is suitable for an extended abstract could later go onto rest on its own merits as a paper/note or journal article. It would need a more impactful contribution.

In addition, one of the things we must be concerned about is making it clear what the venue entails to people both within and outside the field. For instance, those that are new to our community may struggle getting a full paper into CHI, but the posters category is a great way for people to find their way into our community.

Based on this varied history and the realization that “work-in-progress” does not properly encompass the variety of work that is submitted to this venue, the Conference Chairs (namely Jofish) wanted the new “WiPs” chairs to explore a name change.

So why not just “Posters”? Well, this is mainly because there are many different posters presented at CHI (e.g. workshops, doctoral consortium, student research competition, etc.). But also, “poster” indicates the presentation type but says nothing of the form of contribution. This is the crux of the question we had to answer.

The Name Change Conversation

We decided to use a name that focuses on the benefits of this venue. First benefit is that the publication format is brief, which allows for integration of the paper into a full publication later on. Thus we wanted a name to connote the size of the contribution. Second benefit is that the format of presentation allows us to accept a larger number of works for presentation - thus the quality is still high, but small due to brevity and format. Thus we wanted to ensure we weren’t belittling the quality of the work by insinuating it isn’t finished. Third benefit is that it is work that is ready to be presented closer to the actual conference deadline and so it is likely more 'fresh'. Thus, the venue should encourage new and innovative work and should be of interest to the conference attendees to see what is on the horizon. Another driving factor was to be open to all of our different communities - in other words, not everything submitted to this venue is a research study. Thus “Reports” did not feel right for a design exploration. Thus, we landed on Late-Breaking Work.

How Did This Play Out in Submissions and Reviewing?

Now that the submission and review process is complete, it is interesting to take a look at the numbers. This year, after the review process, we asked the ACs to characterize the papers as either a WiP, LBW, or Other (they could then add a comment as to why they deemed it an Other). They could also choose more than one option. In total, LBWs had 647 submissions. Of the 647 submitted, 281 (43%) were accepted. Table 1 is the breakdown of those submissions by type and Table 2 is the breakdown of type by Accept/Reject.

Table 1. LBW Submissions by AC-described Type (n=647 submissions)

WiP327 (51%) LBW191 (30%) Both50 (8%) Other26 (4%)

Table 2. LBW Submissions by AC-described type by Accept/Reject

TypeAcceptReject WiP117 (42%)210 (61%) LBW111 (40%)80 (23%) Both36 (13%)14 (4%) Other7 (3%)19 (6%)

The submissions we received clearly tended to favor work-in-progress, but we also had a substantial number of late-breaking work as well. There does seem to be a clear trend that work-in-progress were more likely to be rejected than late-breaking work. And those characterized as Other also had a tendency towards being rejected. Papers that were Accepted but deemed a type of “Other” had explanations such as: “Could be a short paper”, “First phase of a bigger, exciting research project”, or “The system itself seems very matured (i.e., paper) but the study is rather a work in progress”. On the other hand, those papers deemed by the AC to be of type, “Other”, and had been rejected had explanations such as: “Technical report/incremental work”, “Small study”, “Literature review”, and “An implementation report”.

As this has never been done before we have no prior data to compare these findings to. However, these results are pretty interesting in and of themselves and could spark some discussion as well as future data gathering initiatives.

We are interested in hearing your feedback on this name change. I’ve done my best to explain our thinking process. And by doing this, future chairs can consider a name change based on these facts instead of having to recreate the story like I did from old CHI websites.

It's time to get cruising on making your CHI slides... to help inspire your imagination, we've updated the "Guide to a Successful Presentation" page to include a few real CHI decks that look great and do a great job telling their story:

https://chi2016.acm.org/wp/example-presentations

We've deliberately picked examples with different styles, from different fields, using different authoring software. There's no one way to make a great deck, but the common denominator is always putting yourself in the audience's shoes and making a deck where you would really care about every slide. The CHI Web page has a great set of tips for making a solid CHI talk, but perhaps even more important than telling you how to make a great CHI presentation is reminding ourselves why it's worth the time to prepare a fantastic presentation.

First, the selfish side of all of us should recognize that a memorable talk is often as good for our careers as a memorable paper. The person who is engaged and left wanting more, who comes up to you after your talk to compliment you and ask questions, could be your next collaborator or your future employer. [Editor's note: our group has hired more than a couple interns based on seeing their awesome CHI talks.]

Second, we should collectively realize that people pay real money to come to CHI, and they want to both learn a ton and have a subjectively good experience. I'll assert without proof (though I don't expect any argument) that we learn way more from a great talk with less information than from a dull talk packed with p-values. So let's give our audience the experience we owe them, and have them all walk away saying that attending CHI was a great investment.

And finally, sometimes folks who are less comfortable on stage may feel that "giving a great, memorable talk is just for those natural performers out there, so I'll just focus on getting my facts across". I'd argue the opposite: yes, there are great, natural performers among us. They can captivate the audience with no slides, or just with bulleted lists in Courier. If that's you, by all means, don't prepare. For the other 99.99% of us, we're all capable of turning someone on to a new field, entertaining our audience, and getting attention that's good for our careers. We just have to spend more time preparing than the elusive natural performer.

So put your time in now... don't get so caught up in getting your results in that you forget how interesting your problem is, and how important engaging your audience is. Make slides that look amazing, or make jokes, or do a live demo, or break the presentation up with multiple speakers, or do whatever else fits your topic and your personality to give a memorable experience.

In short, please make your CHI talk legendary.

In the interest of transparency, we’ve made available all of the (mostly rather dull) communications used to coordinate the CHI 2016 Program Committee (PC):

http://chi2016pc.org

This site includes:

The schedule of what happened when

All the emails we (the chairs) sent to SCs and ACs

The slides we presented at the PC meeting

We hope this helps make the inner workings of the PC more visible, helps solicit feedback on how we can improve the process, and helps remind the community just how much work goes in to selecting a high-quality program each year... in other words, we hope this reminds you to volunteer to be an AC! (2017 volunteering coming soon...)

We are happy to announce that there will be childcare at CHI 2016 in San Jose. The care will be available for children 6 months to 12 years old, with age appropriate care. Initially, we’ll have childcare is open from 8:30am to 1pm and 2:30pm to 6 pm Monday through Thursday; we may be able to add Saturday and Sunday if there is interest. The childcare will be located in a room within the conference center. The carers will be from a professional childcare company, Kiddie Corp, all of whom are CPR/first aid certified and with child care experience.

There will be activities and play materials such as arts and crafts as well as appropriate toys for babies (including baby bouncers). There will be a quiet area for napping with blankets and pillows.

Signup: Signup to register your interest when you register for CHI. Initial childcare slot signup will be approximately 4 weeks before the conference (separate from registration), although it may also be possible to sign up at the conference itself. Priority will be given to providing childcare to children of parents who are presenting during that particular day of the conference.

Price: Childcare will likely be $7 per hour which can be paid in cash or by credit card at the conference. Childcare will also likely be on a day-by-day rather than hour-by-hour basis.

A few practicalities (particularly to international visitor who might not be used to US regulations):

• Parents will need a photo ID for the security procedure for check-in and check-out of children. A driving license with photo or passport is fine.

• Parents are allowed to visit at any time but unauthorized adults are not allowed in the area.

• There will be parent consent/waiver forms to sign

• There will be check-in/out sheets

If you have any questions or concerns email us at [email protected]

-Louise Barkhuus, Judd Antin and Funda Kivran-Swaine

“It is more fun to talk with someone who doesn't use long, difficult words but rather short, easy words like "What about lunch?"”

~ Winnie the Pooh (Pooh's Little Instruction Book, by A. A. Milne)

The CHI conference experience goes beyond stimulating presentations and interactive exhibitions. Be it seeking a mentor, collaborating on a future project, or catching up with friends, CHI serves as a social and communal platform that brings people together.

Diversity and Inclusion Lunch - May 10, 2016, Tuesday

Sign up when you register for the conference!

The CHI 2016 conference will host its first Diversity and Inclusion Lunch. Building upon the tradition of ‘CHI Women’s Breakfast’ at previous conferences, this lunch is both an expansion of scope and deepening of engagement with multiple aspects of diversity and inclusion. These issues include but are not limited to race, ethnicity and culture; age and professional experience; gender identity and sexual orientation; (dis)ability and impairments; religious beliefs; work-life-balance and needs of parents.

This lunch is intended as a social-peer platform to celebrate diversity, discuss challenges, and collaborate on strategies to bolster diversity and inclusion in the CHI community. During the lunch, a list of select speakers will share their personal views and life experiences on specific issues. These talks will highlight everyday challenges and strategies and serve as prompts for table discussions among lunch attendees.

During the lunch event, guests will be invited to join a ‘Theme table’ of their choice to discuss in small table groups on topics concerning diversity issues. Our goal is not necessarily to cover every single issue imaginable but to enrich dialog on these complex issues, strengthen communal ties, and to foster a culture of care, respect, and inclusion among lunch attendees.

More here

CHI 2015 Women’s Breakfast, Seoul, Korea (Photo by Allison Druin)

Volunteer speakers needed:

We are looking for speakers to give short 3 minute talks on Diversity and Inclusion related issues at CHI (no presentation / slide shows). If you are willing to share your personal experiences, challenges, strategies, viewpoints, and success stories on the topic, then we want to hear from you! Please email us ([email protected]) and we’ll get back to you with more details.

If you have any thoughts/questions, please feel free to write to to us ([email protected])

Diversity and Inclusion Chairs, CHI 2016

I’m really happy to be able to announce that, for the first time, students from developing countries are able to apply to the Marsden Fund to support attending CHI (or any other SIGCHI-affiliated conference). You can get up to $3000 to support your travel costs, including registration. You’ll need to have an accepted publication (of any kind, including Late Breaking Work), or doctoral consortium submission, or be an accepted student volunteer.  You’ll also need a note from your advisor. The deadline to email your submission for the CHI 2016 funding round is February 20th at noon PST. There will be no extensions.

I’m particularly happy to have this award available in time for CHI’16.  The SIGCHI Gary Marsden Student Development Fund is named after Gary Marsden, an HCI researcher and pioneer who died suddenly in 2013. 

photo cc-by afropicmusing on flickr

Gary was a great friend and a great person, generous and enthusiastic with his time and attention. I miss his friendship, his support and his vision. He made an enormous impact on the field, and in particular on the growth of HCI in Africa, and I’m delighted we’re able to honor his memory this way.

With CHI on the horizon, the lunch@chi chairs (Hajin Lim and Ali Alkhatib) are excited to announce “lunch@chi”!

It’s an easy, fun opportunity to meet fellow researchers in a small group setting over lunch on the first day of the conference. And best of all, we’ll handle all the planning, making groups based on similar interests and reserving tables at delicious local restaurants!

We’re looking for people to lead these lunches, so if you’re an established researcher and you think you might be interested, fill out the form with your name, email address, institution, and 3 topics of interest; this’ll help us match you with a group based on your interests.

If you’re a student and you’d like to participate, be sure to indicate your interest when you register for CHI.

If you have more questions, feel free to read more about it here.

Looking forward to seeing you at CHI!

If you have any questions, feedback or suggestion, we would like to hear from you. Please contact us at [email protected].

lunch@chi Chairs, 

Hajin Lim, Cornell University

So the PC meeting is on Thursday, and I wanted to see if rebuttals had had an effect on scores. I looked at two dumps of the submissions database from PCS, one from November 18th (before the rebuttals were released) and one from this morning. Here’s the results:

2330 papers examined Average scores went up between rebuttal and PC meeting: 456 Average scores stayed the same between rebuttal and PC meeting: 1650 Average scores went down between rebuttal and PC meeting: 212 Paper was removed between rebuttal and PC meeting: 12

So, yes, rebuttals change reviewer scores. I’m glad to see that.

But there’s a bigger question about whether rebuttals actually change outcomes. That’s harder to measure, and obviously we won’t know full results until after the PC meeting. But as a proxy measure, we can pick an arbitrary number and see how many papers have gone from one side of that to another. For example, if we pick a 3.0 cutoff we can see how many papers used to have a score below 3 that are now above 3:

Average score went from below 3 to equal to or above 3: 45 Average score went from equal to or above 3 to below 3: 31

Another measure might be above or below 2.5, given that papers below 2.5 are unlikely to get discussed at the meeting unless an AC specifies that they should be:

Average score went from below 2.5 to equal to or above 2.5 : 12 Average score went from equal to or above 2.5 to below 2.5 : 24

We’re excited to announce that Workshops have been selected for CHI 2016! This year, we have our biggest workshop program ever, covering a diverse set of topics from long-standing research problems to novel and emerging phenomena in HCI. 

Workshops will take place on the weekend preceding the main conference (May 7-8, 2016) and are scheduled as one-day or two-day sessions. We‘d like to encourage members of the CHI community (and beyond) to apply and participate in workshops. Workshops are an important part of the CHI experience, providing an opportunity for researchers, designers, practitioners, and educators with common interests to come together and engage in focused and interactive discussion.

A preliminary list of workshops has been posted to the main CHI2016 site here. Please check that page for updates over the next few days (hopefully,. Most importantly, once the list is completely finalized, we will add the schedule for workshops, so you know which workshops will take place on which days. In addition, please note that the workshop organizers are hard at work getting their websites online -- not all of the links are active yet, but they will be updated by December 7th.

Some Changes for Participants

We’d like to call your attention to a couple of changes to workshops this year designed to improve the participant experience:

Early Notification Round

This year, we’ve asked organizers to provide an early notification round for position papers. This means that you can submit position papers and find out if you have been accepted to a workshop as soon as December 21, 2015. This change is a response to a desire stated by some community members to plan attendance using 2015 Q4 budgets, rather than 2016 budgets.

If this isn’t a concern for you, then you are welcome to take your time and submit your position paper during the general acceptance round; deadlines may vary by workshop, but will generally happen on or before January 15, 2016.

Sharing Results

One other change this year is a requirement that workshops share their ideas, discussions and/or results in some form. This will vary across workshops from blog posts to videos to articles to special journal issues. We’ve made this change to ensure that the valuable time and and effort you put in as a workshop participant can further benefit the larger CHI community.

See you at Workshops!

Once again, we are extremely excited about this year’s Workshop program (here’s another link, in case you’re scrolling finger is tired). If you have any questions or feedback about this process, please feel free to email us at workshops @ chi2016.acm.org. 

We look forward to seeing you at CHI 2016 Workshops!

Proposal for Artworks: Open call for interactive artworks of any form

Art leads design. Design leads product development. We want to showcase the best and newest interactive art that can point the way to future interaction design and future interactive products. Even if we cannot see the implications of the art, we want to show it and stimulate the community to think creatively, to engage as interaction professionals with interactive art. Beyond that, we just want to show the best interactive art in this first CHI art exhibition.

You may have demonstrated an artwork in the Interactivity section of CHI in the past, but this year there is the new, additional, opportunity of exhibiting in a public gallery devoted only to art. CHI takes on Art and we will be showing the value of our best work in a full art context.

We are calling for exhibits, both for a catalogue and for the exhibition “Inter/Action: digital art that responds” presented by ACM SIGCHI CHI2016 and WORKS/San Jose Gallery: an exhibition that will be open to the public and last six weeks. feature work selected from submissions to CHI and representations of historically significant interactive art.

Launching at the conference, the exhibition will extend until mid-June.

Details: https://chi2016.acm.org/wp/art-exhibition/

So if you haven’t filled out on a poll on this question, go and answer now before you keep reading!

https://www.surveymonkey.com/r/RB73LZ7

Ok. 

Now that’s out the way, let’s talk about this! 

Before we go into any of this data, I want to make one point: all of the predictive models in the world make the same assumption about the world. The assumption is that previous performance predicts future results. And with the number of improvements, optimizations and tweaks we’ve made to CHI 2016, my hope is that we’re going to have by far the biggest and the best CHI ever. 

That being said planning something as big as CHI is tricky. One of the big questions: how do you plan for how many people will show up? Some things you can leave to the last minute - how much coffee to make, how many bagels to provide for breakfast. But other questions are trickier: how much money will you have to support Interactivity? Can you afford to have carpet in the exhibit hall? How many student volunteers should you have? 

So to explore this question, we’ve tried a few different ways to predict the future. 

Crowdsourcing

One popular way to predict things nowadays is crowdsourcing. It’s cheap, and surprisingly accurate. People in the USA may have heard about NPR’s attempt to get people to guess the weight of a cow, and there’s handfuls of books on the topic. 

So I did the same thing: I asked people how many people they thought would come to CHI. And I threw a tweak into the mix. Half the people were just thanked for “answering this brief survey”; the other half were shown this:

Conference (Year): Total attendance Seoul (2015): 2,902 Toronto (2014): 2,997 Paris (2013): 3,443 Texas (2012): 2,616 Vancouver (2011): 2,822 Atlanta (2010): 2,384 Boston (2009): 2,358 Florence (2008): 2,361 San Jose (2007): 2,620 Montreal (2006): 2,250 Portland (2005): 1,947 Vienna (2004): 1,815 Thanks for answering this brief survey.

We put links out to the survey on the CHI Meta Facebook group and on our Twitter account, and sat back and watched.

We had 172 responses in just over 24 hours. The average was 3,318 people; when we cut out the outliers (8 <1000, 4 >5000) we got 3,238. The people who saw historical data predicted 3,342; the half who saw no historical data predicted 3,283. 

You can look at the data yourself, but a few more breakouts:

The people who saw the call on Facebook averaged 3,357

The people who saw the call on Twitter averaged 3,265

People who have been to CHI 5 or more times averaged 3306

People who have never been to CHI averaged 3240;

People who have never been to CHI but saw the table of data averaged 3694 (there’s 19 of these people)

People who have never been to CHI and saw no data averaged 1800 (there’s only 6 of these)

It is pretty remarkable that all of those numbers are within a few percentage points of each other.

More Data

But I’m not really happy basing our spending plans solely on a bunch of people guessing. So I wanted to explore a question:

Can we predict CHI attendance based on the number of submissions?

Again, here’s the data if you want to roll your own analysis. I put a few more columns of data on there, because if you find something which makes a useful measure then more power to you. @ayman​ did some analysis in R, and came up with a prediction of 3841, through a process which he later described to me as “0.75σ+µ. Maybe .5. Some scalar. Aσ+µ.”  

Alternatively, here’s a quick visualization of submission and attendance numbers from the talented @duto_guerra which suggests a 2016 attendance figure of 3269 or so. 

So with all that, it’ll be fascinating to see what the final result is. Feel free to make your own prediction here based on the data above (or based on any other data you feel like); you can see what other people have predicted here. 

Traditionally, we don’t release breakdowns of scores by subcommittees. The reason is basically because we’re dealing with comparatively small numbers of papers across comparatively small numbers of subcommittees, and differences are both pretty trivial, and don’t persist from year to year. We don’t want to give the impression that some subcommittees are “easier” and some “harder” to get accepted in, because, well, they’re really not.

With all of that in mind, here’s a look at the breakdown of average unweighted-by-expertise paper review score. What you’ll notice is, in fact, how very similar they all look. Everything has a median and mode in the range of 2.5-2.7 or so, which is pretty damn similar.

I wanted to dig a bit into why some ACs felt like they had quite so much work to do in writing their 2ACs. We had carefully designed the process so it'd be relatively low amounts of extra work, but then some people felt they really had a lot to do, and that seems just strange. Basically, if you've got, say, 12 papers to 2AC, and we have a 25% acceptance rate, then it'd be a bit surprisingly if you had more than, say, 6 2ACs to write. We have 2331 papers & notes in the system right now, of which 1133 received 2AC reviews, or about 48%. 

So I crunched three sets of numbers to try and understand this a bit more. The first is the total number of 2AC reviews individual ACs wrote. 

Min. 1st Qu.  Median    Mean 3rd Qu.    Max.  1.000   4.000   5.000   5.519   7.000  13.000

Ok, that's rough! Some people wrote 13 2AC reviews? That's crazy! Remember to buy your AC a drink. Oh, you can’t, because they’re anonymous. Just buy someone with a Program Committee ribbon on their CHI badge a drink, and let’s assume it’ll work out.

Now let's look at the number of 2AC reviews they were *required* to write, on the basis of either score (>=3.0) or score and SD (2.5<=score<3.0, sd>1.3). 

Min. 1st Qu.  Median    Mean 3rd Qu.    Max.  1.000   3.000   4.000   4.313   6.000   9.000

So pretty much as expected. Most people wrote 4 2AC reviews. That seems about what we were thinking. Some people were unlucky, like that 9 review person, but in general, we're in pretty good shape. Just so we’ve got the data here:

There were 983 “required” 2AC reviews. Of these:

134 2AC reviews were on papers with scores >=4 (these are usually just scores with no reviews)

810 2AC reviews were on papers with scores >=3 (including >=4)

472 papers had 2AC reviews because the 1AC requested discussion

50 papers had 2AC reviews because their score was between 2.5 and 3 but they had high standard deviation, indicating that some reviewer thought it was valuable.

And note that all of these may overlap.

Now, you were allowed to write a 2AC review for any paper you wanted to, which I think is really important. That's part of being a good 2AC. But this starts to look really strange to me:

  Min. 1st Qu.  Median    Mean 3rd Qu.    Max.  0.000   0.000   1.000   1.206   2.000   8.000

And that's where I think we're running into problems. Look at that graph. Nearly everyone wrote 1 or 2 optional 2AC reviews. Great! That's the sort of due diligence I want to see.  But 9 people wrote an additional SIX or more 2AC reviews.  

That's a huge amount of work, and we know we’re seeing complaints about AC overwork. The authors may well appreciate the extra review. But, overall, it suggests to me that, basically, some ACs were writing 2AC reviews that they didn’t have to. Some ACs may have felt that that was their responsibility as an AC; others may have been confused about what was required.