#stats-web

Now that we’ve published a lot of our findings about the data we saw over the open studio weekend, we are releasing all of the feedback we received when we asked participants to “Share Your Story.” All of the feedback is now available in the Shared Stories area of the GO website; we’ve tagged stories consistently, so you can easily discover what themes emerged. You can also search by participant type (artist, voter, volunteer) and neighborhood, so it’s possible to see the feedback coming from participants in one area and compare it to experiences in another.

As we reviewed the feedback, Sharon and I were struck by how much participants talked about “discovery”—one of the largest tags in the cloud. This feedback has been overwhelmingly positive and inspiring.  If you take a look, you’ll find participants discovering so much about their communities and neighborhoods and expressing their gratitude for the project.

Voters canvassed their neighborhoods finding familiar faces in places that were not accessible prior to GO; artists reported a greater awareness of others working near them, often in the same building or just down the street. Many participants felt GO facilitated a greater understanding and a coming together that wasn’t present before the project.

In our previous posts, we discussed much of the feedback we’ve seen. The data suggests that while hot spots existed, the traffic during the open studio weekend didn’t necessarily align with people’s assumptions. We discussed the weekend’s hours and participants desire for a multiple weekend event. Plus, we received plenty of feedback about technical problems, our app, our website, and even our printed maps; we’ve talked about this quite a bit.  Despite all these hurdles, you’ll find that almost every story says something along the lines of “We must do this again.”  

This is the final post in a three-part series about the use of mobile technology during the GO. During the open studio weekend we wanted to provide as many mobile options as possible, but simplification could have helped us a great deal. We spread ourselves too thin in a quest for accessibility. Despite the comment below and similar sentiments shared via other forms of feedback, Sharon and I still believe the high bar is necessary, and we wouldn’t change the requirement of registration and seeing many studios in order to nominate; we also believe that removing the nomination process from the weekend activity allows for needed reflection. However, within this set of requirements, we can and should simplify the means to get there.

As I look at these posts about texting and iPhone use, there are many lessons to be learned. If I had to do this year all over again, we would ditch texting in favor of telling people to write down codes, and we would have eliminated mobile web in favor of more staff time allocated to the iPhone app. Though we ran into unforeseen problems, I think having more time could have only benefited us and the additional time would have meant having more hands on deck when we started to see issues crop up because we would have been supporting fewer options. As a powerful visualization, take a look at a detail of door signs which were on every artist’s door; this is the area of the sign that tells people how to check in to a studio and I’ve added arrows for emphasis:

This is the second in a series of three posts about the use of mobile technology during the GO open studio weekend. The first looked at basic trends and SMS texting; this post will delve into the iPhone app, and the last will consider general lessons learned. Few people know that we almost pulled the iPhone app from the store right before open studio weekend. Late Friday afternoon, the app was rendered totally unusable as it crashed at every tap and we wondered if it would be a better user experience to pull the app in favor of using other less crash-prone methods to check in.

As it turns out, for as much testing as we did, the one thing we couldn’t replicate easily was the load on our servers going into the open studio weekend and the app wasn’t handling spikes in server traffic very well. After a very late night we managed to stabilize the situation prior to the open studio weekend, but not before we frustrated a lot of participants.

I wish that was the extent of the problems we faced, but there was more. At first load, the app would check to make sure its list of studios was up-to-date. In the week leading up to the open studio weekend we had about a hundred artists cancel their participation, so the app had a lot of work to do to update. You can imagine what’s next...yup...crash. Once the app crashed on the initial startup, it wouldn’t crash again, but this was on the heels of the Friday night issue and there was plenty of frustration with no easy way to tell users to hang in there.

We went into the weekend knowing there were problems, but that most had been resolved or would be resolved in good time. We had no idea if participants would stick with us or ditch the app in favor of the other platforms we offered, so we held our breath and waited to fight more fires, but I remember seeing this tweet at some point over the weekend and feeling some sense of relief that someone out there was having a good experience.

There were some additional positive experiences around the app that came as part of the “share your story” feedback, but it felt as if these were the exception to the rule. So, I was floored to see this big, blue sea of check-in data in the chart below—despite all the trouble, this app was used quite a bit. Given this popularity, we are not sure if the app worked mostly as intended or if people used it with continued trouble. Were the people having trouble just louder about it or was their experience indicative of most users?

This is the first in a three-part series about the use of mobile technology during the GO open studio weekend. Today, we’re going to look at basic trends and talk in depth about the use of texting; later this week we’ll look specifically at the iPhone app and lessons learned. When we first looked at platform use during the open studio weekend, we were a little surprised at the data; when voters checked in to studios, they used the website almost half the time.

As we started to look at the data over time, we started to understand the trends a bit more. You can see from the chart below, the iphone app dominated during the open studio hours and web traffic started to increase substantially at the end of each weekend day and on Monday, during the check-in grace period. You’ll notice at the exact times we saw a drop off in traffic to studios (6pm), there was a rise in web traffic indicating people headed home to input the artist codes they had collected throughout the day.  

Of all the feedback we’ve received, issues around GO’s hours really bubbled to the top. A common refrain from voters and artists alike was one weekend was not enough; so many people have suggested we split the event up over multiple weekends. Sharon and I are sympathetic to these concerns; each of us created overwhelmingly large itineraries and spent the weekend seeing as many studios as possible throughout the borough, but for as many studios as we managed to visit, we couldn’t see them all. However, of all the changes we may consider with possible future iterations of the project, the idea of splitting up GO over multiple weekends isn’t one of them. 

When planning GO, we discussed the idea of a multi-weekend event at length and, in the end, decided against it. A multi-weekend event would mean breaking Brooklyn into zones because neighborhood boundaries are too fuzzy to be helpful in this scenario; zoning seemed antithetical to the project. If we did split things up, we worried about issues outside of our control creating greater disparity and unfairness. The bad weather we experienced on Saturday or the issues with MTA service are both great examples of this. It’s one thing if you have bad weather and we all experience it together, but it’s a totally different story if the bad weather falls on weekend your neighborhood has open studios when another neighborhood benefits from perfect conditions. Of course, we could run the project over multiple weekends throughout the entire Borough, but we didn’t think it would be realistic to ask artists to be in their studios for more than two days. Speaking of artists, we heard time and time again that the open studio weekend hours—11am to 7pm for two consecutive days—was just too much. As we look at what worked and what didn’t, we wanted to dive into the check-in data to see if GO’s hours worked for both artists and voters. David Huerta, one of our web developers, pulled together the data so we could take a look.

With the artist nominations just announced, we wanted this next post to take a critical look at the nomination data and consider the results. Remember this heatmap of check-ins over the weekend?

At first glance, most people looked at this map and assumed the majority of nominees would be coming out of the hottest of hot spots, but the resulting nominations are surprising because they run counter to these assumptions. Compare to the heatmap of nominations:

You can see what’s happened here—nominations were dispersed throughout. Even though an artist may have been in a high density building or even a hotspot, the nominations in those areas were dispersed among the many studios open. Was there something in the traffic patterns and the resulting metrics that meant a confluence of events could occur—was there a sweet spot?

Corinne Hoener, one of our web developers, is going to show us a couple of examples:

There were 1708 artists in 758 buildings with open doors for GO. Of those buildings, 188 housed multiple studios. If we look at the data on our top ten artists, six of the ten were in multiple-studio buildings, and five of those six were in buildings with 8-15 studios, such as 925 Bergen and Invisible Dog (both had 15 studios). Naturally, this led us to wonder if 8-15 studios is a “sweet spot” in terms of having enough studios to draw visitors to your building, but not so many that visitors are overwhelmed by choice. If we crunch the numbers, it certainly looks that way. On average, an artist’s studio saw 30.35 check-ins, but artists in buildings with 8-15 studios had higher than the average with 44.09 check-ins. On either side of this sweet spot, there were fewer check-ins; artists in buildings with fewer than 8 studios had 22.92 check-ins and buildings with more than 15 saw 40.94 check-ins.  Studios in buildings with 8-15 studios also received about two more nominations than average. That might not sound like a lot, but with tight margins like we have, two additional nominations can make all the difference. Speaking of nominations, it’s important to remember that an individual visitor can check-in to as many studios as they want, say 100 or more, but they can still only nominate up to three artists. So if we’re digging into the numbers behind our top ten artists, we have to consider the number of visitors to their studios. And again, we can see a sweet spot effect for those buildings with 8-15 studios.  Among our top ten artists, studios in buildings with 8-15 studios saw on average 223.25 unique visitors. These buildings had, on average, 12 studios in them, so there is an average visitor-to-studio ratio of 18.6. Contrast this with two mega studio buildings, which had an average of 72 studios in them and an average of 429.5 visitors—nearly double the average visitors that the studios in 8-15 buildings saw. However, since they also had about 6 times the number of studios, that leaves them with an average visitor-to-studio ratio of only 6.0.  There were far fewer visitors relative to the number of studios in these major buildings. And as we pointed out earlier, fewer visitors means fewer potential nominations. There’s also some interesting data related to intra-neighborhood traffic, the visitors going to see artists in their own neighborhoods.  Across all the neighborhoods, an average of 32.69% of check-ins came from visitors who lived in or near that neighborhood. However, across the nine neighborhoods represented in our top ten, on average 42.07% of check-ins came from people who live in or near those neighborhoods. In fact, seven of those nine neighborhoods had intra-neighborhood percentages that were above average. It’s possible that this factor combined with a sweet spot (or even the high intra-neighborhood traffic alone) can help to boost nominations in a rally effect. Intra-neighborhood stats have now been added to the raw data, so you can take a look.

We should recognize that the artists who are in the top ten are there for a reason. Sweet spot or no, many artists were nearby (some even in the very same building), but it’s the work of these ten that resonated. Also, we didn’t see either sweet spot traffic or the rally effect cause a “sweep,” or dominance of one area or building in the nominations. Even so, as Sharon and I think about possible future iterations of the project, we wonder how can we avoid the sweet spot and continue to level the playing field. Simply put, in order to lessen the effect of sweet spots, we need a critical mass. With 1,708 artists, we think we’d need at least 17,000 eligible voters going to at least five studios each.  Though we estimate we had 18,000 voters, only 4,929 were eligible to nominate artists; many of our participants were just visiting studios without checking in or nominating.  We both feel that the high bar to become an eligible voter is critical and non-negotiable in future; we would also want to continue encouraging all kinds of participation (voting and non-voting). Given what we saw this year, we think the metrics to get us to statistical significance and wisdom of crowds decision-making would be a tough uphill climb at any point.

As Sharon looked at the results, she found interesting work not only at the top, but also among the middle of our long tail (and beyond), so as we think about future iterations of the project, we find ourselves asking about participation models that could surface even more work for consideration.

We expect to announce the nominated artists tomorrow; we're running just a little bit behind schedule. In advance of announcement, we wanted to share some of the neighborhood data and think about what happened over the open studio weekend in context and in relation to some the feedback we’ve received. Throughout GO, participants have expressed concern about the process being unfair. We’ve been asked if we’d weight results or adjust data to create an even playing field, but our answer has always been that we’d let the data stand as is; we simply didn’t know enough about traffic patterns or participation from the outset to create a model that would work for everyone. We felt it was best to let the weekend play out and look carefully at the resulting data to see if we would need to make adjustments in possible future iterations of the project. That said, we expected things to surprise us and they did. Many participants were concerned that studios in low-density areas (i.e., not typical “artist hubs” like Gowanus and Bushwick) would receive less foot traffic and thus be at a serious disadvantage in an event like GO. If you look at the neighborhoods with the most check-ins—Gowanus (6442), Red Hook (5800), Sunset Park (4751), Bushwick (4128), Greenpoint (3959)—at first the findings seem to align with these concerns. However, if we dig just a little more deeply, we find the data reveals far more nuanced results. 

Gowanus, Red Hook, and Sunset Park had something in common; each had many studios, most within easy walking distance and some buildings with large concentrations of artists (Brooklyn Art Space, Screwball Spaces, and Chashama, respectively). As a result we saw a higher than average rate of check-ins per studio; basically, a visitor could go into one of these neighborhoods and rack up a bunch of check-ins all at once. However, those same areas were below average in terms of the amount of visitors in relation to number of studios open. In other words, even though there were a lot of studios with high numbers of people checking in, the ratio of visitors to open studios was relatively low in comparison to other areas.

As you saw in Shelley’s previous post, we were thrilled about the level of participation over the open studio weekend. At the same time, we remained anxious about how many of the 4,929 eligible voters (those who checked in to more than 5 studios) would continue their engagement into the nomination phase. We anticipated that we would lose some people, but again, as has often happened with GO, we have been pleasantly surprised with the results. Our nomination period ended on Tuesday night. We received 9,457 nominations over seven days. Of the eligible voters, 78% of them took the next step to recommend artists. While we allowed visitors to select up to three artists, many chose fewer—23% nominated 1 artist; 10% nominated 2 artists; and 67% nominated 3 artists. Perhaps the most exciting aspect of this phase, however, has been that many visitors also took the time to leave provide comments and feedback. Using the virtual guest book feature on the website, visitors left a total of 2,289 comments for artists, ranging from short notes such as “great work” to longer, detailed ones about experiences in the artists’ studios. We have also received 361 responses via “Share Your Story.” These have run the gamut from surprises and disappointments to troubles with technology and the MTA, but almost always offering advice if we repeat the project. We will be releasing this feedback in its entirety on the website, but in the meantime, we are highlighting selected stories on the homepage. So, what happens next? We will announce the names of the top ten nominated artists on or about September 25. We are looking at the results with a careful eye (and some fancy programming) to ensure that each artist who has been nominated is eligible according to our terms. I will then visit those artists’ studios along with a curatorial colleague and select the work for the exhibition. The featured artists will come from those nominated by the community, but the focus and structure of the show (particularly what themes may emerge from conversations with the artists as well as how many artists will be included) is all up for grabs.