FanFiction Data Science

EDIT: Hey folks! Thanks for the feedback on our previous post. Our vague wording may have misled people as to our intentions, for which we apologize. We are currently taking a step back to rework how best to communicate our intentions in a way that is the most sensitive to the needs of the community. Thank you for your engagement! We have closed the survey link for now, but here's the survey description for future reference:

What is it about our human-human interactions that makes creatively writing together so compelling and unique? What about that is completely lost in human-AI writing interactions? Do you think ChatGPT is bad at creative writing? Do you have big feelings about any of the above topics? Then keep reading! We are conducting this study because we are interested in analyzing collaborative creativity methods among fanfiction writers. The goal is to criticize the current state of AI-assisted creative writing and offer suggestions from seasoned creative writers on how it could be improved and designed to actually help the people it affects.

First off: what does human-centered mean? The goal of human-centered research is to design technologies based off of HUMAN interactions, and these technologies should be for HELPING humans without replacing, displacing, or marginalizing them. 

If you've tried interacting with AI tools like ChatGPT...you might notice they're unhelpful, and even outright bad, when it comes to writing creatively. The goal of this research is to find out: Do people even want them to be helpful? CAN they be helpful in any way? Is it impossible for AI to produce creative writing that can hold a candle to anything a human could write? Why might it be impossible?

There’s a lot of research being done in this area that is not very human-centered - it involves making AI tools for creative writing and then asking people how they feel about them, instead of the reverse. We believe that a better approach would be to ask people how they feel about AI tools and whether or not they can be helpful, and propose design guidelines based on that. We believe that this is particularly relevant to fanfiction authors: due to how AI tools are trained, a large proportion of the dataset for AI-based creative writing is likely comprised of fanfiction, due to how much of it there is on the internet. 

We’re looking for fanfiction authors aged 18 and above who co-write fanfiction with one or more collaborative partner(s). This can be short-form (co-writing one-offs, single chapters) or long form (co-writing entire fics, long-term collaborations) - we’re essentially interested in the methods that you and your collaborators use together to produce works of creative fiction.

The provided survey will take approximately 15-20 minutes to complete. If you’re interested in telling us more, you can sign up for a 30-45 minute interview at the end of the survey. Ideally, you and your writing collaborator(s) would be able to attend this interview together. Every interview participant will be compensated with a $10 gift card.

All parts of this survey were approved by the University of Washington Human Subjects Division Institutional Review Board (IRB) to ensure the protection of your rights and welfare as you take this survey. Your responses will be kept confidential, although we may publish aggregated results. You may exit the survey at any time.

If you have questions, comments, or concerns, reach out to [email protected]. 

------------------------------------------------------------------------------

Hello! We are researchers at the University of Washington Human-Centered Data Science Lab, and we are studying modes and methods of collaborative creating writing.  We’d love to have you participate in our study!

Survey link: [now closed]

We’re looking for fanfiction authors aged 18 and above who co-write fanfiction with one or more collaborative partner(s). This can be short-form (co-writing one-offs, single chapters) or long form (co-writing entire fics, long-term collaborations) - we’re essentially interested in the methods that you and your collaborators use together to produce works of creative fiction. The eventual goal of this work is to suggest more human-centered guidelines for AI-based creative writing tools.

The provided survey will take approximately 15-20 minutes to complete. If you’re interested in telling us more, you can sign up for a 30-45 minute interview at the end of the survey. Ideally, you and your writing collaborator(s) would be able to attend this interview together. Every interview participant will be compensated with a $10 gift card.

All parts of this survey were approved by the University of Washington Human Subjects Division Institutional Review Board (IRB) to ensure the protection of your rights and welfare as you take this survey. Your responses will be kept confidential, although we may publish aggregated results. You may exit the survey at any time. For questions about our research, contact Nisha Devasia at [email protected]

Hello! We are researchers at the University of Washington Human-Centered Data Science Lab, and we are studying modes of and preferences towards content recommendation.  We’d love to have you participate in our study!  

Survey Link (https://docs.google.com/forms/d/e/1FAIpQLScB96LBCcB3Voe-cxcrQN2mqEtHL0-69sgicAM1f_t_qfE9lA/viewform)

We’re looking for fanfiction readers and authors aged 18 and above. We’re interested in hearing from anyone who actively engages in AO3's folksonomic recommendation system (using things like Bookmarks or recommending content to other friends etc.) and also uses platforms like TikTok, Instagram, Twitter etc. This screening survey will take approximately 5 minutes to fill, and if you’re eligible, we will contact you for a 1-hour interview.

All parts of this survey were approved by the University of Washington Human Subjects Division Institutional Review Board (IRB) to ensure the protection of your rights and welfare as you take this survey. Your responses will be kept confidential, although we may publish aggregated results. You may exit the survey at any time.

For questions about our research, contact Sourojit Ghosh at [email protected]

Hello! We are researchers at the University of Washington Human-Centered Data Science Lab, studying questions of monetary compensation for fanfiction authors and readers. We’d love to have you participate in our survey!

Fanfiction Survey Link

We’re looking for fanfiction readers and authors aged 18 and above. We’re interested in hearing from anyone who has either received compensation for content they have authored, or paid fanfiction authors for their work. This screening survey will take approximately 5 minutes to fill, and if you’re eligible, we will contact you for a 1-hour interview. You will be compensated $20 via online gift card.

All parts of this survey were approved by the University of Washington Human Subjects Division Institutional Review Board (IRB) to ensure the protection of your rights and welfare as you take this survey. Your responses will be kept confidential, although we may publish aggregated results. You may exit the survey at any time.

For questions about our research, contact Sourojit Ghosh at [email protected]

Every review on fanfiction.net has an associated timestamp telling us exactly when it was posted, or so we thought. When trying to find the hours of peak review activity across different fanfiction.net fandoms, I saw some fandoms with very uneven review count distributions (shown below). 

What made this even more confusing was that some of the fandoms had much more pronounced spikes at 7 and 8 UTC. I compared the fandoms with large spikes to those without. I noticed that ones with spikes tended to be fandoms with many reviews in the early 2000s. I wanted to look at how the distribution of review time changed over each year. I chose to make a heat map of the average daily distribution by year. I did some data wrangling so that I could put year on the Y axis and hour of review on the X- axis. Below is the result.

First Exploratory Visualization:

The resulting visualization made the situation much clearer. Every year should look like the ones between 2012 and 2017, where reviews are relatively evenly spread across the day with small variations at peak hours.  Before 2012 we see very different behavior. Around 60% of reviews have a time stamp of 7 UTC, and the other 40% have a timestamp with 8 UTC. We see absolutely no reviews with timestamps for other hours. 2011 is a unique case where we have almost all reviews posted during the 7-8 UTC with less than 1% posted at other hours. To get a closer look at 2011, I filtered the data to just 2011 and used months instead of years on the Y-axis.

Second Exploratory Visualization:

This graph reveals two important clues as to what is wrong with the dataset. The first is what the split between 7 and 8 values is from. 7 and 8 values each have a specific time of year where they are the only review time, switching in March and November. I realized that something else important happens with dates in March and November, daylight-savings time. I looked up the daylight-saving times for 2011 and it was from March 14th to November 6th we see those dates reflected exactly in the data. We even see that March is evenly split because the 14th is close to the middle of the month. In November  the 6th is close to the beginning so we see an uneven split. The other thing that this graph shows us is that at some point in December the dates started to match the expected values. To get the most accurate value of the date that this happened I had to switch to looking at the day instead of month, and found that on December 27th all dates are 0 UTC and then after that they seem to be accurate to the minute.

Now that I had the issue clearly defined, I had to figure out why this was an issue in our data in the first place, and hopefully fix it. Instead of exploring our collected data, I saved a lot of time by going right to the source. I went on fanfiction.net, found some old reviews, inspected the webpage to find the UTC time stamp, and converted the timestamps into datetimes. I found that all the old reviews on the site were either 7 or 8 pm. I wasn’t able to find an exact reason that the site is inaccurate, but I believe that when the ff.net backend was built in 2000 they decided to save some hard drive space by only saving dates by the day. 

Here is 2015, a typical year, showing what the review distribution should look like. We can see people reviewing later in the day during summer and winter break. 8 UTC is midnight Pacific and 3 am Eastern, we see the lowest usage during the three following hours when many of the US reviewers are sleeping. 

While creating these visualisations I learned: 

1. Visualise both before and after processing data. Before calculating the month from the date, the visualisations didn’t discover the inaccuracies. After splitting date into the year and hour of the day variables, the visualisations showed the problems with the dates.

2. Look deeper if something seems weird. When I first saw the problem by accident I almost dismissed it. Going off on a tangent ended up making a discovery that will be helpful for future research with this dataset.

3. Creating a presentation can help with findings. When creating a presentation to the group I built an interactive version of the graph (linked below). The interactive version showed that the data was missing a lot of reviews from when we were scraping the site in late 2016 to early 2017, another important thing to know when using this dataset.  

You can see the code to create the visualizations here: https://travisneils.github.io/dates/dates_vis.html

Hello! We are researchers at the University of Washington Human-Centered Data Science Lab investigating people’s participation in online fan communities like Fanfiction.net to better understand how people form communities in online environments. We’d love to have you participate in our new survey.

Fanfiction Survey Link

We’re looking for Fanfiction.net users aged 13 and above. We’re interested in hearing from anyone who has used Fanfiction.net to read, review, or post fanfiction stories. You don’t need to be a current user of Fanfiction.net - we’re also interested in hearing from people who used the site in the past. The survey contains 14 questions and you are not required to answer every question. 

All parts of this survey were approved by the University of Washington Human Subjects Division Institutional Review Board (IRB) to ensure the protection of your rights and welfare as you take this survey. Your responses will be kept confidential, although we may publish aggregated results. You may exit the survey at any time. 

For questions about our research, contact Niamh Froelich at [email protected].

A study on fanfiction stories’ update frequency and number of reviews received

As a grad student, I often find myself debating over finishing tasks all at once or spacing them out over a reasonable time period. In the fanfiction community, we have seen stories where multiple chapters are posted on the same day, while others are updated every few months or even years. As authors, if our goal is to attract readers and reviews, how long should we wait between chapters? Is it better off to satisfy our readers with content all at once, or to keep them hooked by posting a bit at a time?

Our Approach

To answer these questions, we defined the “frequency” of updates in a story as the average number of days between chapters posted, and looked at stories with more than one chapter and at least one review from fanfiction.net during the period of 1997 to 2017. In this particular study, we are considering the first story posted by each author to avoid miscalculating the accumulated review count for their subsequent stories. Note that the original chapter publish date/time was not available in the dataset so researchers estimated it from either the story publish time or time of the first review. As a result, this dataset is representative of stories with more reviews. 

What We Found

In this graph, each data point is a story mapped to the total number of reviews received (y-axis) and average days between chapters posted (x-axis). The x-axis is then divided into 14 bins, to represent “buckets” of stories where chapters were posted every 1, 2, 3… 14 days on average. While there are quite a few stories with up to hundreds of reviews, the median line plotted for each bin indicates that the data is skewed to the right. 

My initial guess was that stories with chapters posted 3-4 days between each other might receive the most reviews, as readers are likely to revisit the same story for updates every few days. This graph seems to be consistent with this speculation and shows that the first peak is at five days. This means that half of the stories with chapters published five days between each other are observed to have 9 reviews. Other peaks are observed at ten and thirteen days.

Your Thoughts?

Author: Sourojit Ghosh

As a creative writer myself, I’ve always been anxious about getting reviews on the content I put out there. As I’m sure others who publish any form of writing can attest to, reviews form an integral part of our development as writers. However, I also find myself paying attention to not just what a review says, but also how it is said. Specifically, the emotions expressed in a review often shape my interpretation of it.

With that in mind, we at the University of Washington Human-Centered Data Science Lab (UW-HDSL) are interested in researching the emotions present in the multitude of reviews by the fanfiction community. By investigating a correlation between the lengths of reviews and the emotions expressed in them, we aim to understand the growth of relationships between members of the community as they share likes and dislikes.

Introduction

Our previous research with the fanfiction community has found widespread encouragement for budding relationships in its distributed-mentoring setting. The members of the community, mostly young adults from all over the world, are incredibly expressive in their words and often eager to support each other in the writing process. Most of the reviews we have seen in the community are rife with emotion, with the words jumping off the page with their expressiveness. This collectively supportive environment not only seeks to bring out the best in each individual but also to form meaningful relationships that extend beyond that of anonymous writers and readers of fanfiction.

Methods and Findings

For this exploration, we examined 1000 reviews of various fanfiction stories published on the site. We decided to classify them as exhibiting one of 11 emotions: Like, Joy/Happiness, Anticipation/Hope, Dislike, Discomfort/Disgust, Anger/Frustration, Sadness, Surprise, Confused,  Unknown, and No Emotion. Figure 1 shows an example of a review coded in this way using TextPrizm, a  web tool developed by members of the UW-HDSL.

Figure 1: An example of a review being coded for emotions

By coding these reviews for emotions, we are trying to gain a better understanding of the trends in emotions expressed by reviewers across the community. By identifying such trends, we hope to learn how relationships are formed between users sharing common interests and having similar reactions to certain content.  

Figures 2 and 3 display our preliminary results so far. Figure 2 represents the number of reviews being classified as having each emotion, while Figure 3 shows the average lengths of reviews in the dataset expressing each emotion.

Figure 2: A bar graph showing the no. of reviews each emotion was assigned to.

Figure 3: A bar graph showing the average no. of words in a review expressing each emotion.

The high number of reviews expressing Joy / Happiness and Like is an encouraging indication of the fact that most users took adequate time to express their positivity and support towards the writers. Another emerging trend can be seen in the reviews marked as No Emotion. This small number of reviews averaging at about 80 words per review was found to contain thoughtful discussions on global issues like religious tensions and sexual violence. While the previously discussed reviews highlight the positivity inherent in the community, these reviews remind us of the incredible maturity and depth of thought that the members also possess, a fact even more inspiring given that the community is mostly comprised of young adults.  

Conclusion and Future Work

This initial examination of a small set of reviews offers some insight into the correlations between emotions and review length. An exploration of a larger set of reviews may offer some basis for providing statistically significant findings along the lines of the currently observed trends and can provide further insight into the ways in which reviews are integral in the process of users on relationship building on Fanfiction.net.

We would love to hear from you, members of the fanfiction community, about what you think of our work and how you view the emotions expressed in reviews of your writing. At the same time, we would also be interested in knowing if you express certain emotions in your reviews more extensively than others! If you have any questions or concerns about our data, feel free to respond to this post or send up an ask, and we would be happy to get back to you. And, as always, stay tuned for our future work with your wonderful fanfiction community!

Acknowledgments

A time-shifted serial correlation analysis of reviewing and being reviewed.

Acknowledgements: Investigation by Arthur Liu with thanks to Dr. Cecilia Aragon and Jenna Frens for feedback and editing and also to team lead Niamh Froelich.

Is it true that giving someone a review will make that person more likely to write reviews as well? Conversely, is it true instead that writing more reviews yourself will help you get more reviews from others?

In this post, we explore one avenue of reciprocity by analyzing the time series of reviews given vs. reviews received. 

Of course, you have to be careful with this technique. The inspiration of the analysis we utilized comes partly from Tyler Vigen’s Spurious Correlations site (http://www.tylervigen.com/spurious-correlations) where he shows interesting correlations between clearly unrelated events. With a humorous perspective, he reminds us that correlation is not evidence of causation (since sociology doctorates and rocket launches are totally coincidental), but the analysis techniques here are an interesting technique to investigate potential relationships between two different time series.

Back to our topic of reciprocity, we wanted to investigate the relationship between reviews given and reviews received. We had two hypotheses that we were interested in testing: first, we were curious if users who received more reviews would be more inclined to give reviews themselves. Second, we were curious if giving reviews would help increase the number of reviews you personally received.

To get into specifics, here is an example plot of a real user’s review activity.

Let’s break it down. This plot follows the activity of a single user over the course of several years. It plots the total amount of reviews that they gave (in red) and also the total number of reviews that they had received on their fan fictions (in blue). What this chart shows us is that this is a user who has had a very consistent amount of activity in terms of giving out reviews. It also captures spikes in the number of reviews received (blue) which may correspond to having released a new chapter.

If there was a strong link between reviews given and reviews received in either direction, we would expect to see that increases in one is followed by increases in the other. Here is an example where we witness such a relationship:

Since it is harder to analyze the change in activity level from these cumulative plots, we then looked at the total number of reviews given each month. Here’s what that looks like for the same person:

This time, it is more apparent that there is a similar pattern in the activity behavior for the reviews given and reviews received. For this example, that similarity is a similar spiking pattern.

From Vigen’s website, we could naively apply a correlation calculation here, but there is a glaring flaw: one of the time series is clearly ahead of the other. So, what if we just shifted one of the time series so they overlapped and then computed the correlation? This is the basic intuition of serial correlation: we apply a range of possible shifts and then compute the correlation between these shifted graphs. The one with the highest correlation would be the one with the best match.

The results for different shifts:

The best shift of “11 frames”:

In other words, for this person, giving a lot of reviews correlates well with receiving a lot of reviews roughly 11 months later. Of course, this doesn’t prove any sort of causation, but we can speculate that the increased amount of reviews this user gave helped boost the amount of reviews they got later!

From this analysis of an individual person, we were curious how this extended to the larger community to see if these same trends existed! The short answer, “eh, not really,” but it is interesting to see why this cool pattern might not generalize adequately.

1. Not all individuals get reviews and give reviews at the same scale

Some users just like to give reviews and some users just like to write reviews!

For instance, here is someone who gives a lot of reviews and didn’t get many themselves.

Here is someone who gave some reviews, but then focused on writing stories and received a lot more reviews instead!

For graphs like these, it is hard to apply the analysis we did earlier because the relationship is likely a lot weaker or there might just not be enough data points to capture it anyway.

We can summarize these examples for the overall population by looking at the ratio between reviews given to reviews received.

For this sample of 10k users, we see that those who primarily receive reviews will have a larger ratio (right), and users who primarily give reviews will have a smaller ratio (left). In more detail, a ratio of 1.0 means that they only received reviews. For example: 10 reviews received / (10 reviews received + 0 reviews given) = 1. For a ratio of 0.0, it means they received no reviews. For each ratio, the graph shows the total count of the 10k users who had that ratio.

To address issue (1), we reduced the scope down to users who had a relatively equal ratio of reviews given vs. reviews received.

Additionally, we pruned for users who had received at least 10 reviews. This way, we would have enough data points to use for our analysis. In fact, this is also why there is a large spike in the 0.5 ratio which consisted of a lot of users who had written one or two reviews and received an equal amount.

With this cleaned up, we also computed the lags on a finer scale--weeks--instead of months since we noticed that months were not granular enough. We computed the most common lags, and here is a plot of the results. This lag is the shift applied to received reviews, and the correlation is how well the two series correlated with each other after the shift. A correlation of 1 means that as one increased, the other increased as well, a correlation of -1 means that as one decreased, the other increased, and smaller values such as 0.8 mean that the correlation was positive, but less strong.

So the result here is both a little messier and structured than we had hoped from our hypothesis, but that’s part of the research process!

To elaborate, in the X dimension, the lag, there isn’t a particular range that was significantly denser than the rest. In fact, if we looked at the histogram, we see something like this:

So we lied a little, it looks like that last lag of +20 weeks looks really popular, but this is actually an artifact caused by the serial correlation process. If you recall this graph:

The red line is the chosen lag at the peak. In this case, the shifting actually peaked, but if we had truncated the graph at 5, it would have simply picked that highest shift.

Not convinced? Here’s the same analytics, but now we calculated up to a lag of 40.

Looks like the 20 bucket wasn’t particularly special after all.

So ignoring this last bucket (and the first bucket for a similar reason), we notice that our histogram matches this noisiness that we observed for the lags.

What does this mean? It suggests that there is no general pattern that can succinctly summarize the larger population, and that we are unable to conclude that there is a common average positive or negative lag relationship between the number of reviews someone has given and the number of reviews that they have received. Some authors sent more reviews after receiving more reviews (positive lags), some authors received more reviews after getting reviews (negative lags), and some authors did not exhibit much of a relationship either way (the first and last buckets which didn’t find a reasonable shift). Although these relationships do exist, the timing was not consistent overall so we can’t say anything about fanfiction.net authors in general.

So...

2. Looking across users, we do not see consistent behavior in a time-shifted relationship between a person’s received and given review count

Even when we look at the lags with the highest correlation (r > 0.7), we see that this even distribution of lags still holds.

In summary, this isn’t the dead end! (With research, it rarely is!) But it helps paint a better picture of the users in the community and why this approach may not be well suited to encapsulate it well. We see that the relationship between reviews received and given doesn’t follow a necessarily time-shifted relationship and that in fact, this shift can go either direction. Try taking a look at your own reviewing trends, and see where you would be located within these graphs! Are you someone who has a positive shift or a negative time shift… or no strong correlation at all?

Introduction

One of the questions we occasionally get from authors is: “What kinds of submissions get the most reviews?” We think this is a really interesting question and we’ve started doing some exploratory analyses related to the quantity of reviews that authors receive based on a variety of factors. One of the factors that we decided to check out was the number of words in a chapter. We were curious: Would shorter chapters get more reviews because they might take less time to read? Or longer chapters because there is more for reviewers to dig into? Or maybe there’s a sweet spot somewhere in between?

Methods

To look into this we took a random subset of 10,000 authors from FanFiction.net with chapter publications over a 20 year period from 1997 to 2017. We then created a scatterplot with each point being one of these 10,000 authors, the x-axis showing the median number of words across their published chapters, and the y-axis showing the median number of reviews received on those chapters. The points are segmented into six groups based on percentile of the total number of reviews received on all chapters they have ever published. We then put trendlines in for each of these segments, so we can more easily observe if there are any relationships between chapter length and reviews received across each of these groups. We also performed this analysis at the chapter with similar findings. The results are preliminary and warrant further exploration, but we’ll share what we’ve found so far. 

Results

It turns out that the small number of most highly reviewed authors in the top 1% saw an increase in reviews received up until chapters of almost 5,000 words in length, at which point their chapters began to receive fewer reviews on average.

For those authors whose works are in the top 25% of reviews received (excluding the top 1%), as chapter length increases, the number of reviews received on those chapters does as well. Interestingly, there does not appear to be the same drop off in reviews received for longer stories for these authors as there was for the authors in the top 1% of reviews received.

On the other hand, the remaining authors whose chapters are less highly reviewed saw little change in the length of chapter published with the number of reviews received.

Conclusion

These preliminary results point to some interesting potential implications on how an author might be able to get the most reviews. For the most highly reviewed authors, shooting for a chapter of around 5,000 words in length is most likely to result in the highest levels of engagement. However, for the vast majority of authors, writing longer chapters is not likely to have a negative impact on engagement from reviewers, and may even result in more reviews. 

How about you?

Authors: Netra Pathak and Kush Tekriwal

Hey there! We’re back, the researchers studying the fanfiction community at the University of Washington’s Human-Centered Data Science Lab. This time around, we’ve created a prototype feedback tool that we hope will be helpful to the fanfiction community. The tool will contain dashboards with concise summary reports and trends of an author’s reviews that may help the author reflect on their writing. We have a personal motivation to enhance the joy of writing, and are interested in hearing what authors think of our prototype tool.

Introducing the Concept

We’ve found the fanfiction community provides just the right kind of encouragement with its self-sustaining, distributed-mentoring setting. (Distributed mentoring differs from standard mentoring because it’s shared in small pieces by a large number of people.) This environment improves the writing of many authors but also boosts self-confidence. Hence, we thought that gathering review summaries and offering a reflection tool of all feedback received might be useful. This might help to further improve writing proficiency.

In this part of our study, the overarching research question we have is: “How can visualizations help fanfiction authors further enhance their learning from reviews?”

We’re interested in your feedback on this prototype visualization tool.

Thank you!!!

(1) FFN is an older site, which was helpful because it was technically more simple to scrape and also contains a rich data from decades of posting

(2) Good question! I believe @destinationtoast has a round-up posted from GeekGirlCon. I’ll work on getting our slides up.