terminate sequence

1. Clean up the fics. I didn’t do any stopword removal here because of step 2. Just removal of funny symbols, etc.

2. Pull out and clean the verbs. I used Spacy’s part of speech tagger for this. I also lemmatised all verbs pulled (so you may see some odd-looking words if you do explore the visualisation). Of course, sometimes the tagger misidentifies words as verbs, so you may see what should be nouns, etc. I’ve tried to remove character names at least by relying on the long list of names I created from running topic modeling on this corpus some time back.

3. TF-IDF weight the words. Basically, count how many times a verb appears in a fic, and then multiply it by the inverse count of how many fics the verb appears in. Words which are more ‘important’ or ‘representative’ of the fic in question should be weighted higher.

4. Perform non-negative matrix factorisation for dimensionality reduction. At this point, I could have possibly gone straight to tsne for visualisation with the tf-idf weights, but the results were even worse (if I recall). So I performed NMF. Like other topic modeling methods like LDA, the number of dimensions/topics to go for is user-prescribed.

I ran NMF from 10 to 60. At each point, I calculated the mean cosine similarity between all topics (we’d want something that’s lower, topics that are too similar to each other aren’t great). I picked 40 topics in the end, since that’s where the mean cosine similarity seems to level off. This is definitely a subjective call, but since I’m just really doing NMF for faster tsne visualisation, I was ok just eyeballing the graph and rolling with this.

(The top 15 keywords for each topic can also be viewed on my Github page.)

5. Run tsne. I used sklearn’s implementation. I also normalised my NMF weights before submitting it for the run. I’m terribly new to tsne, so I’m pretty sure the parameters I selected weren’t great.

I went with a perplexity of 350, PCA initialisation, a learning rate of 100, a max number of 30,000 iterations with stopping if there is no improvement after 500 iterations. tl;dr, the output of tsne is heavily dependent on the parameters you set (there’s a great Distill article out there on this), and with my lack of experience I may have bungled this.

6. Visualise tsne. I used plotly’s scatterplot for this. Each dot is one fic. If you hover over the dot, you can see the top-5 tf-idf weighted verbs for it (the most ‘important’ verbs to that fic, according to the tf-idf metric).

Overall, the visualisation is more or less just a hairball, but to my amusement/chagrin, there appears to be what is a little nest of smutty-seeming fics lurking at the bottom:

1. Preprocessing relationship tags Ship tags can get pretty unstandardised! I took only tags that had a ‘/’ between the characters (e.g. Connor/RK900). I then split each tag at the ‘/’ character and standardised the resulting two (or more) names. For example, Connor (one-sided), RK 800, Connie, Connor - they all fall under ‘Connor’.

As ships may involve more than two characters, for the sake of visualisation, I converted those into a list of every possible permutation of pairs within the listed ship; so Connor/RK900/Hank into [Connor/RK900, Connor/Hank, RK900/Hank].

2. Extracting user interest in ship(s) Again, I took user interest in a ship to be signalled by: (1) being an author of a fic with the ship tagged, or (2) leaving a kudos on a fic with the ship tagged (1) was extracted very simply given that each fic has an author(s) tied to it. (2) was taken from the kudos list shown at the end of every fic. Given that there’s no way to disambiguate guests and tie them across different fics on my end, I discarded that information and focused only on registered site users.

At the end of this process, I had a long list of user-ship interest pairs. 354382 entries, to be exact.

3. Preparing the plot Given the information I have (user-ship interest pairs), I started with a bipartite plot. All that means is that there’s one set of nodes that’s all users, and another set of nodes that’s all the D:BH ships ever written about on AO3. Between these two sets of nodes are links that connect a user to a particular ship if they’ve expressed interest in it. The set of user nodes has no links within it, same for the set of ship nodes.

The bipartite graph can then be flattened/projected into a regular graph with just one type of nodes (in this case, ships, since I’m interested to see how many interested users each pair of ships shares). So we keep only the ship nodes, and a link exists between two ship nodes now if they share at least one common interested user. We can weight the links by the number of common interested users they share.

This graph has 311 ship nodes and 34429 links. It’s not huge, but it’s very unwieldy to visualise. Likely, many nodes also share very weak links (e.g. just a couple of common users). Since my end-goal is really just visualisation, I decided to prune the graph.

4. Filtering the graph I reuse the same filter from Serrano et al., (2009) that I applied on my character co-occurence graph. I set it at a relatively strict level of α=.001. This filtered the graph down to 185 ship nodes and 1251 links.

5. Visualising the graph I made this one less springy than the previous ones, since I realised how annoying it was to explore when the nodes keep bouncing back to place when you tug them out.

Ship node size is determined by number of users that have indicated interest in the ship (bigger=more). Link size is determined by the number of common interested users the pair of ships shares (thicker=more).

First off, credit for a lot of code goes to Gensim’s tutorial.

A traditional way to assess similarity between two writings is to look at their common words (bag-of-words). We run into a problem if the writings don’t use the same words even though we know they’re kind of alike/have some degree of similarity though - e.g. ‘Connor really likes dogs.’ and ‘Connor loves puppies’.

Word Mover’s Distance (WMD) improves on this by tapping on word2vec embeddings (tl;dr, words that are semantically similar/used in similar contexts, like dogs and puppies, should have more similar embeddings than those which are not, e.g. dogs and fruit). WMD calculates the distance needed to get from the embeddings of the words in one document to another. In other words, the shorter the distance needed to travel, the more similar the two documents should be. (Obviously the implication here is that the word2vec model doesn’t suck completely as well, and hm, well, not completely sure about the strength of my D:BH one.)

With that out of the way, on to the methodology:

1. Load a pre-trained word2vec model. As I mentioned, I’ve already trained one on my collection of D:BH fics.

2. Clean all the summaries. For this exploration, I dropped all not rated fics (leaving me with 12,647 fics). I removed all names and stopwords (really common words like ‘the’, ‘a’) from the summaries. Stopwords were removed because they don’t contribute much to understanding how similar two summaries are in this case.

I dropped any summary that had a length of 0 after cleaning. The mean length of a cleaned summary was 21.3, median was 16, and standard deviation was 17.5. Boxplots for cleaned summary length broken down by ratings, for the curious:

3. Run WMD (partially). I used Gensim’s implementation of WMD. Looping over each summary, I calculated the WMD between it and every other summary, recording only those that shared a .55 or above similarity with it (note: arbitrary decision on my part for that figure). Each query takes quite a bit though, anywhere from 25 seconds to over 200 seconds - so I stopped at about ~1000 summaries (out of the total 12,647), since I just wanted to get a gauge of how this was working on my fic dataset.

4. Plot out network for marginally easier exploration. It got a bit tiring trying to scroll through the printed results, so I decided to put whatever I have into an interactive network so I can more quickly get a sense of the results. Again, I used pyvis.

Nothing fancy here, nodes are summaries (I cut them off at 150 characters, otherwise some would run on for too long) and they’re linked if they have at least .55 similarity with each other. You can hover over nodes to glimpse at the summaries, but I didn’t add any other story details/numbers, because the network would take too long to load otherwise.

I did label the nodes by rating, however, so - red nodes are Explicit, orange nodes are Mature, pink nodes are Teen & Up, and white nodes are for General Audiences.

This is more a descriptive summary of a dataset than an actual analysis, so this is really just a collation of very basic steps.

1. Standardising character tags AO3 has a ‘character tag’ category, so I tapped on that. However, the tags aren’t very standardised (there are at least more than 100 ways writers have tagged Connor alone), so I manually sorted all these 60k+ tags. There were also tags that were more of the writer’s commentary than anything (e.g. ‘possibly more in the future’) - I put those under an ‘Other’ category and did not include them in the network.

At the end of this process, I had a list of 118 unique characters. I then relied on this manually sorted list to automatically relabel the character tags of every fic.

2. Retrieving basic character and character-character information Okay, so now we have standardised character tags on every fic. With that we can already the most basic question about each character, i.e. which fic rating do they usually pop up in?

We can also go a bit further and start looking at character-character pairs. This is easy to do - just create every possible pair permutation from a fic’s character tags. For example, a fic tags RK900, Hank, Connor - we can create a set of pairs, [(Hank, RK900), (Connor, Hank), (Connor, RK900)]. So now we can also start thinking about questions relating to the relationships between characters e.g. what is the most common rating Connor and RK900 typically appear together in?

Specifically, at this stage, I derived the following info to dump into the network: 1) The most common rating the character typically appears in 2) How many unique characters the character has appeared with 3) The top 5 characters that appear with the character 4) How many fics a character pair has appeared in together 5) The most common rating a character pair typically appears in together

3. Adding ship information This network focuses on character occurrence, but I also wanted to add some info on ships for easier comparison (just because two characters co-occur a lot together =/= there are a lot of shipfics of them; think Connor and Sumo).

For this I relied on the ‘relationships’ category for tags and a lot of the work that I did earlier for the ship network. As before, I looked at only tags with the ‘/’ between characters (i.e., don’t count tags written in the style of Markus & Simon as ships). Again, the tags are unstandardised, but a lot of the cleaning work has been done for the earlier network so I tapped on that.

With standardised ship information now available alongside the standardised character appearance information, we can start doing other simple comparisons. For example, of all the fics the character has appeared in, what percentage actually has them in a ship? Note that this is just a rough number; three characters (unspecified humans, Reader, AP700) have percentages that exceed 100% (i.e., they appear in more ship tags than character tags).

We can also create permutations as we did for character pairs for polyships (e.g. Connor/Hank/RK900 into [(Hank, RK900), (Connor, Hank), (Connor, RK900)]) that can help answer questions like: of all the fics the character pair has appeared in, what percentage actually ship them? Again, this is just a rough number; 7 pairs (RK900/unspecified humans, Ralph/Reader, Traci/Traci, Connor/SWAT, AP700/Connor, Connor/Reader, Reader/unspecified humans) have more ship tags than tagged appearances.

At this stage, I added the following info: 1) How often the character is shipped whenever they appear 2) The most common rating of the character’s shipfics 3) How often a pair of characters are shipped together whenever they appear 4) The most common rating of a pair’s shipfics

4. Creating the network As before, I relied on the library pyvis to create this interactive visualisation. The characters are nodes and they’re linked if they’ve appeared on a fic together at least once.

Node qualities: x Node size: The larger a character node, the more unique characters the character has appeared with. x Node shape: Star-shaped nodes indicate characters that are shipped in 50% or more fics that they appear in. Diamond-shaped nodes indicate characters that are shipped in 25% to 50% of the fics they appear in. Regular circle-shaped nodes are characters that are shipped in below 25% of the fics they appear in. x Node colour: Node colour is determined by the fic rating the character typically appears in. Colour reference: grey = Not Rated, green = Gen, blue = Teen, orange = Mature, red = Explicit.

Link qualities: x Link thickness: The thicker the link, the more often that pair of characters has appeared together. x Link colour: Link colour is determined by the fic rating the character pair typically appears in. The colour scheme follows that of the character nodes.

Hovering over nodes and links also provides more information (for specific numbers, ship information for comparison, etc). For the specific percentages (e.g. 28.97% of fics shipping Connor are Explicit; 83.33% of fics that ship Kamski and ST200 are Explicit) - these are given for the majority category; in other words the top rating for Connor shipfics is Explicit, the top rating for Kamski/ST200 fics is Explicit.

But wait! If you look at the final network in the gif/Github, it’s obvious that there aren’t the 2000+ links I mentioned and it doesn’t even look like there are over 100 nodes. What I did at this final stage was run the network through a disparity filter.

The original network looked like this:

Practically unreadable. The issue is that many links between many characters are likely to be spurious (i.e., really only a small handful of fics had them appearing together). The disparity filter I used is from Serrano et al., (2009) and basically tests the links of every node for significance (your usual p<=.05 stuff).

This keeps only the links between characters that likely did not occur to mere chance alone, resulting in the final network you see in the gif/Github. So while the network stats were built off all the details collected from 13k+ fic, for clarity I visualise only a condensed backboned version of it.

Before I go into the methods, two questions to think about:

1) There are a lot of ways to talk about the prominence of a character - how many times did they appear (and what counts as an appearance; just walking across the street, or actually being the focus of action)? How much screen-time did they get? How many lines did they say? How many times their name appeared or they were referred to? Maybe even how much influence they had over events?

2) If we’ve settled on an answer for (1), how do we compare that between different mediums? D:BH is a game and fics are, well, mainly text.

These were questions I thought a lot about before proceeding. In the end, I settled for rather simplistic operationalisations. For the actual game, I relied on the dialogue transcriptions from the Detroit: Become Text project. For fics, I relied on the character tags provided by the AO3 authors.

1. D:BH ACTUAL GAME DIALOGUE Dataset: Some people have done wonderful work transcribing most of the dialogue in the actual D:BH game (please see Detroit: Become Text on Github!). I worked with all the material they had - 4728 lines of dialogue.

1.1 Preprocessing Everything was in HTML so it was a simple task of writing some code to pull out the character names and counting the number of dialogue lines each individual had. x Assumption 1: Share of dialogue is indicative of character prominence in the game. I decided on this mainly because I was limited by the information I had (D:BT almost fully transcribes dialogue only) and also because D:BH is a very conversation-heavy game. x Assumption 2: Sometimes characters might be interchangeable for certain lines, e.g. Josh or Simon can tell Markus, “Be careful, Markus. Our people need you.” I chose to drop these lines and not assign them to any character’s count since there were only 15 of them in the whole transcription. x Assumption 3: The game has multiple scenarios based on your choices. I chose not to distinguish between them, taking the whole transcription as a long text document. My justification is that if the writers decided a character was important enough to have/be in several variants of the scenario (think Connor’s multiple endings versus, maybe, the android Markus picked up paint from), I should not discount that.

Following this, I selected 40 named characters (i.e., no random policeman/passer-bys) for visualisation and comparison. These 40 characters together had 3745 transcribed lines.

1.2 Results

Note the percentage is calculated from the lines of these 40 characters only (i.e., does not cover dialogue counts of unnamed policemen, unnamed deviants, etc).

Connor, Kara, and Markus take up almost 50% together. I think Markus has slightly less lines than Connor and Kara since it’s possible for North to take his place. Fan favourites like Gavin hold only a tiny sliver of the pie. Note that according to this operationalisation of prominence, RK900 is at 0% (since he doesn’t say anything in-game).

2. CHARACTER TAGS FROM D:BH AO3 FICS Dataset: 13,933 English-language fics scraped from AO3, spanning May 2018 to mid Oct- 2019.

2.1 Preprocessing I collated all the tags that fell under the AO3 character class (61333 tags). I did a fair bit of manual cleaning as usual (for misspellings, different ways to tag a character, etc). x (Long) assumption: Picking out dialogue lines from fic and attributing it correctly to characters is an extremely difficult task (I think I've seen machine learning papers on it). So, I chose to go with tags - I assumed that characters that are important or at the very least more than a one-liner would receive a tag. At the same time, I assumed that characters that are really just one-liner appearances (like “He saw Nines walk across the room.” with zero mention of Nines in the rest of the fic) would not get tagged.

Clearly this is not foolproof since everyone has different ideas of what constitutes a character that should be tagged, but it’s what I worked with (otherwise I would have to read and check every fic). Stuff to note:

1) Ambiguous tags like “everyone” were not assigned to any character. 2) Tags that mentioned multiple characters (e.g “mentioned North and Josh”) or ships were also not assigned to any character. 3) Tags that were in the style of “Connor (mentioned)” were still added to their respective characters. Again, I think most writers have varying ideas of what a “mention” actually means, so I decided to be less stringent and more inclusive here. 4) Tags for ‘Jericho’  and ‘Jericrew’, etc - I added 0.25 to the counts of Markus, North, Simon, and Josh each time such a tag appeared. Whether I did that or not probably wouldn’t have drastically changed the results though, since there were only 87~ Jericho-related character tags.

I picked out the same 40 named characters I looked at from the actual game dialogue. These 40 characters together had received 53274 tags.

2.2 Results

Again, note the percentage is calculated from the total number of times these 40 characters were tagged (e.g., does not include tag counts for OCs - which were tagged quite a bit as well).

We see Connor still holding on to his share and huge jumps from Gavin and RK900. DPD members in general also seem to have slightly bigger slices of the pie in fandom.

3. Ending notes It’s pretty difficult to see some of the information from the two pie charts, so I put the information from both into the bar chart that I introduced this whole post with: