Inq's blog

Keyness (AntConc, TF-IDF)

Term freq. / inverse document freq.

Uniqueness, TF-IDF

How labor talks of "government" relative to conservative

Research report: finish TF-IDF keyness

Reflection: how to move forward w/ this type of data, question, what has been tried up to this point

Go over everything you did, give estimate of interesting venues or pipelines for use on these RQs

LLMs, etc.

GitHub

Today I got to chat with some people from the CDH (I thought I'd take advantage of their walk-in hours since I was already working at the Drift anyways). I asked them about how to tackle other terms that I don't have set categorizations for in a data-driven way. I got some advice and I just want to keep my notes here in the blog, so I don't lose them! Sander (if I remember his name correctly) was very kind and took me to meet his colleague in the CDH office, who was of great help in answering my questions.

From the CDH folks:

Run or train a topic model

BERT, LDA -- which one?

Word embeddings, vector representation of words

Bigram, two embeddings, multiply them with each other

Could try (if you have embeddings for 50 bigrams) to create a 2d plot, PCA, see clusters (side note: I did try this at the start of the internship, and it gave me "bs" unique groups like names. PCA tends to highlight unique cases anyways. Or maybe I'm not doing something right.)

Similar bigrams, clustering

Comparing BERT and LDA: find a similar study and use it?

SpaCy for sentiment? More machine learning and more built in models (vs. NRC?)

Try it with the LLM, especially for tone

BERT or LLM to classify sentences that refer to the past or future / time elements, then work on training it

Make a standard set, see if model is classifying correctly

Close reading on a set to get some indication (e.g. people in this political spectrum tend to use xxx more) -- though, this would be a lot of work. Not sure if it's feasible for the internship.

Notes from prior meeting sessions:

clustering to find labels of crisis and other terms?

HDBSCAN?

Agreement score (but by topic)

TF-IDF within periods

Collocations

Word2Vec

Find turning points? What do we see more or less of? E.g. 2010s, pre or post 2000s, etc. What shows?

Find most common associations (per year or decade) with clustering techniques

Window of data around "crisis"

Time / implicit ways of right wing language

Cluster them w/ topic modeling + WEs

Co-occurences

I'm not sure what to say here, since "undemocratic" (in terms of counts) showed up as a very small percentage of words, compared to the other words like "democratic" or "crisis" or "fascist." Even so, I'd just like to point out the top "undemocratic" words.

Similar to how "accountability" showed up with "democratic" in our other bigram works, especially in the '90s onwards, you can also see that "unaccountable" showed up with "undemocratic" in the same eras. I'm not sure if the lower number of counts mean that "undemocratic" is negligible or insignificant, but I think it's an important remark to make nonetheless. "Unaccountable" also sounds like a very uncommon word in general to me. Interestingly, a shift from a democratic who to a democratic how was also paralleled here in "undemocratic." From how an undemocratic body or government shifted to fundamentally or profoundly undemocratic, or to some kind of unaccountable and undemocratic. Interestingly, while "undemocratic government" showed up in the '80s, it didn't make the top bigrams in the '90s or '10s. And I'm not so sure what this means, or if it is significant at all. I don't want to be over-analyzing some statistically-churned out data sometimes.

The presence of both "democratic" and "undemocratic" (albeit "undemocratic" having a lower count) does suggest something to me: that there was indeed a debate going on! And that matters!

Of course, we also have to remember that we don't always have to say "undemocratic" to say undemocratic, right? It is also equally possible that the positive adjective form, "democratic," is compounded with other negating words (e.g. this is far from democratic). So, I think within "democratic" itself, if we examine the surrounding contexts -- like how I've made a .csv with KWIC -- we surely have been double-counting "undemocratic" into the "democratic" category too...

This past week, I've spent a lot of time asking Mistral and Llama to label the "democratic" terms (and damn, this is just "democratic" alone) to quite little avail. Not because it wasn't working, but because it took too long to run! I tried my best to let it run its course over the weekend, and I eventually had to use my computer on Sunday night for my Visa documents. I am still terribly anxious about my UK Tourist visa... I just want to spend two weekends with my close friends and catch some concerts and events with them!!! Please, Mr. Starmer, I genuinely do not want to secretly find work in your country!! No, I am not broke. No, I do not want to immigrate to the UK. Yes, I can afford both trips.

Anyhow!

I worked only with "democratic" thus far, as I am still finding ways to categorize other terms. After using KWIC to extract left and right contexts (from the lemmatized corpus) surrounding each bigram occurrence -- which, In My Humble Opinion might be the most accurate way to use LLMs to label -- I asked both Mistral and Llama to categorize "democratic" in a similar manner, given the context it has been given. I am doing this in the hopes that this would help clear up the confusion or miscounts that emerged from words that could fit into more than one category of "democratic." I fed the LLMs this prompt:

So, they will have to pick 3 words from the corpus that they best believe justify the categorization choices. Obviously, I ran 5 test cases first. I did this with Llama (llama3) 🦙. Here are the results:

Bigram: democratic right Context: par rather one boys one mr speaker rest new member old part goodly company heirs great tradition exercise behalf represent [democratic right] freedom take perhaps grant long enjoyment country envy many people many land like air breathe little noticed presence value beyond Result: ('5', 'democratic right freedom') Bigram: democratic society Context: stansted roskill commission also rule stansted yet people find stansted still one lead contender third london airport find mysterious meant [democratic society] hope government approach matter open mind base judgment cost factor grateful call early debate welcome general tone content gracious speech Result: ('6', 'democratic, society, people') Bigram: democratic right Context: edens ideal propertyowning democracy repeat today floor devil mean mean u property right participate democracy somehow less human few citizen [democratic right] property kind think underlies strategy municipal asset strip involve wholesale selling council house firmly believe party stand firmly believe people Result: ('5', 'right, property, participate') Bigram: democratic party Context: paul johnson lead socialist journalist former editor new statesman use word believe considerable importance conservative party great repository freedom country [democratic party] party individual freedom party rule law last line defence possess Result: ('1', 'Party, Conservative, Freedom') Bigram: democratic socialist Context: many case fear totally disappeared way country grow prosper participation people involvement society see grow prosper nation people unless establish [democratic socialist] government country whatever argument shortterm longterm aim main philosophical divide remain u recognise person value society value people individual value Result: ('1', 'Party affiliation, Socialist')

However, I faced issues running through the entire "democratic" bigram occurrences. With over 33,000 entries alone for "democratic," this is definitely not my job to undertake. And as you may already know, even Mistral and Llama took a very long time to handle this. Based on my calculations, I'd have to leave my computer running for a whole week just to get through "democratic" alone. So, I think I'll focus on other things, such as modeling within periods, or modeling to find other ways to categorize terms like "crisis" in a data-driven way. I think clustering could work well here.

Say no more: from 5 to 3 "reasoning" words...

I had to kill this program twice. Initially, I asked it to pick 5 words max. from the corpus to justify its categorization. It took an insanely long time to run, so I decided to put it down to 3 words. (Even so, it still took so long that I had to kill it just to upload visa documents.) But I noticed something quite interesting with the sample runs. For one thing, you already can't expect LLMs to be entirely consistent each time they categorize it -- but let's consider that negligible for now (the same way we'd consider air resistance negligible because we're too lazy to deal with fluid dynamics). Here are the things I got from running the sample lot.

With 5 words max. reasoning:

With 3 words max. reasoning:

I suppose what I'd expect is that when I ran it again with just 3 words, those 3 words would at least be found in the 5 words -- ya know what I mean? And that's not the case for a fair share of the entries. I don't know if this should be a surprise, given that LLMs aren't always consistent, but maybe this exposes something inconsistent in the use of LLMs to reason. Or, is this just Llama, given that I ran my samples with only Llama? Despite different reasoning words, the number assigned stayed mostly consistent.

I still don't know what to make of it, but I'd like to leave this as an open question. If you have thoughts on this, do let me know.

In a separate notebook, I started working on bigrams, in line with the EE pipeline! From last week's work, I was already able to produce the top 50 bigrams per each decade, for each keyword I was looking for (you know, the democracy, fascism, crisis nouns and adjectives). I was able to obtain bar charts such as these (both in count and in relative counts) for all 6 words in question, in all decades. I saved the .csv and .png files, and I'll only show a few here.

Now, what I found interesting in both the case of bigrams and the earlier work I did on relative frequencies is the distinction between nouns and adjectives. Let's take the case of democratic (adj.) versus democracy (n.).

Democratic

In the '80s, "democratic" appeared to refer mostly to either party names, systemic matters, and principles, characterized by "social democratic" (which could be either party or practice), "democratic party," "democratic process," and words alike. By the '90s, while "process" still remains, terms such as "accountability" and "deficit" started to take up the top spots. "Social democracy" also ranked lower in the '90s+ decades.

So, what does this signal? At best, a qualitative analysis is just ✨correlation not causation✨, but this could imply a few things. To me, the top labels of the '80s suggests that terms relating to democratic systems -- one could say -- were more popular. Or terms that give some sense of identity around democracy, be it a party or what kind of democracy (e.g. "social democratic", "democratic party"). However, past the '90s, "democratic" appears more closely to words relating to democratic processes or how democracy functions. Perhaps they were debating whether democratic institutions were working properly? Or perhaps issues of European integration? The end of the Cold War? Who knows. But there is definitely some '90s-ish shift from democratic actors to democratic qualities.

Comparison to older decades?

As a part of last week's work, I also compared bigrams relative to their prior decades: overlaps, recurring ones, and more. I also used spearmanr (Spearman's Rho) from stats.scipy -- again, in line with what they did in EE -- to statistically interpret the recurring bigrams. Using this package naturally churned out a p-value as well. It's good to know that a lot of these calculations were unlikely to have happened "by luck."

Labeling themes

Now, this is where the tricky part is. In EE, a group of researchers met and agreed upon how to label their bigrams (see my prior post for the 6 labels they used). With me alone on this project, I decided to give it a twist by asking locally-run LLMs (Mistral and Llama) to label the bigrams. Of course, I participated in labeling too. This is now an interesting case of methodology. Initially, I tried to use Claude's or Gemini's APIs, but they weren't free given I would be using them somewhat heavily. As for my local LLMs, I gave a prompt which included the 6 labels, and asked it to give a max. 10 words justification. I asked it to run through a .csv that I compiled that included all unique bigrams. Here are some printed results for "democratic," as "democratic" is one that we already have a framework of themes to work with from EE.

(Later on, I would modify this prompt to tackle labeling with context -- I'll save that for another post).

[1/82] 'democratic process' | Mistral: 5 (Refers to a specific democratic procedure.) | Llama: 5 (Procedures and practices linked to specific processes.) [2/82] 'social democratic' | Mistral: 4 (Refers to a political ideology combining elements of socialism and democracy.) | Llama: 4 (Refers to broad democratic principles.) [3/82] 'democratic party' | Mistral: 1 (Refers to UK parliamentary party name.) | Llama: 1 (Party affiliation, UK parliamentary party name.) [4/82] 'democratic unionist' | Mistral: 1 (Refers to UK parliamentary party name (Democratic Unionist Party)) | Llama: 1 (Party affiliation, UK parliamentary party name.) [5/82] 'democratic accountability' | Mistral: 4 (Refers to democratic principles.) | Llama: 4 (Refers to democratic principles for broad ideas.) [6/82] 'democratic right' | Mistral: 5 (Refers to a specific democratic practice or procedure.) | Llama: 5 (Procedures and practices linked to specific processes.) [7/82] 'democratic society' | Mistral: 6 (Refers to societal issues.) | Llama: 6 (Societal issues discussing broad societal topics.) [8/82] 'democratic system' | Mistral: 3 (Refers to democratic structures or institutions.) | Llama: 3 (Referring to state institutions and political questions.) [9/82] 'democratic government' | Mistral: 3 (Refers to state institutions and political questions.) | Llama: 3 (Referring to state institutions and political questions.) [10/82] 'democratic control' | Mistral: 5 (Refers to a specific democratic procedure or practice.) | Llama: 3 (Referring to state institutions and political questions.) [11/82] 'democratic institution' | Mistral: 3 (Refers to state institutions and political questions.) | Llama: 3 (Referring to state institutions and political questions.) [12/82] 'democratic republic' | Mistral: 2 (Refers to countries (German Democratic Republic)) | Llama: 2 (Reference to international divisions, country name.)

With the exported .csv file, I modified it to also include my own rankings. I then visualized all three models' (counting me as one 🙋‍♀️) results.

I also looked at Cohen's Kappa (already in scikit, luckily) to measure the agreement between models. Here are the values that I got:

Inq vs. Mistral: 0.366

Inq vs. Llama: 0.296

Mistral vs. Llama 0.602

Source of image: https://www.statology.org/cohens-kappa-statistic/

Evidently, the LLMs are the most consistent between each other. It definitely makes me question my own labelling, as I often categorized what I felt unsure about into democratic procedures, as that is a vague-enough category. As aforementioned, there were many instances where a word could be many categories at once. I guess the next step is to now investigate how the LLMs justify it.

Here are some other frequency plots to help visualize the decades.

To agree, or not to agree?

I then produced a confusion matrix (in a way, I don't know if this is an appropriate name for it since I'm doing agreement counts in matrix form than an actual vs. prediction thing, but at least it's a confusion matrix in spirit and presentation).

You can interpret this matrix in terms of counts. For example, let's look at the matrix just below this text (Inq vs. Mistral). The top left says that there are 3 instances where I have labeled "party affiliation," that Mistral has also labeled party affiliation. The diagonals are basically the number of agreements I shared with Mistral. However, if we look at the 4th cell on the top row, there are two unique bigrams that I have labeled party affiliation, that Mistral considered democratic principles. The darker colors suggest a higher count. Interestingly, there were 14 that I categorized as procedures, that Mistral read as principles.

Following this, I used KWIC to extract a .csv file with 20 words to the left and right of each bigram occurrence to allow the LLMs to read bigrams with more context. I was not involved in manually labelling the 33,000 something entries myself, obviously, but this is a great case of testing free local LLMs! I'll write this in another post!

Some other notes:

Yes, I did look at "undemocratic" as well, RE: the last post and the relative frequencies. It is interesting to see how it fares similarly to "democratic" in terms of the trends (the increases and decreases, and overall shape). However, the overall counts are so small compared to "democratic" it's almost negligible. It's something I'd keep as an appendix.

Periodization: would also be interesting to look at my multivariable plots and find turning points and such. I also should periodize sentiments.

The concept of time / reverence is also interesting, but I'll have to find a data driven way to approach it.

Perhaps this is a good point to catch up on what's gone on thus far. Firstly, I've expanded my criteria from "democracy" or "democratic" to a broader list of

words_to_find = ["democratic", "democracy", "fascist", "fascism", "crisis", "critical"]

as they are more in-line with both the literature and scope of research. I got to wrap up the first parts of the "Extension or Expansion" (let's call it EE) pipeline on relative frequencies. I was able to upload it onto a github repository as well! Here, I generated my own graphs, but also had separate blocks of code by which I asked AI to modify my existing code to make my plot look more presentable for the final research paper. I made my use of AI clear in my Jupyter notebook. I hope you can understand that while I do know how to modify my plotting code with packages like seaborn, or change the plot backgrounds / fonts / colors to make my plots look nice, it is simply not worth my time to tinker with these language-analysis plots until they look humanities-research-pretty! I obviously had to modify plots before, especially during the astro lab that I took at Pomona, but STEM standards prefer functional over pretty plots! With this project, I don't want to spend time fixating on small details and background colors, especially if it's something I already know how to do / I know it will consume time, and AI can write code that modifies those elements for me.

Here is the link to the GitHub repository.

Anyhow!

I saved my counted frequencies (from lemmatized / processed .csv files) and saved those stats as new .csv files. I then proceeded to produce basic plots.

Here, I think what is interesting is that among our three selected issues (for this project) -- democracy, fascism, crisis -- I try my best to look at the noun vs. their adjective counterparts. This is something I've tried to do differently from EE, that only looks at "democratic," the adjective.

"Crisis," I must admit, is a bit of a weirder case, as the only adjective counterpart I can think of is "critical." Though, these two words can be used in so many different contexts, and in such different ways compared to pairs like democracy (n.) and democratic (adj.), or fascism (n.) and fascist (adj.). Below is an example of my plot that is not yet pretty-fied.

Here, I thought it would be nice to see how these distant-reading trends fare with the dominance of different parties (if there is any correlation). So, I got some data from this website and overlaid them on my plots. Now, I can give you some prettier graphs!

Since the Election Data Vault website contains data for both seats won and % of votes, I thought it would be worth looking at both. More so, the % of votes, as you can see the increased or decreased popularity represented by votes, which doesn't necessarily result in a seat.

Seats won

As aforementioned, "crisis" and "critical" can refer to a whole plethora of topics and evidently have the least correlation with the seats that are represented in parliament. As we get closer to 2020 we can see that "crisis" peaked -- most likely due to Covid. In fact, when I did bigrams later and isolated 2020-2021, Covid indeed dominated the "crisis" discourse. Later on, for my bigrams-by-decade, I decided to exclude 2020-2021 for that reason.

As for the counts of the word, these are relative frequencies, thus are normalized.

% of votes

I believe here's where we see more correlation -- maybe with the exception of crisis or critical, of which we should look at separately. With the uptick of parties such as UKIP, we see an overall decrease in democratic, the adjective, suggesting fewer descriptions or discourse of things that are democratic in nature. Interestingly, there is also an overall decrease in "fascist" talk. Adjectives saw an overall decrease, but look at the nouns! They seem to have an uptick trend, both for "democracy" and "fascism," which feel really paradoxical to me!

And I'm not so sure what this means on a deeper level! What is this about? How do I investigate this?

https://converis.jyu.fi/converis/portal/detail/Publication/273628261?lang=en_GB

Research questions:

"Which words were increasingly or decreasingly used in a similar way as "democratic" in this period, and what do these challenges tell us about long-term changes in political discourse?" (p. 73)

"Which new themes and issues did MPs define with "democratic"?" (p. 73)

"Finally, in an exemplary manner, how did competing uses of "democratic" appear in concrete political struggles?" (p. 73)

Bigrams / word pairs were manually classified by 6 criteria:

Reference to party names

Reference to countries (like democratic in DDR)

Constitutional issues ("democratic structures")

Principles / ideas ("democratic legitimacy")

Practices / procedures / practicalities

Societal issues

This will likely be a long log. I have had 2-3 meetings with Pim between now and the last log on this blog, and I will have another meeting with him tomorrow. It's a bit of a tight schedule, but we have to shift the meeting a little earlier due to the end-of-week holidays.

During the past couple of weeks, I have been exploring other methods of sentiment analysis -- namely, drawing some ideas from multivariable calculus and differential geometry -- and, last week, trying to gear myself back to the literature and research questions a little more.

It's really easy to get caught up in the methodology (guilty as charged!!) especially when I get to experiment with so many things, like trying to incorporate a little multivariable or differential geometry into it! That said, I do see myself being more of a research engineer than a researcher. I don't think I can commit to a PhD, or lead my own project for years in a way that a PhD would require, but I still love being involved in research! I also fancy the idea of being a part of a research project, and being able to be more methodology focused / experiment with it to yield results for different projects. This definitely makes me feel relieved, since I always thought research engineer positions would mean being complicit in evil finance or military companies!

Perhaps I should check out opportunities in the Baltics, since I have always wanted to move there.

I also received suggestions / took some meeting notes, and I will make those remarks clear in this log.

Without further ado...

Based upon previously discussed research angles, I tried my best to investigate these three themes:

Crisis, Democracy, Fascism

Namely, looking at how often they were mentioned, and who (or which parties) mentioned these terms. Of course, there's always the question of context / if things were spoken of in alternative ways / whether everything gets counted, but I have discussed that with my supervisor. It does, after all, depend on whether we care for just the times things were mentioned, or for the context around it too. I also tried to look at how time was referred to or spoken of. However, I am still working with a definition-driven approach, than a data-driven approach, something I hope to work on based on the literature Pim sent me in our recent meeting.

past_terms = {"past", "history", "historical", "tradition", "heritage", "again", "back", "restore", "return", "former", "old"}

I also tried to define proxies of demagogy based on Juan Linz's four signs of demagogy: rejecting democratic rules, denying the legitimacy of political opponents, tolerating or encouraging violence, and being ready to curtail the civil liberties of opponents. (Quoting this from the research angles document).

demagogy_proxies = {"rejecting_democratic_rules": {"rigged", "fraud", "illegitimate", "overturn", "betray", "traitor", "treason", "stolen", "corrupt", "undemocratic"}, "denying_opponent_legitimacy": {"enemy", "traitor", "treason", "betray", "betrayal", "illegitimate", "extremist", "saboteur", "subversive"},

and so on...

(Also sorry, the formatting is really weird especially when I try to use the "chat" font)

But let's take things a few steps back. I started off by doing more "basic" work: normalized frequency counts of words relating to crisis, democracy, and fascism themselves. As these percentages are quite small, I scaled it to counts per million (though proportionally, the percentages still hold).

Plotting them by party:

I can imagine that around 2020, for instance, "crisis" would refer to Covid. As for some pure speculation, the Gulf Wars around the 1990s? Involvement in the Middle East in the early 2000s? Strangely, the financial crisis of 2008 isn't reflected as prominently here. Plotting all the parties, I got the plot below. While I don't doubt the accuracy of Python's counts, I do have some reservations about what

While I don't doubt the accuracy of Python's counts, I have some reservations about what this data is showing. I guess we can all agree that UKIP is a big deal in this whole "undermining democracy" topic. However, with UKIP founded in 1993, in the latter half of the dataset, they would still have "less" overall data compared to, say, the Labor or Conservative parties. While this doesn't affect the normalized counts, it does affect the overall representation. As you can see in the following plot below, I tried to map "crisis" counts over time for the top 5 parties that have the highest number of those counts.

UKIP did not make the "most mentioned" cut, though we know UKIP is a big deal, and we know that they are extremely outspoken on issues such as Brexit. It is likely because they are a newer party, and that like the SNP, their more extremist views mean that they are more outspoken on fewer or more select matters.

Suggestion: show an isolated UKIP plot or manually add it. Britta also mentioned the fact that words are often not used in its direct form, which is very true!

I did the same for democracy.

I also displayed some df headers for words relating to democracy or democratic, so I can read around it and see the context a little more. However, this is difficult, as there are so many entries and the words are now lemmatized / cleaned up, so it doesn't make as much sense anymore! I might have to simply use this to locate speeches and manually look them up.

Going back to bigrams and collocations, I also looked at what words showed up around our terms of interest. The results do not show anything particularly surprising, especially with looking through the entire corpus for it.

Suggestion: look at this per party?

Building upon prior shenanigans...

I'm quite happy that I was able to build upon my more mathematical / multivariable / diff geo side quest and implement it in sentiment analysis! So, two weeks ago, I worked with the sample data set (70s / mid-80s) to reduce the computational load in testing out these things. In the previous week, I was able to look at rhetorical trajectories: I was able to track the changes over time, using the VADER module to give a bigger picture of positivity-negativity metrics held against NRC emotional scores like fear and trust.

The paths of the points for each party follow time changes. Though, when I ran this for the full dataset... good lord it was messy!! Next up, I produced multivariable plots where I could look at these metrics. I also then used some Gaussian filters to smooth it out later on.

Similarly, I looked at fear vs. trust vs. time for speeches concerning Ireland, for the Conservative part, for instance. I think this has an interesting data point.

Suggestion: extract the turning point and let the code point to what it is so I can read it up in the original speeches.

Ran this for the entire set as well!

Earlier today, I was able to join in on Pim's meeting with two research engineers. It's awesome to get to know them and have that connection, and it was also really interesting to see how they cooperate with researchers on their work. This is definitely a career avenue I would consider, as I enjoy collaborating on projects (and not being the only one working on it, or leading it). I also don't want to go as far as committing to a PhD, but this sounds like a great place where I can still apply my coding or STEM skills, but still be able to connect with and be involved in history work!

The two women (Shiva and Parisa, if I recall) were also very sweet, and told me that I could reach out to them if I needed advice on my work. I might take up that offer -- even if I have prior coding experience, my experience is far from work with texts or LLMs. There are also so many models out there on the internet when you look it up, and it can be overwhelming. I think having their advice, or reaching out to them for pointers / what they think the most appropriate models are, will be really helpful to my project too!

I also got to hear them explain their BERT modeling, use of LLMs and more. I'll put the link to their GitHub for my own access later. I'm very grateful for a chance to connect with these two women!

Well, more like writing up an internship (back)log for what I was able to show to Pim on our last meeting on the 20th. I'll keep this brief!

Firstly, I tried some clustering, to not-so-meaningful results. I used the TF-IDF approach, where I then applied PCA (primary component analysis). It reminds me of doing SVD back in my linalg class... well, maybe linalg isn't useless after all...

However, the results don't feel very interpretable, as PCA bases its primary axis on the data with the highest variance, which might simply be ones that have a lot of typos or irregularities. PCA would've been very useful if our dataset is perfect -- and that is not the case. Evidently, the clusters produced are also not entirely matching in topic.

Next, I tried to perform sentiment analysis, as was listed in our "next steps" from the prior meeting. I firstly tried using VADER, a sentiment analysis / valence tool. Here, I looked at the overall valence / sentiment. VADER works by tagging tokens with positive or negative word associations. The VADER scale goes between -1 and +1, negative to positive respectively. With the graph below, I was trying to "look around" for general trends or patterns. It wasn't entirely fruitful (though, a good testing ground for VADER), but it is interesting to note that out of everything in all the speeches between '79-'85 -- my test sample -- the average sentiment has been positive. There has not been a single negative average! Of course, this is difficult to interpret as a million topics gets discussed in a year's worth of debates, and different parties "weigh" these issues differently as well. I highly doubt that all the talk was entirely positive.

So, I decided to use a little helper: NRC emotion lexicon. You can check out the metric / website here, but basically, each word is "rated" by a combination of pre-set emotions, and applying the NRC emotion lexicon works through your dataset with that repository.

So, i decided to run it through that! Our data then becomes a little more meaningful.

As you can see, there was very little trust around July-August 1980 in the UK. I can suspect that it has something to do with unemployment being at a record-bad rate, but that would require further investigation as I am only seeing correlation, not cause!

I then broke it down by party, and this is when I got some interesting results! I was surprised by two things.

The Labor and Conservative parties aligned with each other on these metrics a lot more than I expected. Could it be because they were simply debating the same issues? Or could the sentiments they expressed be because they were equally content or discontent with simply different aspects of the same thing? I can only think of a more obvious case in the US -- healthcare. Both the Democrats and Republicans are discontent with the healthcare system as is. The Democrats want something a bit more socialized or accessible, while the Republicans want something with more privatization. That said, by using NRC to rate and plot emotions, it won't read into the context. It would simply see the Democrats as dissatisfied and the Republicans too. Thus, even if those parties have starkly different outlooks, sentiment wise they relate to each other a lot more than expected. And this is my guess for what's going on here between the Labor vs. Conservative parties.

The Scottish National Party, as you can see, has pretty erratic lines. I did bring this up with Pim in discussion, and we hypothesized that there are factors (or, a combination of those) at play. Firstly, it is possible that being a smaller / less represented party, they have fewer speeches and thus a smaller dataset to "average" values on. That said, the Scottish National Party does have speeches in the thousands. Which to me, should at least suffice for a decent dataset. But compared to the amount of speeches that the two larger parties have, this would be nothing! Secondly, Pim brought up the fact that smaller (and more "nationalist" in this sense) parties tend to be extremely outspoken on only very specific topics and issues that they can find themselves intervening in (and not more "generic" issues that take up most of the conversation). It may perhaps be issues such as Scottish independence that are often sensationalist in nature.

Other graphs:

Lemmatized dataset -> main freqs -> clean up more

idea of time

idea of democracy

sentiment analysis (multivariable?)

Some post-meeting recaps that might as well double-up as the weekly / fortnightly recap.

As of my last meeting, I faced some issues with getting Word2Vec to work properly. The error messages suggested some gensim issues. I wasn't sure why, since gensim should work like any other package! That said, it turns out to be a Python edition / compatability issue: I was using Python 3.14, and gensim has been having issues / bugs with it -- according to GitHub and Stack testimonies. So, I downgraded my Python to 3.11, which has allowed me this past week to run gensim and Word2Vec normally! Granted, I had to reinstall numpy, pandas, and all that basic stuff all over again for my now Python 3.11 kernel...

Firstly, now that Word2Vec is working, I was able to get some cosine similarities from a list of defined words. This is something that we can experiment with and change later.

"nation", "identity", "immigration", "culture", "people", "sovereignty", "border", "citizen", "tradition", "community"

From this, we can obtain cosine similarities, that we can then plot or visualize as, say, a heat map.

Next, I tried some PCA (Primary Component Analysis) which basically "reduces" dimensions while maintaining as much of the value as possible. My understanding of PCA is a little limited, but as far as I am concerned -- and correct me if I'm wrong -- PCA looks for the data with the highest variance, one that "spreads out" the most, to generate its first component, PC1. Then the mystery of it goes from there. What I do know is that this is heavily based on SVD (oh boy, I don't miss lin alg)! In SVD, Singular Value Decomposition, you can decompose any matrix into 3 simpler matrices, and the diagonal contains singular values.

Took the image from here. Sucks to not have Latex on Tumblr! I can't help it but to look at the math behind it!!

I guess in SVD you don't have to deal with variance and stuff...

Anyhow, this is what I got from doing PCA:

Again, because of how PCA also tackles variance and factors like uniqueness, interpretability is one of the challenges that I am facing: are my results meaningful? Are they interpretable at all?

Similarly, I share these doubts with my attempt with plotting weighted graphs. I used the networkx package to generate weighted graphs. However, using a package in this way (as I have also discussed with Pim) feels like chugging something into a black box! I have doubts with my results, since unless I go down its documentation rabbit hole I have no idea how these results, knn (if at all), or weights have come about. I don't yet know the internal mechanisms of networkx. I will keep this in mind but probably will not continue with networkx. I could also experiment around more with thresholds, but honestly I am not so sure what all this "means" in interpretation. I'm sure the answers are in the documentation though.

Another way of going about it...

Then, instead of curating a list of words, I tried doing things slightly differently by focusing on certain words and stemming things from there. For this, I looked at two words: nation and Ireland. Via model.vw (and I just love how .vw is called Vowpal Wabbit 🐇 how appropriate for Easter), I was able to find some top neighbors for "nation" and "Ireland."

The top 20 neighbors for "nation" are:

britain: 0.731 country: 0.725 countrys: 0.716 comity: 0.699 oas: 0.697 slither: 0.693 democracies: 0.692 world: 0.690 europe: 0.688 un: 0.682 united: 0.682 americas: 0.671 unite: 0.671 kingdom: 0.665 oilimporting: 0.661 hispanic: 0.658 europes: 0.657 cur: 0.656 worldwide: 0.656 underdevelopment: 0.653

The top 20 neighbors for "Ireland" are:

northern: 0.986 province: 0.864 irelands: 0.771 ulster: 0.743 northen: 0.739 republic: 0.738 nothern: 0.732 irish: 0.692 nationalists: 0.646 allireland: 0.637 foyle: 0.636 protestant: 0.633 dail: 0.628 kingdom: 0.626 republican: 0.615 devolved: 0.612 antrim: 0.610 legislates: 0.609 assembly: 0.608 mainland: 0.606

We can then expand those words and grab their cosine similarities:

(Attempted) Topic Modeling

I furthermore attempted some topic modeling, but didn't go too far. I basically tried to "periodize" those two words that we are trying to stem from, and also tried to look at similarities within semantic pairs. The image on the left is exclusively for "Ireland."

It ain't much but it's honest work!

Side quest

As for the side quest -- where I can be a bit of a GR gremlin -- I am thinking of continuing with sentiment analysis but drawing elements from differential geometry (or at least multivariable). haven't been able to get nltk's pos, neg, or valence to work properly yet with tagging words...

However, this is a rough idea that I have, and instead of "cosine similarities" we can look at Christoffel symbols, covariance (derivatives), plot it against time, see how things have evolved against time, etc.

Advice from the meeting

Some wisdom from Pim!

Keep close to the literature (he has shared with me a lot of interesting literature during the meeting today)

See what pipelines they have used and steps they have taken in making (grounded) arguments

Ruben Ros' works -- statistical methods?

Instead of 10-year models or so, try overlapping models? Also, how do we account for language evolving?

See the library from last meeting for modeling

Generate topics based on adjectives, nouns, adv., verbs, etc. -> pos tagging

Next step: sentiment analysis sounds good!!

Possibly a meeting with research engineers?

Other stuff:

For funsies, I would love to try to make my own generative AI model or chatbot. The caveat is that it will be trained exclusively on this very British parliamentary dataset!

In other (good) news, my vacuum tubes arrived at the right pick-up place! I'm excited to pick them up and build the starved cathode effects for my bass guitar 😉. I can't wait to find a day to go to Radio Twenthe.

Last Thursday, I was also able to catch up with one of the researchers from the QTA class and help her work through the introductory notebook. It was a lot of fun and it was also great hearing about her art history research!!

Topological approaches to data (iykyk)

Arduino / Raspberry Pi actuation for archives (I really need to find time to go to Den Haag)

Re-read all that GR stuff, it's been a while and I feel like my diff geo skills might be rusty! I miss doing pure math things

Just had a meeting with Pim! So, here's what I'm up to thus far, including what worked and what hasn't worked, and what his suggestions are. So, in our last meeting, we talked about Ireland as a potential case study topic, or at least a place we could zoom in and test things on. Again, this is all very exploratory work at this stage of the project.

One of the things I tried involved looking at Ireland-related speeches and counting them per party. That said, one of the qualms I had with my own approach (that I brought up with Pim, and he made a really good point about it) is the fact that I am defining my own set of terms related to thematic things, such as Ireland, or later on, security, sovereignty, democracy, etc.

As of now, I feel like it is a little arbitrary, but I also haven't worried too much since the great part about coding is that you can always come back to change the list of related words later, maybe when more literature-based grounding are found. That said, Pim did mention as well that sometimes, that is fine as we can also say we are defining it from the scope of our expertise. He also suggested that I could manually inspect the speeches themselves, and then see how these words are used in context, then justify my list of related definitions.

Here's an example of what I mean by the list of terms I defined related to something:

conflict_terms = {"conflict", "division", "sectarian", "sectarianism", "catholic", "protestant", "community", "communities", "tension", "violence", "dispute", "hostility", "unrest", "civil", "peace"}

I was definitely a little worried about how the method is scanning for these specific words that I chose ((I don't trust myself to be the credible one giving all these definitions, the only consolation is that code can always be changed later)). Scanning for words would also come with obvious limitations, such as not being thorough enough, or not being able to consider words in context (e.g. tone, sarcastic remarks with that word), etc.

Anyhow, out of all the Ireland-related speeches that I filtered out into an Ireland df, here is the party breakdown of the amount of Ireland-related speeches per party:

That said, I also looked at top words within those Ireland-related speeches for the Conservative and Labour parties. Then, I also looked at other words in the window surrounding it -- basically collocations to provide possible context words. As for counting and scoring things, I obviously used the normalized values. For instance:

I also looked at yearly trends, as an early attempt on computational discourse analysis.

Visualizing them, we get graphs like:

As with most computational things, it's not uncommon to get moot results for a good 30%-40% of plots produced (and the reasons for why they are moot would often be obvious, i.e. historical context or whatnots explaining the "counterintuitive" graphs). Here I am attaching the graphs that are more well-aligned with what we know regarding Ireland at that time.

Nostalgia is an interesting factor to me. One of the questions we have set out to investigate is related to the use of time and "the past." Thus far, I have defined:

nostalgia_past_terms = {"tradition", "traditional", "past", "restore", "return", "heritage", "historic", "history", "once", "again", "old", "former"}

I can't say I am entirely happy with this set of words right now but hey! We can always change it later 🧐...

I am also curious myself what the rising"nostalgia" could mean based on this graph. I think the only way to find out more about it is to manually inspect the .csv files. It might not be feasible to do a close reading on everything when working with such massive datasets, but I don't see another way out of this in making sense of an "increased sense of nostalgia." I guess I could get the code to show me excerpts of where they are scored in the speeches?

That said, we can also look at these same graphs by party.

Not sure if the graph above means much, or if I should also include the one that tracks over time in a more historicized manner, since for what it's worth, this is so far based on my sample dataset of 5 years (which is already large enough to create massive run times).

I also made an attempt with TF-IDF clustering techniques, looked at variance, etc. But, like the issue I had with PMI, one of the axes picked up only upon the extremely unique finds (a.k.a. names and typos) so it wasn't particularly meaningful. I got two plots: an uninterpretable blue blob and an uninterpretable colorful blob that was supposed to give me more party-based information. It did not give me information.

Other meeting notes and ideas:

TLDR: It's about time to lock in on a certain pipeline and go with it.

Talked about sentiment analysis / sarcasm analysis: not directly relevant to this project but a fun discussion nonetheless -- how is this approached computationally? Will we ever be able to do this without training full models or LLMs or GPTs?

GR / topological methods as experimental work

Note on cynicism / irony that I brought up: Pim made a good point about this that even if an MP used that same word cynically, it is still worth pushing through with the same approaches since the MP is still addressing, e.g.) the value of democracy. Of course, this only becomes an issue if we want to look specifically at democratic vs. anti-democratic intentions...

I want to stay more on-track with my research questions.

Try different methods on the same question and compare them for what's optimal

Word2Vec, cosine similarities, historicize the discourse, nearest neighbors

Define a semantic field as a series of words (or Word2Vec and see similar terms and use it as a semantic field). Maybe also just find one of the words in the network (without a need for the "main" word)...

BERT, clustering, light modelling, for the Q: What themes come up when Ireland is mentioned?