OH BEARER MINE

shaman visits shrine with proper offerings: auspicious wind reveals cache of gold coins buried under layer of dust

It's been true for a long time that the academic project has accumulated more information than any person can readily comprehend the scope of, so if you want to learn something the immediate challenge is figuring out what is already known about it, and where to find it. Library science is structured around this problem, and has some interesting approaches and answers! Mostly following a sort of decision-tree model, where the organizational categories are necessarily somewhat arbitrary, the important thing is that they're clearly defined enough that you can ask a series of questions about "it is this or that?" and arrive at a location.

But the set of arbitrary categories does unavoidably influence the way knowledge is generated and interpreted, the "silo effect" of different fields of study not knowing how to talk to each other about their areas of overlap, and not necessarily even thinking to check what areas of overlap exist and who they could and should be talking to.

And hypothetically digitization provides the potential to completely change this up! You can build database structures that can be filtered by many different criteria simultaneously! You can look for certain keywords wherever they appear regardless of category!

But then you face a whole new conceptual headache, which is figuring out how to categorize things in a parallel access structure. And you quickly have to contend with the fact that different fields are often using completely different words and conceptual approaches, and how do you link these things up across the database? How do you provide relevant context for what kind of background framework is in play, for someone coming in with an unknown level of background knowledge in potentially a completely different approach?

And then - this is where I get unclear on the historical timeline. For some reason, from the early digitization grappling projects to the internet I grew up on, something happened where the internet got very full of surface-level answers, and also extremely technical answers, but there's a huge slice missing in between, which is SPECIFICALLY the college level "introduction to how this field is thought about, including slightly more than surface information but also an orientation to the framework of thought which has been built in academia so far, and which will be useful for knowing how to find and interpret more technical information on this subject." This type of thing is INFURIATINGLY difficult to find on the internet, and has been even since back when google worked. There's a handful out there of thoughtfully curated websites - usually hosted by some university where someone took the trouble to build out an explainer for their field of study - but they've always been random lucky unicorns to find. As a fallback you can try downloading some random textbooks and reading them, but I can't call that a good approach.

And then, in parallel to that, there's the search engine thing. Google early on hit a really interesting and compelling strategy for sorting webpages by their relationship to a web of knowledge - but the proxy for this was a web of links, which rapidly became Extremely gameable. For a while they were on top of managing this, somehow, I don't actually know the technical details.

There was a brief window where search expanded from just including your exact keywords, to triangulating sets of concepts related to your keywords - this was like 2015? There was a fleeting moment where a search of several meaty keywords could actually tell you what the field of study you wanted was, could point you to things slightly beyond the scope of what you'd asked for, which was so cool. And then fairly rapidly over several years this collapsed into showing you instead things that were related to vaguer, less technical versions of your keywords, and then eventually to the point where if you had more than about 3 keywords it would just start ignoring most of them.

This seems to have happened in parallel to a hard turn towards microadvertising and to google itself as an ad service. I have a suspicion that google stopped TRYING to get ahead of the gaming of their searches and instead just leaned into it, because this brought them more add revenue via content aggregators who ran their ads and also gamed their searches aggressively. Resulting in the rapid search engine sloptimization of everything.

Meanwhile database structural approaches really lagged, because search was so easy and powerful through google for a while, that straight keyword searching felt clunky and high-effort in comparison. There could have been a lot of work in this field that just... failed to materialize, as far as I can see.

Instead the data science world leaned in hard on machine learning - which could offer some interesting approaches! I think not unrelated to what 2010s google was up to actually! - but then all that got swept up in the push to natural language models. Which is INCREDIBLY frustrating, because the output you're going to get from a language model for a search query is AT BEST equivalent to asking some guy who's read approximately everything ever to summarize of the top of its head. In practice it's worse than this, because a language model does not actually have a conceptual understanding of what it's read, a fact which is only semi-compensated by the sheer volume of information it's processed - if a clear conceptual approach is strong enough in it's text sources it's statistically likely to cough that back out, but that's entirely a gamble. But even aside from that, this completely fails as an indexing strategy because it can't send you anywhere! It can't tell you where it got anything! It's a total dead end on the project of organizing and accessing the body of knowledge which exists!

WE BUILT THIS RESDAYN