#exocortex

Yes, I know that's not what Shakespeare meant. Not like that's ever stopped anybody in marketing or people with too much money and hungry for more.

Okay. You've been inundated in what the industry is calling AI technology for months on end. I don't need to introduce it because the only way you could have avoided it is to have been in the middle of nowhere for the last year 1 or so. I would ordinarily have said "in a coma" but the way word gets around it would surprise me not a bit if folks would be talking about this, that, or the other AI thing at the bedsides of coma patients, to say nothing of the televisions that are always turned on in hospitals these days.

Yeah, that went a bit off to one side for a minute. Keep that in mind, I'll bring that up later.

When LLM technology finally got effective marketing going for it ("artificial intelligence") it was inevitable that two major markets would make heavy use of them: Click farms and scammers (whether or not the former is a subset of the latter is out of scope for this article). Just as spammers implemented "I'm not a spammer - honest!" measures almost immediately and everybody else struggled to catch up. If it can be abused, it will be abused as soon as possible. Nowadays you can't swing a dead cat inside your search results without hitting at least one site that's all AI slop and zero useful information. Self publishing has gone from a somewhat questionable (in both quality and subject matter) industry to a potentially lethal race to the bottom because, no matter what, publishers are making money regardless. More and more students are using LLMs rather than their brains to do their homework (which defeats the whole purpose, which is to say, practice what you've learned so that you understand it better) and driving their instructors nuts. What isn't being mentioned much is those students listening to their instructors talk with each other about using LLMs to come up with problem sets, homework assignments, and exams and thinkging, "Hey, my teacher uses ChatGPT to come up with homework assignments, why can't I use it to do those homework assignments?"

As for business... LLM technology is getting shoved into everything with a CPU in it, or nearly so. Samsung started with Bixby, which everybody went out of their way to disable as soon as possible because it was as helpful as Skippy and twice as annoying. Microsoft stuffed Copilot into Notepad for fuck's sake. It's almost like they thought Vigor was a good idea and decided to run with it. Or maybe it was because companies that own a lot of physical compute capacity wanted to get more money for it existing, and found another solution to a problem that doesn't really exist. I was talking with a friend the other day, and both of us seem to recall a paper published back in 2017.ev or so that discussed various ways of getting more money for hosted processing power.

It's hard for skilled folks to find jobs these days. There have been multiple waves of layoffs in recent memory but hiring has been significantly lesser in magnitude. Multiple CEOs have been going on the record saying that they have LLMs writing large amounts of code and have gotten rid of organics to cut costs. Copilot (at the very least) was trained on lots of open source code on Github, and I think it's a safe bet that other constructs were also so there is at least something substantial to back this up. I've heard talk that this is being done to justify pounding the salaries of coders down to a fraction of what they are; I don't think that's the intended purpose of it but it might happen as a second order effect. It won't do anything about bringing down the costs of living in those places. Supposedly this is supposed to usher in a wave of unbridled innovation, the market will go wild, the line will go up for ever and ever, and everybody that matters will get rich.

Something isn't making sense, though. Lots of folks got laid off and are having trouble finding jobs 2 and there is a nontrivial number of people who've been out of work for over two years. That means that a lot of folks who are ostensibly the intended customer base of those companies, are living on fixed incomes in effect - living off of savings while they hunt for employment. To put it another way, they're not spending lots of money on those nifty-keen-like-wow LLM powered services. They're not starting their own companies, either, because that takes money; money that pays your rent. 3 Where are those customers supposed to come from? How long are folks supposed to be able to live in those places which have gotten so expensive even when you have a job? Is this all a shell game where companies give the same money to other companies in a vast network of closed, tight loops? Even B2B only goes so far before you need users outside of the business space. Remember, if that line Must Go Up, B2B won't keep that happening.

Things don't exist in a vacuum. Things are connected to other things, which are connected to stll other things. Cause and effect, and second order effect, and third order effect. If you only think no farther ahead than the next fiscal quarter, if you're more concerned about your exit strategy and negotiating your golden parachute... okay, let's put it this way: The purpose of a company is to make money. A gross oversimplification, but let's go with it. Very few people being able to spend lots of money, not starting companies, and probably not using those services are not going to give your nifty-keen-like-wow LLM powered application money. Companies that don't get customer money don't stay in business. They go out of business if they don't get forced out of business from not having customers. Assuming, of course, that they can't pull a few strings and get themselves bailed out.

I think that, all of that said you can see what I'm concerned about. Another house of cards coming down, a lot of people losing everything through no fault of their own, and a small number of people harvesting more for their hoards.

I suppose that as a cyborg I'm being a bit of a reactionary writing this. I might be. Something that I think is important, and you are welcome to disagree with me if you want, is the efforts of organic, living beings, imperfect as they may be. Those blemishes are the important bit. Mistakes, happy and otherwise, which cause you to change what you're doing and go in a different direction. Trusting the process and not the plan. 4 Stumbling over things in mid-step and saying to yourself "You know what? Fuck it. I'm going to try something that makes no sense." Coming up with new ideas in the middle of working with other ones. The false starts, the do-overs, the taking your time so that you can see what happens.

That bit earlier in this post where I briefly talked about veering off in a different direction for a bit? That happens sometimes. When you freestyle a post of this size (or maybe a bit bigger) that happens occasionally. Usually that gets caught and deleted (or at least smoothed over) in later passes over the text. Something that gets drummed into you in school, and which is very useful in college, is that after you've spent some time working on something you're writing, set it aside for a day or so, come back to it, and read through it. You'll find things to change, stuff to delete, things that should be moved from one place to another, places where you forgot something. I didn't go back and rework that part deliberately, because it shows imperfection. The organic touch, if you will. Sure, Windbringer and Leandra helped me write this, but they didn't write the text. They manage and search my archives of knowledge, look for references for specific things, and both back up and version control everything, but they didn't write it. I did.

That is one of the reasons I'm not a prolific blogger anymore. While I do try to post at least once every month I don't like to write when I don't have something worth saying (which I define, because it's my website). I could write analysis of stuff going on in the world, but there is no shortage of what passes for talking heads on the Net doing that very same thing and most of them do it for money, anyway. This is a hobby that I partake of because it makes me happy, not because I'm trying to build a brand or get advertising revenue (though I do like the occasional Amazon credit, it helps me build cool stuff once in a while) or anything like that. And to that end that's why I don't use LLM tools when I write posts. I do this for fun, because I enjoy it, because I like sharing things that interest me with everyone, and using LLM tools would turn this blog into just Another Goddamn Thing I Have To Do, another chore that goes faster if I automate parts of it, more IT housekeeping... I'll pass.

What I'm trying to say is this: It's quality, not quantity. Blood, sweat, and tears, not plugging a three line prompt into a web site and getting a few thousand words to copy and paste into a content 5 management system. Brilliance and experiments and passing fancies and imps of the perverse and fuckups and good ideas at the time. Bad puns and links to things you might not have known exist. Not clout chasing and making people jealous of your trip to Rome where you crammed three weeks into one just so you could have material for a dozen Instagram posts throughout the year. A willingness to try and fail, to make mistakes and look like a dork.

Being an imperfect being that does things because you want to, and not someone who pumps out vapid text because you have to.

I wish to extend hearts-felt thanks to my beta readers who gave me feedback and suggestions when I got stuck about halfway through writing this article. You know who you are.

I think hiking the Appalacian Trail counts for the purposes of being off the grid for the last year. ↩

Ghost jobs aren't a new thing. Over the years I've worked for three or four companies that not only did this to make it look like they were growing (which made them more attractive to investors) but it was written into the corporate handbooks as company policy. ↩

The days of saving up ten grand to start your own company are long in the past. ↩

Tip of the pin to Adam Savage for that one. ↩

Note: I used the tag 'searx' for this post even though I've been using SearxNG for quite a while. There's enough compatibility between the two that the stuff I've written (so far) will work. However, I haven't decided if it's worth the hassle of changing the tag and possibly making things harder to find.

A constant problem when you have a sizeable external memory is finding what you need, when you need it. It's a problem that I've been poking at for a while and, which I probably don't have optimal solutions I've found a couple that work well enough for me, and hopefully might help a few other folks.

A while ago I started using Bookstack for my personal wiki, and I've quite fallen in love with it. It's everything I need, just about nothing that I don't, and runs nicely in shared hosting. I've also been using Searx, and later SearxNG as both a unified search engine and a search API. I've written a bit about how to interface things with Searx over the years; a certain amount of trial and error is involved because, while SearxNG has excellent facilities for connecting to stuff they're not always well documented. So, let's start at the top:

Bookstack's REST API includes access to its internal search function. You have to set an API token and secret per the documentation. The way that you use those two things is that you send them as part of an HTTP Authentication header in your code (whatever that happens to be). It would look something like this if you did it on the command line:

curl -XGET -vv -H "Authorization: Token <API token>:<API secret> https://bookstack.example.com/api/search?query=thing%20you're%20searching%20for"

Note that your search term is URL encoded. A positive search result is a JSON document that looks like this:

{ "data": [ { "id": 985, "name": "Shaarli Bot", "slug": "shaarli-bot", "book_id": 8, "chapter_id": 0, "draft": false, "template": false, "priority": 184, "created_at": "2024-02-18T22:19:55.000000Z", "updated_at": "2024-07-08T22:38:17.000000Z", "url": "https:\/\/bookstack.example.com\/books\/projects\/page\/shaarli-bot", "type": "page", "tags": [], "preview_html": { "name": "<strong>Shaarli<\/strong> Bot", "content": "...tamp of the JWT. UTC. time_t format. Only good for nine (9) minutes.\n{\n"iat": time_t datestamp\n}\n\n\n* Base64 of +\n* A single period (.) + \n* Base64 of \n\nSeriously, just use a library to do it. It's easier.\n\nhttps:\/\/<strong>shaarli<\/strong>.github.io\/api-documentation\/\n" } }, ... ] }

As search results go, this is really straightforward to parse. We can send this to SearxNG's json_engine, which doesn't have the greatest docs but is pretty easy to figure out with a little trial and error. Basically, you give it a JSON document, tell it where the essentials live in the JSON, and make sure it's enabled. Here's the config block that I use (with, I should add, a few directives that I don't for documentation purposes):

# The name is arbitrary and must be unique. - name: bookstack # Use the json_engine to do the thing. engine: json_engine # This is also arbitrary and must be unique. shortcut: bs # Time in seconds before SearxNG gives up. timeout: 120 # If anything goes wrong, display the error in your browser. display_error_messages: true # Categories this search will appear in. # More than one category must appear in a [ python list], as below. # Single and double quotes can be used but aren't mandatory. categories: [ external memory ] # Search results can be returned one page at a time. paging: true # URL of the Bookstack instance's search API. # {query} == URL encoded search term # {pageno} == page number of search results. Defaults to 1. search_url: https://bookstack.example.com/api/search?query={query}&page={pageno} # In the JSON document returned from the API, the key or path the search # results can be found under. For Bookstack, this is 'data' but can be # left out if it doesn't apply. # This corresponds to the JSONpath # $.data results_query: data # Where inside the JSON document returned by Bookstack the URL to a search # hit can be found. # Mandatory. url_query: url # Where inside the JSON document returned by Bookstack the title of a page # with a search hit can be found. # Mandatory. title_query: name # Where inside the JSON document returned by Bookstack the matching text # of a search hit can be found. Tracing a path to a key is done in the # form key_1/key_2/.../key_n # The example below is the equivalent of the JSONpath # $.data.*.preview_html.content # or, relative to $.data.*, # $.preview_html.content content_query: preview_html/content # The maximum number of search results per page. number_of_results: 20 # Is this thing on? disabled: false # An optional set of HTTP request headers. # In this case, authentication to Bookstack's REST API. headers: # Name of header: Authorization # Value of header: # The word "Token" # Your Bookstack API token # A colon (":") # Your Bookstack API secret Authorization: Token <API token>:<API secret> # Optional information about the thing being searched. You can leave this # stuff out if you want but it does help document things for later. about: # URL to the thing being searched, or the thing's homepage. website: https://bookstack.example.com/ # Wikidata ID code for the thing being searched. # https://www.wikidata.org/wiki/Wikidata:Main_Page wikidata_id: Q107122654 # URL to the official search API documentation of the thing. official_api_documentation: https://demo.bookstackapp.com/api/docs#search-all # Using an official API or something else? use_official_api: true # Does the API require a key? require_api_key: true # file format of the results. results: JSON

You can create new categories of search engine by just using them, but they won't show up on the SearxNG page unless you add them to the categories_as_tabs: list in searxng/searx/settings.yml.

The upshot of all of this? Search for "!bookstack nfc rfid":

Search engine: bookstack

Search terms: "nfc rfid"

I mentioned not too far back that I'd finished migrating my wiki over to a new piece of software, but it was a little outside of what I'd been trying to accomplish in that post. It seemed a good idea to circle back and explain what I meant by that.

Don't get me wrong, I quite like Pepperminty Wiki. It's a fine piece of software - lightweight, configurable, it uses flat files for storage, and it's nice and snappy. Especially in situations where the web hosting provider is badly over-provisioned and moderately complex web applications tend to bog down. But after a few years of using it I found that I needed a little more. A wiki that was more mobile friendly (and not necessarily through the use of a mobile app), that made uploading files a bit easier (in particular, photographs), and which had a nicer text editor. I like Markdown but when I'm trying to write notes I don't necessarily want to have to micromanage formatting, especially on my phone. Maybe I'm getting a little cranky in my old age 1 but sometimes I just want a WYSIWIG user interface so I can get things done.

I'll skip the parts about experimenting and trying to get a couple of other packages to do what I wanted (because I wrote about that already) and cut to the chase. I wound up going with Bookstack and after about a week of messing around with it I decided that it was the right tool. It was pretty simple to set up and the instructions are straightforward. One thing I'd like to point out is that, while technically shared hosting isn't supported by Bookstack I didn't have any trouble with it. This is partially due to A2 Hosting (note: this is a referral link; if you sign up I get an account credit) having a control panel that lets you pick which PHP extensions are turned on for your account. 2

Unlike a lot of wikis Bookstack doesn't try to be overly abstract when it comes to organizing pages. Bookstack gives you shelves, books, chapters, and pages. Pages are pretty self explanatory. Chapters are collections of pages; chapters are optional, incidentally. Books are collections of chapters and pages; you can have books that are nothing but pages. Shelves are collections of books; I think that shelves are optional. When I was first setting up Bookstack, I created a single book because I needed a book ID for my utility (which wound up being the number 1) and uploaded all of my pages into it. Now, here's the thing: You can stop here if you want. If one book in Bookstack works for you, that's fine. Just because a feature is there doesn't mean that you have to use it. However, I wanted to move stuff into loose categories to at least give me an idea of where to start looking, so I set up a few books and moved pages around. In every book I created a chapter called Archive, which I then moved stuff that I was finished with into (but didn't want to delete). That way, finished projects didn't get in the way of pages that I'm still using often.

One thing that Bookstack has going for it is that you can either go WYSIWYG, or you can edit documents as Markdown. This means that you can either pump Markdown files into a Bookstack installation using the REST API (with the /api/pages API rail and the markdown option), or you can pull your pages out of Bookstack as Markdown documents if you want to (using the /api/pages/{page ID}/export/markdown API rail). So your stuff isn't trapped in Bookstack in a format that you'll have to write a translator for. I wrote a quick hack in Python that uses the corresponding Python module to take a bunch of files and upload them into Bookstack. It's not polished, you'll have to edit the code to configure it, and you'll have to get yourself a pair of API tokens for your setup, per the API documentation linked above. Then you just have to run it with a bunch of .md files:

{17:35:53 @ Sun Mar 24} [drwho @ windbringer wiki] () $ . bookstack/bin/activate (bookstack) {17:36:01 @ Sun Mar 24} [drwho @ windbringer wiki] () $ ./upload_wiki_pages_to_bookstack.py wiki/*.py

Something that I found I had to do when I was done testing was that I had to update the links in all of my pages to point to the 'real' domain that Bookstack now sits on. Thankfully you don't have to do this manually. This is a feature which the Bookstack devs built into the Artisan tool, which comes with every Laravel application and it works very well. I made sure that I had one good backup already (which never hurts) and then used the following command:

{17:39:22 @ Sun Mar 24} [drwho @ windbringer BookStack] () $ ./artisan bookstack:update-url https://old.wiki.example.com https://new.wiki.example.com

(Not having a trailing slash in the URLs seems important.)

Then to make sure that the search indices and disk cache were up to date, I manually re-indexed my wiki and cleared the cache:

{17:40:20 @ Sun Mar 24} [drwho @ windbringer BookStack] () $ ./artisan bookstack:regenerate-search {17:40:44 @ Sun Mar 24} [drwho @ windbringer BookStack] () $ ./artisan bookstack:regenerate-references {17:40:51 @ Sun Mar 24} [drwho @ windbringer BookStack] () $ ./artisan optimize-clear

All of that said Bookstack has been really solid for me. I haven't been this impressed with software in a long while. It's one of the first things I open when I start my browser, and it's been indispensible as a knowledge management system. If I'm figuring out a problem, I come across something interesting, or if I have something that I'd like to throw into a scrapbook of sorts, it's my go-to tool these days. If you're looking into setting up a wiki I'd strongly recommend going to the Bookstack demo site and playing around for an hour or so. You might find yourself pleasantly surprised.

I'm a sysadmin, it comes with the territory. ↩