Zandy

I programmed "Travelers" from Outer Wilds in Sonic PI! I admit I'm no musician, but this was fun to do.

Each Traveler has a phase-shifted volume cycle, so each fades in and out. Hopefully this is similar to using the singalscope as planets move around the solar system.

I'm working on an application in Ruby on Rails that saves user data and processes it into documents. This is very common in B2B software. Consumer software can keep data in the format most convenient for the developers, where it can be rendered onto pages. B2B software often replaces an existing process and needs to produce a document that looks and feels like the old process, but is faster for the worker to produce.

Ruby on Rails is built for the typical consumer website. It's probably the fastest way to get a CRUD app from zero to live. Rails also likes a monolith. You keep all your logic in your controllers, which react to user actions.

However, Ruby is not particularly fast. That's normally okay for most cases, as Rails is well-written and can render a page fairly quickly.

So you need a strong case to add a separate service. Here's my B2B case:

Users want to download a document that will be printed and posted to the wall

The document is typically well-structured, so some print-specific CSS won't suddenly make the HTML page fit to print.

These documents are posted in paper and sent in email.

Suddenly PDFs are looking pretty good. They are displayed consistently on all devices and don't need a special license to view with consistency (unlike DOCX, which often displays oddly in LibreOffice or a native viewer).

Now, PDFs bring their own problems:

We need a library to produce them. (Does Ruby have such a library?)

They take a discrete amount of time to process and save.

They need to be produced quickly after "Publish" is pressed, but...

They can't hold up user actions (such as page loads) while being processed.

It is fairly simple to set up asynchronous actions that queue up and process over time, so that's an exercise left for the reader.

But here are a couple drawbacks:

We're using up precious processing time making PDFs that could be used fulfilling user CRUD requests.

Ruby's dynamic typing and wimpy type system are great for quickly spinning up applications, but not great for ensuring integrity and avoiding crashes

Making PDFs takes only data in and produces a product - it doesn't have to keep track of state like a web app. Wouldn't a functional approach be cleaner and easier to reason about?

With all this in mind, it seems obvious that PDF production can move to a separate service, and since we're doing this over HTTPS anyway, even a service that's not written in ruby!

With rust's Tokio, it's easy to create short-lived jobs that take only input, send back output, and keep no state. A lack of global state is actually the easiest way to write a tokio app!

There's obviously a lot more to the actual execution, but I'm happy to report that the advantages of a compiled application and strong typing leads to very fast request fulfillment! Every document takes less than 1ms to complete on a very wimpy Raspberry Pi 5. Even if I failed to make document processing async, users would experience more lag from network issues than document processing after pressing the Publish button.

The "mu" means millionth, by the way.

Of course, this took planning and solid logic. No programming language can save you from a bad algorithm or costly calculations. But the rust service is easy to test, easy to reason about, and has strong guarantees thanks to rust's philosophy and tools.

Thanks to a simple retry queue, upgrading this service is also fast with almost no downtime. Stopping the old version and starting the new version takes the service down for about ten seconds. Tokio is lightweight so the service is back up almost instantly. This is fast enough that the retry queue doesn't flinch. Three retries and a dead-letter queue, plus an admin tool for DLQ alerts and manual reruns means I don't have to think about deployment as a downtime issue.

Sedna has an orbital period of ~11k years, as expected! This is probably the most excited I've been about it so far.

I've added light (and thus body phases), which makes everything look quite a bit nicer. Ideally once the starfield is in place, planets will be practically indistinguishable from stars except for their brightness and color!

Believe it or not, there are spheres at all those locations to scale with the sizes and distances.

Walk in the Woods

Well, for instance, you might get your hands on a book of scripts to generate ephemera for celestial events, only to find out it was written for Microsoft QuickBasic for Macintosh System 7. You quickly discover that this particular flavor of BASIC has no modern interpreter, and the best you can do is an emulator for System 7 where you have to mount 8 virtual floppy disks into your virtual system.

You could simply port all the scripts to another BASIC, but at that point you might as well just port them to another langauge entirely, a modern language.

Except QuickBasic had some funky data types. And the scripts assume a 16-bit integer, taking advantage of the foibles of bitfutzery before converting numbers into decimal format. BASIC is very particular -- as many old languages are -- about whitespace.

In addition to all this, BASIC programs are not structured as modern programs. It's structured to be written in ed, one line at a time, typing in a numbered index followed by the command. There are no scopes. There are no lifetimes. It's just a loose collection of lines that are hopefully in a logical order.

So sure, I could port all these programs. But I'm sure to make mistakes.

Wouldn't it just be easier, some basal part of my brain says, to write your own language that some some modern ameneties, that you compile for your own laptop, that kind of acts like BASIC? A language where you just have to translate particular grammar, and not the whole structure of the program?

Of course it's not easier. But I'm already too far in to quit now.

Memory

Who doesn't love manual memory layout?

In QuickBasic, memory is "kind of" manual. The DEFINT and DEFDBL keywords let you magically instantiate types based on their first letter. I don't know how you deallocate things, because all the scripts in this book finish quickly and don't hang around to leak a lot.

In QuickBasic, this looks like

I guess that the second statement just overrides the first.

There is no stack in a BASIC program, so there will be no stack in my language. Instead you just give names to locations.

creates a symbol named age and makes it refer to 0x1F. The pointer operator should be obvious, and the walrus means we're defining a symbol (to be replaced like a macro), not doing a value assignment during the execution of the program. Now we can assign a value.

Atoms infer types. age knows it's an int.

You cannot assign a new type to an atom.

However, you can cast between types by creating two atoms at the same address, typed differently.

The language does not convert these, it simply interprets the bits as the type demands.

Larger types

Not all types are a single word. Therefore, you can use the range operator .. to refer to a range of addresses.

Note that strings are stored with an extra byte for its length, instead of null-terminating it. Assignment of a string that is too long will result in a compilation error.

Next and Auto

There are also two keywords to make the layout of memory easier. The first is :next which will place the span in the next available contiguous location that is large enough to hold the size required. The second is :auto. For all :auto instances, the compiler will collect all these and attempt to place them in an intelligent free location. :auto prefers to align large structs with 64-word blocks, and fills in smaller blocks near other variables that are used in the same code blocks.

String Allocation

Strings come with a macro to help automatically build out the space required:

This is equivalent to:

That is, a string with capacity 5, a current size of 0, and zeroes (null char) in all spots. This helps avoid memory reuse attacks. ZBASIC is not a secure language, but this is still good practice.

There is also another macro that is similar to a "string literal".

Verbose and annoying! Just like BASIC.

Array Allocation

Likewise, arrays have a similar macro:

Which expands in a similar way as strings, with a capacity word and size word. The difference here is that the type given may change the actual size of the allocation. Giving a type that is larger than a single word will result in a larger array. For instance, f64 takes up two words some systems, so array::empty!(5, f64) will allocate 10 words in that case (5 * 2). Larger structs will likewise take up even more space. Again, all this memory will be zeroed out.

Allocation order

As an overview, this is the order that memory is assigned during compilation:

Manual Locations -> Next -> Auto

Manual locations are disqualified from eligibility for the Next and Auto phases, so a manual location will never accidentally reference any data allocated through :next or :auto.

Here is an example:

This produces the initial layout:

Which, after the code is run, results in the memory values:

Note that types are not preserved at runtime. They are displayed in the table as they are for convenience. When you use commands like "print" that operate differently on different types, they will be replaced with one of several instructions that interpret that memory as the type it was at compile-time.

It makes sense. Some boring stuff happens at the beginning, you get into the action until a Climax somehow happens, then things wind down until the story ends. It's a tried-and-true structure, and every story follows it, if your English teacher is to be believed. But I think the shape of this diagram betrays a fundamental misunderstanding of how a story works.

Where is the problem?

In the words "Falling Action."

It's also known as Denouement (the untying), or Resolution, which are much less problematic terms. But the usage of "Falling Action" at all is puzzling, considering that the definition of Climax is the end of the Action. Again: there is no Action after the Climax. All the conflict has been resolved, and there are no fights left to fight. Sure, characters may busy themselves with different tasks, like cleaning up the loose pieces of the plot, but the protagonist has either gotten what they want or they failed to. They won or they lost. They've achieved their super-objective (their main desire, what they most want) or they haven't, and nothing they do afterwards will change that. If there's still a chance of anything happening, then the climax hasn't happened yet. The story isn't over.

Don't get me wrong. Characters can still do things after the climax, but it's all Movement. It's not Action, because Action serves the super-objective, the getting of things that they want most. It's just motion. It's static. It's not going anywhere.

So maybe the post-Climax portion of our story looks a little more like this:

The symmetrical nature of stories becomes much more evident. Stories are about change, but they're book-ended by a static world. To Bilbo, the Shire is peaceful before he runs off to adventure, but largely unchanging; when he returns it's different, but it seems the changes stick. It won't be the same again. The world changes during the action, but not before or after. The beginning is Exposition, which shows the world as comfortable, happy, and maybe missing something. The end is Denouement, which shows the world as changed, new, scarred, and broken, but with the seedlings of some type of hope. Things have changed, some for the better, some for the worse, but one thing is clear: these changes are irreversible.

Let's look at an example: Moonrise Kingdom (Wes Anderson, 2012). If you're not familiar with it, I'll briefly outline it:

An isolated preteen girl, Suzy Bishop, runs away to meet a boy she met more than a year ago. A boy scout, Sam steals a canoe while at a camp on the island she lives on to navigate the rivers toward Summer's End, the northern point of the island on which her house rests. They hike through the woods while the island's only police officer, Suzy's parents, and the boy scout troop sweep the island to find them. Though captured by the adults, they are broken free by the boy scout troop, eventually gaining the support of the adults when it seems Sam will be taken away from the island by Social Services.

Moonrise Kingdom has certainly tricked many people into thinking it is a romance, thanks to it's dual-protagonist team. As more of a coming-of-age story, it seems we could choose one of the kids as taking a journey, with the other as their helper. Sam may be the first choice for many people. Indeed, given a male and female character, most people will focus on the male. Additionally, multiple characters have complete arcs, which muddies the issue further. The question of a main protagonist could be the subject of a lot of debate. But looking at the composition and structure of the film places Suzy squarely in the protagonist box, in my opinion.

The film begins with a montage, featuring her static home life. Her parents ignore each other, and their children. In each of these tableaus, her younger brothers play, while she listlessly stares out the windows in between reading. The inciting incident belongs to her, when she receives her final letter from Sam, urging her to run away. The fight between the scout troop and Suzy and Sam reaches its resolution when Suzy stabs one of the boys with scissors. The fight resolves little between Sam and his troop, which has gone on the same through most of the film, but fuels a pivotal moment where her parents lose status due to her daughter's violent acts. Suzy becomes even a mother-figure for the boy scout troop at one point. With Suzy as protagonist set out, let's look at how her beginning and ending moments reflect each other; how her action is completed at the end of the film at what that means for her character.

The Denouement re-visits this same home as the Exposition, even re-using the same camera locations as before. Suzy reads again, this time accompanied by her chosen family in the form of Sam, who has snuck into the house. Where before, Suzy's mother shouted up at her with a megaphone, now both parents share this role, unified. We follow the same beats, many of the same motions, and the world is familiar, but something about the motions is distinctly different. Partly, things have changed: her parents, for example. But for the mostly, it's our characters who have changed. Suzy and Sam, with the context and lessons learned from their journey, live their lives differently. The status quo has been destroyed and rebuilt. Many of the pieces remain, re-arranged in an unfamiliar configuration. There's a status quo on both sides of the action, which serves to demonstrate how the action affected the characters.

Wes Anderson practically bashes the audience over the head with symmetry. It is often noted how each frame focuses on symmetrical placement, but the symmetry hardly ends there. Even the music of the opening montage forces the audience to think about the symmetry: it is Leonard Bernstein's Young Person's Guide to the Orchestra, in which a narrator explains how a simple theme changes with different orchestrations and arrangements.

A story is symmetry. It has reflections across many lines. It's something that's not totally obvious, especially given what we're normally told about stories. Thinking of plot and Action like this makes the reflection self-evident, and hopefully helps in the creation of an ending. If the ending is simply a re-arrangement of the beginning, writing an ending becomes a much less daunting task.

And the super-objective? There's no way of knowing what Kara Hayward was using as her super-objective for Suzy during filming, but if we had to guess, it might be "gaining autonomy", "taking charge of her role in her relationships", or similar. So has she achieved this at the climax? It seems so. She grips onto Sam, the one she's most afraid to lose, sure she will jump off the church, in hopes they can escape through the flood. Finally, she lets Sam have some distance, with the understanding she'll be able to keep him around. She gets what she wants in some ways. She's still restricted to her home, but with the help of some adults, is able to take a more active role in her relationships with her family, related and chosen. This is the completion of the action.

And perhaps this is obvious to a number of people, but it wasn't to me from everything I'd been told about story from English teachers.

There’s a real problem with names: there’s no consistency. Family names may come before or after given names. Want to address a Chinese person by Mr. [Family name] but in casual context by [Given name]? Well, maybe you can let the system set either the first or last name as the given or family name. But that’s only one problem.

With that in mind, I’m going with a really lax approach: provide your name as used in a handful of different forms and the system will use them as appropriate for form letters. I plan to support unicode so that you can set the name to anything a computer can handle.

This doesn’t solve all problems. I know that. But I’m one person and I can always change my database schema later.

Examples, given the fields I provide:

“Zandy clocked in at 12:30 pm and clocked out at 4:21 pm”

“Mr. Lowry, please find your contract attached to this email.”

“Zandy (Alexander Lowry)” as the header of a folder containing documents meant for me.

Full name vs. Legal name?

A lot of government forms have really wrong assumptions about names. I know someone with a single-letter family name who needs to add an extra (incorrect) letter to her name to satisfy terrible validation which assumes a family name must be at least 2 characters. I know someone who has no last name, who goes by the last name X. The person’s full name may be different than the name they must fill out on government forms. I want to include the legal name to help with joining with outside data.

What is going on with the pronoun field?

Since the category select doesn’t pull from the pronouns table yet, you just get to choose a number from 1-20 instead. The big issue with pronouns is noun case. Maybe I’m overthinking this, but in English we’ve got nominative, accusative, and two forms of genitive (think “this bag is hers” vs “this is her bag”, and again with third-person-plural/third-person-singular-neuter). Form letters etc. may have a bit of complexity when referring to people. And German has dative as well, and Latin has ablative on top of all that, and Sanskrit has vocative in addition, and I don’t think I’ll ever support Turkish or Hungarian but it’s good to not be anglocentric about these sorts of things.

Hence the pronouns table with columns for each supported case.

Why not a simple gender select?

Maybe that will be relevant for casting and costuming, but I don’t care about that. I care about making hundreds of form letters at the click of the button and hard-coding things into form letters is always going to come back to bite you later.

Here’s a single volume unit. This is perlin noise but it’s too small of an area to really see the pattern.

The code to generate the terrain. The ground() function gets the ground level. The blocks structure holds all the info about blocks, including texture atlases. Blocks are organized in the library by “domain” - e.g. default, colors, debug, which are collections of blocks along a common theme.