#opensource

wireless e ink x particle tachyon, solar fed llm...

Today, we’re abnormally jazzed to announce that we’re open-sourcing the custom framework we built to power your dashboard on Tumblr. We call it StreamBuilder, and we’ve been using it for many years.

First things first. What is open-sourcing? Open sourcing is a decentralized software development model that encourages open collaboration. In more accessible language, it is any program whose source code is made available for use or modification as users or other developers see fit.

What, then, is StreamBuilder? Well, every time you hit your Following feed, or For You, or search results, a blog’s posts, a list of tagged posts, or even check out blog recommendations, you’re using this framework under the hood. If you want to dive into the code, check it out here on GitHub!

StreamBuilder has a lot going on. The primary architecture centers around “streams” of content: whether posts from a blog, a list of blogs you’re following, posts using a specific tag, or posts relating to a search. These are separate kinds of streams, which can be mixed together, filtered based on certain criteria, ranked for relevancy or engagement likelihood, and more.

On your Tumblr dashboard today you can see how there are posts from blogs you follow, mixed with posts from tags you follow, mixed with blog recommendations. Each of those is a separate stream, with its own logic, but sharing this same framework. We inject those recommendations at certain intervals, filter posts based on who you’re blocking, and rank the posts for relevancy if you have “Best stuff first” enabled. Those are all examples of the functionality StreamBuilder affords for us.

So, what’s included in the box?

The full framework library of code that we use today, on Tumblr, to power almost every feed of content you see on the platform.

A YAML syntax for composing streams of content, and how to filter, inject, and rank them.

Abstractions for programmatically composing, filtering, ranking, injecting, and debugging streams.

Abstractions for composing streams together—such as with carousels, for streams-within-streams.

An abstraction for cursor-based pagination for complex stream templates.

Unit tests covering the public interface for the library and most of the underlying code.

What’s still to come

Documentation. We have a lot to migrate from our own internal tools and put in here!

More example stream templates and example implementations of different common streams.

If you are a writer, an artist, or a creator on the internet right now, you aren't paranoid. The extraction is real.

The platforms we used to love have turned into harvesting machines, scraping everything we make to feed algorithmic models that are being used, ultimately, to replace us.

Let me be clear as someone on the front lines: the technology itself isn't the enemy. This is the corporate model. And just like the cryptography that can be used to protect your work, this is a profound misuse of a powerful tool.

I spent 8 years as a lead engineer inside the belly of the Big Tech beast. I watched the roadmap. I saw where this was going. So a few months ago, I walked out. I evacuated the burning building so I could spend my time knitting the water hoses.

Today, I’m quietly turning the first tool on.

app.daon.network is live.

DAON (the Digital Asset Ownership Network) is a sovereign ledger built specifically to protect independent creators from non-consensual AI ingestion.

“Pinchflat can technically download individual videos, but its strength lies in archiving entire channels and playlists at a time. For starters, Pinchflat lets me add channel and playlist URLs as sources, and I can modify the indexing frequency, download cutoff days, and retention settings. On conventional YouTube downloaders, I’d have to pull new videos manually every so often, while Pinchflat can archive fresh uploads after detecting them with its indexing algorithms.”

Personally I have been using TubeArchivist for this, but there are some key differences between the two:

Pinchflat acts like a “Sonarr for YouTube.” Once you set up a source (channel or playlist), it downloads the videos, names them according to your template, and saves the NFO and poster images. It uses low resources.

You install the TubeArchivist Jellyfin Plugin. This plugin talks to the TubeArchivist API to pull in metadata, thumbnails, and watches progress. TubeArchivist has its own dedicated web UI to browse your YouTube collection and is a bit heavier on resources (Requires Redis, Elasticsearch, and the App container).