#distributed systems

Community care becomes intelligible as infrastructure the moment the analytic lens shifts from the isolated household to the dense and interwoven networks that make collective survival possible, because no dyad, no family, and no domestic arrangement has ever functioned independently of a broader ecology of reciprocity, shared knowledge, distributed labour, and collective risk management that…

If recurring cloud service fees for hobby projects cause death by a thousand cuts, I'm feeling like I did the first few times Ashina Elite - Jinsuke Saze kicked my ass in Sekiro:

...but my SRE day job long ago scared me off single points of failure, so I've been hesitant to host services that real users depend on--mostly https://life4ddr.com and https://truebpm.dance--at home.

But then we moved to our new home, where the prior owners had installed a couple of 13.5 kWh Tesla Powerwalls in the garage, and I realized I had power redundancy for the first time. This got me thinking about what it would take to build a sufficiently HA environment at home for hosting community projects and, well, there's work left to do to answer that question.

Might as well start with...

Redundant ISPs!

Step 1: ISPx2

Step 1 is easy: We pay $80/mo for AT&T Gigabit fiber as our primary ISP. I've had a (deactivated) Starlink RV dish and have been waiting for the chance to use it.

The primary downsides of Starlink for RVs are a) the slightly higher cost than residential Starlink ($135/mo vs. $110/mo for residential), and b) RV users are throttled during periods of high demand. In practice, this has never happened to me; I'm on the waitlist for a residential plan.

Starlink is fairly fast:

And the latency to google.com is eehhh, ok:

So why not another fiber or broadband provider?

I've been enjoying not paying Comcast too much

The Starlink dish was sitting in my closet

They're hobby projects, not payment processors that need 5+ 9s of availability and sub 10ms latency

(the primary line kinda never goes down anyway)

Off the griiiiidddd

etc.

Installation is a cinch if you cut corners:

Put it on the roof

Do a great job running and hiding the cables, definitely no eyesores here (not pictured)

(sorry to devon)

Through the wall and into the router

First time using this stuff to seal the hole drilled for the cable, plus some Sikaflex concrete sealant since, unlike regular silicone caulk, it can be painted once dried.

There may be an embarrassing part omitted here involving drilling into an "electrical wire", panicking, and in the end discovering that it was chicken wire & part of the stucco...

Not pictured: cable grommet for the 1" hole that has not yet arrived from Amazon

"That's a giant hole" Yeah, because of this thing :(

The official routing kit comes with a 3/4" drillbit. I used the 1" drillbit I had on hand.

Final result:

Step 2: Automated failover

At this point, I had a separate SSID that I could manually switch devices to, but I didn't want to have to deal with doing that manually when I'm away from home. And I might not always be available, and don't want users waiting until I am...

Enter the Ubiquiti Dream Machine (UDM) Pro!

Ok, so I really just copied a friend here without doing a ton of research:

But it ticks the boxes:

WAN failover (WAN load balancing not supported... but we can handle a few seconds of downtime)

Remote management interface

...That's it?

It does way more than this, but my goals are not lofty.

Once it arrived, it only took a few minutes for initial setup:

Plugged primary modem into WAN port 2, SFP+ 10gb (with an RJ-45 adapter)

Plugged Starlink router into WAN port 1 (with a Starlink ethernet adapter)

Turned on the UDM Pro, paired via bluetooth and finished guided setup through the iOS app

I then changed the port configuration to make WAN 2 primary and WAN 1 secondary. In my head this felt like a step toward >1Gb home Internet (AT&T offers 5 Gb today 😱). In practice, this led to a lot of packet loss and continual failovers to the backup link; it's probably an issue with the adapter or cable somewhere in the chain but i didn't feel like figuring it out.

I then discovered that you can configure port 8 as the secondary WAN link, and shuffled connections around so both WANs were cabled without the need for an adapter. This fixed the failover flapping and packet loss.

Somewhere along the way I also changed the echo server from the default Ubiquiti server to Google DNS (8.8.8.8). Ubiquiti pings the echo server to make the decision to fail over or not. I'm not sure this contributed to solving my issues, but it has for some

Success!

And kind of a sick name...

I didn't have to simulate failover because I broke things plenty of times during the setup process:

In practice, I saw 5-10 seconds of packet loss before Starlink took over. And automated recovery when AT&T started pinging again! More than adequate for users of a DDR BPM calculator.

What's left?

So this could be a pretty big deal in terms of making ammonia production greener. But what I think is equally impressive is how it can help decentralize ammonia production:

MacFarlane added that the new technique had the added benefit of being scalable to both small and large applications, opening up a wide range of potential uses, including in green fertiliser production.

“You don’t need a huge chemical engineering setup. They can be as small as a thick iPad, and that could make a small amount of ammonia continuously to run a commercial greenhouse or hydroponics setup, for example,” MacFarlane said.

“It means that the distributed production of fertilisers becomes possible because the ammonia manufacturing unit is so small and simply constructed.”

A few weeks back I also wrote about new technologies which are making it easier, more cost effective, and more robust ways to literally pull water out of the air. And we’re all familiar with home-energy systems, like solar panels or wind turbines.

Just imagine a farm, 20 years from now, that does not buy water or fertilizers or energy, bc it just produces its own. Maybe it produces its own eco-friendly pesticides - or may even uses robotics to manage weeds and pests. Imagine how this tech could help people in developing areas which lack sufficient infrastructure.

And who knows what other chemicals we may start producing at home. Maybe we’ll develop some general purpose material producers, where users can ask the device to turn ambient air into a variety of substances - like producing different types of plastic spools for home 3d printers.

https://mises.org/wire/why-we-need-free-markets-fight-pandemics

Some ideas we had:

If you're reblogging as part of a conversation with someone, ping the person you're conversing with over IM. Of course, some other people you didn't ping might be composing replies!

Copy the post's text, delete the post, and then repost it with your edit. I think this is worse than #1, since it doesn't abort the in-progress reply effort.

I think it's impossible to prevent all effort wastage once you've posted, since posting could immediately kick off a reply.

I came up with the following, unfortunately very busted, algorithm, which I will now spew onto tumblr in an effort to nerdsnipe y'all into devising a better one:

Consider how your post could trigger a reply. Most of the time, it's going to be via the post showing up in the dash of someone who's following you. So,

1. Let S := your set of followers 2. Go through S and message every user in turn that you're about to edit. 3. Once you've done that, reload the post's notes. If there are no reblogs since you last checked the notes, you're done! Post your edit! Otherwise: 4. Someone's reblogged your post. Oh no! Their followers could be replying! 5. For each of the reblogging users you found in #3, download their set of followers. 6. Let S := union of all sets from #5. 7. GOTO #2

This is not guaranteed to terminate, since you're racing the outward wave of causality from your ill-considered post (you're probably hosed if your post goes viral). But, assuming you're fast enough, you're eventually going to halt the damage.

Unfortunately, this doesn't actually work because:

You can't actually enumerate the set of users who might have seen the post. Nonfollowers could see it on "in your orbit", or search, or tag subscriptions, or by navigating to your tumblr URL. This probably isn't fixable, so let's assume your followers are the only users that see your posts.

Someone might see your post between the time you look at your notes in #3 and the time you post your edit.

#Hashgraph consensus time stamps -- Emmet LIKES !!! "The #blockchain is obsolete, the future is #Hashgraph" http://sawconcepts.com/index/id91.html