Oh No Computer

If you decide to take the plunge, I encourage you to chat on the mesh, and don't get discouraged when messages don't seem to be heard. Take notes on where mesh coverage seems lacking, and coordinate with the community on how to grow it. Community provided infrastructure needs us all to collaborate to make it a reality. I'm happy to attempt to answer any questions y'all may have - if we are not already connected, you can find a few ways to reach my via my personal website which is linked from this blog.

If you are in the Bay Area, or Black Rock City during the summer, I hope to see you in the party line chat. I will be the cute nerd sending messages about sending test messages from the durian display case at the local grocery outlet.

It is the responsibility of the associated business product owners to ensure that both core and casual project contributor have appropriate tasks present in their scrum backlogs. There is an underlying assumption here of the organization empowering engineers to collaborate on projects which do not only solve their current problem du jour. With a tight feedback loop between product owners and community managers, the links between project tasks and business needs will remain in sync. With an organization that is empowered with aspects of prioritization flexibility, a more sustainable software ecosystem will no doubt result.

A underlying theme of my work these days is encouraging the formation of sustainable adhoc organizations to further our shared goals. We have seen progress but at speeds which many are unsatisfied with. While I may personally notice the timeline stretching out almost into infinity, there has been no sign of the end goal changing. I look forward to working more with communities at large as we adapt to fostering more and more adhoc organizations.

This project doesn't yet provide such an experience. If you are the kind of nerd who has access to a shared Vault server and sometimes needs to share secrets with people, then you should check out lachash. If that isn't really your jam then just stay tuned because I've actually found the thing which will motivate me to finally delve into the realm of mobile development.

In 2015 one of the most important things I learned, career-wise, was to have less opinions in the workplace. Don't get me wrong - I still maintain approximately the same number of opinions. What has changed is the number of opinions I am actually willing to dig in my heels and fight for are diminishingly few. One concept that is part philosophical and part technical I am unlikely to ever to concede is that of semantic versioning. The concept of semantic versioning, now in it's second major revision, is one which strives to bring an ability to sanely coordinate the complex interactions incurred by interactions amongst disparate software systems. By taking into account such wide reaching concepts ranging from backwards compatibility to the sort of incremental changes typical of day to day software evolution, it structures workable transitions between variants of the same piece of software. I won't go into the all of the specifics of semantic versioning, as there is a pretty good writeup on their website. Another thing that happened in 2015 is I switched jobs several times, and ended up implementing the same piece of software three different times. The first two times I wrote this software occurred while I was gainfully employed. The third time occurred as I slid into a period of funemployment, and I made sure to slap an MIT license on it so I would never have to write this software again. This software, Avakas, represents my codified opinion on how semantic versioning should be used in a build and release pipeline. Avakas is designed to allow one to easily query and manipulate version metadata from software projects in a variety of contexts, across a variety of programming languages and frameworks. Currently, it supports Erlang (rebar app.src files) , NodeJS (package.json), Ansible Roles (git tag only), and a "plain" version file. I added the plain version file because as it turns out, not every piece of software I use provides a clean way of specifying versions. Fortunately, everything seems to provide some mechanism (either directly, or through good ol' UNIX environment variables and lesser shell sorcery) to read in static information from a file. I am sure the reader will be surprised to note that Avakas is moderately opinionated in a few key ways. One of the more drastic opinions it enforces (probably not for ever) is a requirement on Git for source control. Before any modification is made, your Git repository must be clean. After any modification is made, the changes are committed, a git tag is made, and then everything is pushed to the specified remote. One invokes avakas by specifying (at a minimum) two arguments; the action and a directory to manipulate. There are three actions; show, set, and bump. There are several options which are shared between these three actions. You can specify the Git remote and branch, as well as a prefix to use for the Git tag. I've tried to align the defaults for these settings with what appear to be best practices for using Git. On the actions which involve making changes, there is also a dry run mode. This will make all the modifications locally, but not actually commit or push anything through Git. The action which spurned this kind of integration with build and release pipelines is that which shows the current version string (always immutable and dynamic, never static). On top of the general options, you can also ask to append either the build or pre-build version components. This is where another opinion kicks in - While I firmly believe that prebuild and build version components are incredibly useful, I do not believe they should be must be persisted in source control. Thus you can specify either one may be included, however they will not be written to the filesystem or pushed to git. While there is no written record of these builds enshrined within source control (build systems generally keep logs), each build artifact must absolutely remain immutable. The next most commonly used action is that which will automatically adjust the version number based on the type of chance. When invoking this action, you specify a third argument that corresponds with the version component to be modified. The core components set by semantic versioning are major, minor, and patch. The final action allows one to explicitly set a version number. I tend to only use this when initializing or migrating projects to use avakas with a plain version file. I've been moving many of my personal projects to use Avakas, and have started using it at my current place of employment as well. I also use this tooling to interact with the version metadata of Avakas itself, which makes for a nice self-referential example. While I realize that nobody likes having opinions forced upon them, I do sincerely hope that these codified opinions are closely enough aligned with those of others for some to find this software helpful.

Once during a change of employer I found myself in a new environment featuring pre-existing infrastructure for which I was destined to implement a replacement. This existing infrastructure had been designed and implemented with CloudFormation and Ansible (the same tools I would later choose). Unfortunately, the deployed infrastructure had entered the realm of once automation driven but now mostly manually managed. Such environments provide contextually unique challenges when it comes to replicating what already exists. Challenges such as never being certain if your automation routines will do exactly what is printed on the tin. If one is are lucky, the automation tooling chosen will support an audit (or dry-run) mode which can provide some elements of safety gates. As often the case, going back to automation on a live system after a period of manual handling is generally more of a dice roll than the level of determinism we sometimes dream can be expected from our interactions with technology. My initial task was to update the automation, bringing both the Ansible and CloudFormation components up to date, not only improving usage of underlying technologies, but also giving consideration to a more long-lived deployment infrastructure. At some point between starting on this task and it actually deploying we had to scale the existing infrastructure up. Fortunately the tooling that Amazon provides for their EC2 service around this kind of operation is actually not completely terrible. Over the course of one particularly foggy morning I was able to manually perform two simple steps using the keyboard and mouse attached to my laptop. I can remember thinking at the time how bloody simple executing this task by hand had been. By creating what is essentially an infrastructure mold one can create new copies of existing piecemeal infrastructure and weave them together in meaningful fashions. It was a great reminder of how easily it could be to once again fall into the trap of creating, and later copying, hand crafted systems. The initial hand crafting of a custom computing system starts out harmless enough. These hand crafted systems may be freely copied to ones hearts content. Many modern cloud providers provide the necessary constructs to facilitate these actions. The trouble begins to come about when talk of updates begin. Relatively complicated concepts such as backwards compatibility and demographic based availability suddenly become a little more onerous. Subsequent steps generally involve actually adopting automation technologies, or painstakingly working your way by hand through runbooks detailing step by step commands. And then the process of crafting new infrastructure molds and the creation of new copies begins again. A core piece of the narrative around the DevOps movement has been automation. By providing the opportunities for operations and product engineering groups alike to define an idealized infrastructure many of the historic delays around creating new systems has been removed. The reality is that such a dream, while not always an exercise of tilting at windmills, generally involves one of two extremes. At one edge is to fully leverage externally defined process-as-code regardless of whether or not it may be ideal for ones team. The other extreme is to adopt a full development project to manifest an ideal automation construct tailored exactly to what the team believes is needed. The closer a group is to creating their own ideal automation stack the longer it will take for the initial changes to be ready. The more you can leverage external, already existing resources, the quicker you can move once unleashed. This is why many early organizations begin with simple deployments that are based around shuffling code around and then, once the organization has stabilized, migrate into something more reminiscent of shuffling around idempotent build artifacts. The important concept to remember is that while crafting infrastructure by hand may feel easy at the time, the long tail effect of such work is often obscured. While certain styles of engineering spiked projects may sometimes deserve no more than a manual implementation, most projects are meant to last longer than a sprint or two. In order to ensure a long term sustainable maintenance life cycle for infrastructure endeavors automation must be a first class implementation consideration.

When pursuing technical goals born of far reaching project life cycles finding the balance between fundamental architectural changes can be quite difficult. While it is important to not constantly be changing the underlying tooling at the whims of our emotions, it is equally dangerous to lay prone at the altar of antiquated technology. In a similar vein ensuring a minimal amount of overlap in the different ways of pursuing goals can reduce the siloing of tribal knowledge and the resultant downward spiral of overall productivity that may inevitably result.

One common devops interpretation is the implementation of dedicated operations engineering teams focusing on automation technologies used to maintain deployed software. On the surface, this almost seems like a natural reaction. Teams that already have developed the tribal knowledge from actively working with production systems, given the chance to build upon the shoulders of pre-existing automation giants, should be able to manifest any and all tasks involved in shipping an organizations product. With some teams, given appropriate workloads, this may very well be the case. In reality, the dedicated devops team approach can end up so far from the desired target that it is essentially a cliche of how to not approach the challenge. One would expect an increased deployment velocity; instead artifical barriers between departments become reinforced and the ability to rapidly iterate on existing product components is hampered. The approach of a dedicated devops team tends to develop barriers between product and operations groups with resulting backlogs of product components ready to leave the nest save for some requisite devops engineering which occurs isolation. The inter-departmental relationships that arise are functionally the same as they always have been. Operations and product groups end up completely disconnected. Communication, if teams are lucky, occurs solely through built software artifacts. Product provides the limestone blocks, while operations assembles the desired pyramids. It is certainly possible to build a renowned product on this model; operations and product have traditional been isolated throughout the history of the software technology industry. It's also quite likely that the organization will end up in a much better place; devops does place an incredibly high value on automation and this will buy you, if nothing else, a sound disaster recovery strategy. More likely is that shifting product requirements will result in teams being unable to address technical debt or perform maintenance on existing systems as they will be overwhelmed with requests from the product driven side of the house. Deployed automation components stagnate, code paths diverge, and entropy eventually sets in. There are many ways to approach sustainable collaboration between product and operations engineering teams. One particular flavor is straight forward and jives with the concept of microservices (which are quite hot these days). Instead of having a team dedicated to build and deployment pipelines for not only the underlying systems but product components themselves, have the team instead focus on reusable automation building blocks. Modern automation technologies allow organizations to separate out underlying architectural components such as installing bespoke artifacts, or wiring up monitoring and alerts. The resulting product-related components end up being succinct and generally differ only in name and app-specific configurations. Given an emphasis on tying these building blocks together, coupled with heavy emphasis on self-generated API documentation and continuous integration style testing, ecosystems evolve in which product engineers are able to produce value in a frictionless fashion. At this point engineers are able to construct their own deployment strategies based on well maintained building blocks which are iterated on by the entire organization. What results is a team dedicated to providing their colleagues with tools and pre-assembled blocks with which to deploy and manage their software components as opposed to addressing challenges thrown over the wall from other departments. These tools may manifest as template-based infrastructure generation, re-usable software automation routines, API's built around any number of automation concepts, or any appropriate combination. Obviously the implementation is going to differ between organizations yet there will always be philosophical parallels. Product engineers can be expected to interface with this componentry via an API or similar social contract. The operations engineering team can now focus on addressing underlying platform concerns and support of the building blocks they have provided for their customers; the overall engineer organization.

A single point of failure is any entity that, upon sufficient disruption, causes part or all of a shared reality to collapse. Even if the failure is minor the point may endure as a liability that could result in a complete breakdown should other precipitating factors coalesce. These situations are inevitable and the best strategy is adaptive management. In the world of software engineering, this flavor of situation often involves (so-called) legacy knowledge or software. Cases that I have experienced include a near retirement COBOL programmer who had diligently maintained a payroll system for decades, and the loss of specialized engineers who were fluent in esoteric programming languages that were necessary to keep startup-legacy software components alive. The resulting losses can but may not necessarily be catastrophic. This is particularly true of legacy software that runs well because it has been finely tuned over decades. As such, there are few maintenance requests, and their resolution is routine or possibly even fully automated. Stories abound on the Internet of software and hardware having gone largely un-maintained for decades, yet continuing to serve their original function. One particularly extreme case involve a semi-forgotten servers that had been sealed behind drywall for years. As the world moves on, adaptation via end-user ingenuity circumvents inherent system flaws. It turns out that humans are remarkably adept at working within the constraints of a dreadful user experience in order to make their work more bearable, let alone possible. However, the other end of the spectrum is less pleasant. Perhaps the technology in question is being used in a constantly changing environment – one in which entropy rules supreme thus preventing appropriate longevity related work on the system. In such cases, despite inordinate levels of energy being invested in keeping archaic or otherwise fragile systems alive, there remains inevitable decay and loss of functionality. This may be due to a lack of necessary resources, a lack of expertise, or perhaps insufficient diligence in adopting use of the latest and greatest concepts - be they technological or other. One of the ways that nature addresses the inexorable force of entropy is through redundancy. By manifesting an alternative entity that can function in the same (or similar) capacity, there can be a form of reliability in the face of otherwise disruptive events. In an ideal situation, this adaptive stability allows the system to remain functional. Useful actions can range from expanding a response team, through automating job functions to the level they may be triggered by a simple push of a single physical red button. When chasing engineered achievements of various forms, there will always be single points of potential failure. The management difficulty lies in identifying and addressing the most hazardous risks, while adapting to any that remain. Failure to do this will inevitably lead to systemic decay, from which sparks of life may not endure.

A common theme across many aspects of the tech industry is overwork. Between crushing deadlines and funding instability (especially with smaller organizations), the impetus to throw oneself full-on into the project is easily found. Stories abound featuring long days of planning meetings coupled with even longer nights spent trying to accomplish something - anything, really. At the extremes you will encounter diligent employees who have brought their own sleeping accoutrement into the office. After all, if you are going to sleep curled up in a fetal position beneath your desk, it may as well be in familiar pajamas and sleeping bag. While these periods of overwork are typically cyclical in nature, the cadence can vary wildly. Sometimes a cycle spans across months of intensive labor with only a brief flash of downtime until the next. Other examples may find an extremely high tempo during the last two weeks of every quarter of every year going back for decades. With exception given to incident response, this seemingly immutable additional workload tends to manifest around dates tied to clients, investors, other external entities. Without knowing how to properly estimate a development team’s realistic output one may end up rapidly overwhelmed with work, but with the same immovable, etched in stone, dates. When I have personally found myself in such situations the team has generally pulled through in the end. The anecdotal data I've collected from discussing this subject with colleagues and friends suggests the same. Most (but certainly not all) eleventh hour all-in endeavors seem to succeed, for some value of success. The definition of that success is crucial: it is not uncommon for the success metric to become warped from it’s original intent. It is certainly not uncommon for acceptance cases to slip once a team is at the very edges of complete mental and physical exhaustion while tantalizingly close to the prospect of crossing the finish line. Generally the heroic efforts of the team is then acknowledged. Sleepless nights deep in the depths of creating something from nothing while not engaging in normal social patterns are rewarded with some form of financial compensation, career progression, a pat on the back, or, worst, nothing at all. Many organizations recognize these heroic efforts in front of a larger subset of employees. The love and thankfulness of colleagues, while mild, can seemingly act as a narcotic for some individuals. They begin to feel like a hero - having had the audacity to choose work instead of sleep in order to reach a milestone. The novelty generally wears thin quickly as work moves on. After indeterminate periods of time, the cycle will begin anew and employees find themselves burning their midnight candles at both ends, producing value while praying that the light at the end of the tunnel is not, in fact, a train. The details of these cycles can vary wildly. Some themes, however, remain steadfastly constant, not least of which is sleep deprivation. Many an engineer, artist, or analyst has bemoaned our species’s stubborn refusal to evolve beyond the need for sleep. And yet somewhere out amongst the masses of humans across this globe, we will find them sitting in a nameless office block, barely noticing as the sun rises yet again through a high rise window, staring at a blinking cursor on the screen. Not having much rest over recent weeks, and no longer knowing how long they have been staring at the cursor, their mind has wandered to completely unironically wondering whether the cursor itself is mocking them. Any moment now they will resume the seemingly sisyphean tasks ahead, only to inevitably make one mistake which will cause their goal to slip further away into the aether... Most mammals, as it turns out, need to maintain an appropriate balance between wakefulness and resting. Failing to do so has numerous well known effects both short and long term. Cognitive impairment is one which comes to mind, given the context of making complicated decisions under pressure. This is a self-reinforcing cycle in which bad decisions made during exhausted work sessions will inevitably lead to instances of further sleepless nights and further bad decisions. When downtime between cycles is so small as to be practically insignificant, organizations need to start paying attention. It is easy to end up in a situation where every action item is priority and absolutely must be completed and thus the very concept of a real crisis soon becomes moot. At this point, the work output of the organization will become focused around simply maintaining enough forward momentum to reach every new goal, rarely if ever finding time to improve upon what already exists. Granted, the trial by fire aspect of occasional crunching deadlines can not only help form bonds but also provide valuable insight into workload planning. As this is not what tends to happen, the work from sustained high tempo operations may be of lower quality than what may otherwise have been produced. While some workers seem to thrive on this constant cycle of pressure and corporate hero reward, others are simply too new to the industry to know any better, and even more feel shackled to their jobs by by tight job markets or financial obligations. Those who have been in the industry for a longer period of time have likely seen that not every work environment features a non ceasing workload. They may be more easily tempted away by offers that allow a more appropriate work / life balance. They may quit in a moment of emotional and physical exhaustion. Or they may finally burn out and consistently produce lower quality work. By allowing a culture of overwork and heroism to persist, organizations remain vulnerable for waves of regrettable attrition as more experienced workers follow their colleagues to greener pastures.