Luminary

Mr Paul Dix @ Influx Data has asked the OSS community to “get real”.

My summary is “hey we’ve got to make some money and feed it back into the tech”. Fine. But would you buy support from a company which admits that maintenance is a dead end?

The “open core, closed extensions” model that their entire company is based upon, as he admitted, has to be driven by funding that comes from paid support. If in his view, this is not the way to scale the company to a particular size that he wants, then I would be forced to conclude that he has seen the “light at the end of the tunnel” and nobody should be deploying Influx commercially today in new applications.

Essentially what they’re arguing for, as I can perceive, is to achieve an altruistic monopolistic position. From a position of monopoly (instead of truly a position of strength) certain plays can be carried out, much like how an acquired firm can be put in run-off until all of its remaining customers naturally churn. The company could still be making some money in the interim, but it is a horrible place for a creator to be.

Furthermore, sometimes the design of OSS philosophies have inherent, intentional inefficiencies, which are required as checks and balances, and can be mistaken for deficiencies. Basically, some business models can not be built on some components with specific licenses without further effort. This seems fair to the creators. People who control financial leverages should be civilised enough to not want to take every last dollar.

Furthermore, again from what I understand, OSS projects can be used as an insurance policy, to ensure that an engineer remains relevant within multiple professional sub-circles, only one of these belonging to their employer. Therefore, I am not a fan of the post picking on Matt, insinuating that were he looking for another job, he’d have to try and negotiate ability to work on his existing OSS project. No, from my view, the OSS projects that bear employer logos die when the creators leave. There are way too many examples to list them all.

In conclusion, total alignment is a myth, stemming from the need to control. The Influx Data post shows exactly that. They think centralised organisational control is a good thing for OSS, because it guarantees control, and thusly funding, making it acceptable for commercial use. However, the countercultural streak of OSS is, by design, against the grain, so as to reward risk takers. With centralised control, there are no longer relative risks, and the inherent reward that comes with successfully navigating these risks will also normalise to zero.

The Perfect Implementation Fallacy, put in plain words, is an advanced version of the “If you build it, they will come” fallacy which can infect software system providers who want to create ‘platforms’.

If the platform maker creates a new component in hopes that it will be widely adopted and bring in profit, but also requires that third party integrators follow outstanding (written and implied) guidelines impeccably in order for all of that to work, you may consider the platform maker’s thinking muddled by such a fallacy.

Such a fallacy is not always deadly if the platform maker has enough cultural pull (for example, Apple’s iOS App Store in 2008) to ensure a critical mass, or a deep pocket to ensure that the platform maker is the last one to die, after driving all other inferior approaches out of the market (i.e. being the one that sucks the least).

Ironically, if the newly introduced component or API were to be properly restrained, it will also lose expressive power. This is because design-time constraints from the systems provider both a) put developers on a guided path, and b) hinder serendipitoous, exploratory, high-beta development.

Bruce stated in the opening keynote at ElixirConf 2015 that the pendulum is swinging back and forth between a fat client and a fat server. After a chat with a few guys in a bar I think it warrants a further discussion because of the following reasons.

In the beginning you have a beefy central node responsible for all communications, exchange of messages, routing and offering general control. This node interacts with many client nodes that are dumb and wholly dependent on the central node. The central node must not go down — a requirement for Erlang.

Then came the explosion in user-local computation power offered the end user additional possibilities and personal computers proliferated, creating a market for self-contained applications. The Internet made the world multi-polar and peer-to-peer applications appeared during Web 1.0. The world rediscovered paradigms invented by the three muskeeters of Erlang.

Then Web 2.0 came along and cemented single-page applications that rely on a beefy client node for nuanced control and feedback which would be prohibitive to process on the server all at once. Then came mobile and the inherent unreliability and energy requirements on a mobile device pushed most of the heavy lifting back to the central node. Peer-to-peer distriution replaced with CDNs and BitTorrent replaced with browser downloads over a gigabit connection.

The world has gone mad and made JavaScript king of the hill. It would seem the pendulum is swinging back but this is not entirely true. The pendulum metaphor itself is going rapidly out of date because the client and server are now working together:

Technologies developed in the “big iron” era assumed unreliable clients and focused on coalescing all processing on the server.

Technologies developed in the “fat client” era assumed that the client can do more and tried to offload more embellishment to the client.

Both the server and client are actually growing in computation power and maximum allowed complexity.

This means you can mix technologies and mindsets developed in different eras and create a best-of-breed hybrid that has a higher reliability, lower latency and more importantly, higher resilience.

News has it that Vesper is shutting down. The application itself is obviously well-engineered so the root cause can be attributed to either founder burnout or an insufficiently built business case.

From my observations the iOS ecosystem is currently consisted of the following categories:

Utilities that complement a client-server workflow, i.e. Calendar, Mail, Twitter.

Applications that allow consumption of media or time, i.e. Infinity Blade, Movies, Amazon Prime Video, YouTube.

Utilities that encapsulate a complete workflow and optionally allow sharing of interim editable versions elsewhere, i.e. OmniGraffle, Word, Excel.

Utilities that encapsulate a complete workflow but do not allow editable versions to be shared elsewhere, i.e. iMovie.

You’re in deep trouble if the only iOS product you have is also the only product you have. UIKit is a superior choice to HTML and App Store superior to Nginx when it comes to application design, development and delivery, but I still firmly hold the belief that iOS applications are either loss leaders or loss generators, that iOS devices themselves are thick terminals, and that a proper iOS execution strategy must be backed with a useful service either involving real world consequences (i.e. get a ride or get groceries delivered), or a wider cross-platform strategy (i.e. build your document on one platform, revise on another).

The Software as a Service (SaaS) model flourished due to an increasingly casual attitude regarding data, workflow and infrastructure sovereignty, accompanied by renting software capabilities becoming indisputably and increasingly economical, compared to building these capabilities in house, which requires significant investment in money, time, and political capital should efforts go wrong.

Having things done by ‘computer people’ who are ‘better with computers’ is an easy way out for institutions who have time and can spare the requisite monies, just like how companies can justify spinning out non-core competencies to other companies, sometimes located in far-flung geographies.

However, an increased reliance on software written externally, and software creation and maintenance capabilities paid for and de-risked by a third party, may spell the demise of an institution should the competition master the art of creating custom software at a higher level of performance and efficiency. Those who are serious about their own software will build their own hardware, and those who are serious about their business will write their own software.

* * *

An ISV sets its own priorities and acquires customers in order to meet its own economical goals. In other words, if it is more economical for an ISV to disregard a feature request than it is to implement it, the feature will not get implemented. The customer is supposed to vote with their wallet and to award their upcoming contract to a more capable provider, but not without loss of time, effort and other inconveniences during the transition period.

The real reason why ISVs can survive on providing software is pretty much like how insurance companies get rich. The Software as a Service model is an insurance policy, paid for by institutions either afraid or unable (lacking in money, skill or prestige) to write their own software solution.

Let’s talk first about the ones that are afraid of writing their own software. I would wager that if you corral a bunch of techies they will not be able to tell the difference between ‘visual programming’ and a good joke. It is a legend that every year some random startup pops up and tries to make programming so simple that ‘a sales person can write their own program’, yet few have succeeded and you can find names for ones that have on consultants’ business cards. Writing new software is complex, messy, and politically unsafe in any corporation not run by software engineers; there is no stable timetable to point to and it is too easy to make sweeping changes so we all make them all the time. Most new software projects fail, and the ones that do not fail require time and effort to support. In any case, a new software package written and used is a new package that is now in your care, left in your team’s capable hands, ready to page you at 3 in the morning for whatever reason it fancies.

It is no wonder that we are afraid of writing software. It is much easier to hire an external team to write software and to maintain it, so when things go wrong, all fingers within the company can point in the same direction and the appropriate amount of monies docked from the upcoming support contract payment.

Next, money shortage that makes writing software in-house nearly impossible. This is further complicated by the fact that ‘good programmers’ want more money than ever and it is increasingly unaffordable for a fine corporation to pay through the nose just to move some bits from here to there. The average cost of a very small software team will have to cover a project manager, an engineer or two, a tester, a business analyst and a document writer. Even if you put all these hats on a few people you’re looking at an expense close to $500k a year; if you have hired people who over-engineer then double the amount twice. It is no wonder that software subscriptions then start to look very palpable.

Lastly there is lack of prestige, because prestige can blind a young and bright mind and make a software engineer work for less than she’s worth just to pad her CV. A stint at a well-established company can net the worker with a lifetime supply of recruiter calls, all these recruiters working for the same big and small companies that supposedly understand software and try to out-compete each other by throwing increasing growing amounts of monies against the same pool of engineers. With no prestige, it is less likely for an average company to hire software engineers at the correct level of caliber that is required for the project’s survival. The smaller the team is, the higher caliber you need in the people who are put on the project, because the project is already more likely to fail and rookies get you killed.

Doing a $50-a-month deal with a website which provides support suddenly sounds like a miracle come true.

* * *

In the startup circle there was a running joke: Uber for this, Airbnb for that. Outside of the startup circle I am now seeing the same gag: SaaS this, SaaS that… SaaS for this business model in this country… that thing, but SaaS. Money chasing problems waiting to be turned into a specialized SaaS package which is no more than a collection of good-looking forms.

We are in a war of SaaS-everything and there will be blood, blood being the steady erosion of personal and institutional digital sovereignty. In the very first incarnation computers were built to solve specific problems and machines were built to crack codes and win the war. Networking between computers was designed and developed as a national security project of great import. Then the personal computer came along as a box which can do anything, the Internet came along and captivated the mass entertainment market at lightning speed. We got amazing hardware nowadays driven by amazing software and technologies all wrapped up as services, purveyed by vendors with supposedly genuine intentions, and we trust that companies will not be too evil. It is all too convenient to not write our own software and too beneficial to just enroll in the group insurance policy that is SaaS.

* * *

The personal computer was built as a truly personal device. In order to operate it, you must understand it thoroughly. Deep penetration of the personal computer — one computer on every desk — is basically impossible with this level of understanding being an absolute requirement. Slowly and gradually the end user is asked to understand less of how to work with the computer and the computer is asked to understand more of how to work with the human. Whilst this can be seen as a triumphant (and profitable) breakthrough, it can also be considered a species-wide risk, as now the power to manipulate and control information is cemented within few hands capable of understanding the true nature of computing, fewer mind its real implications.

I have previously blogged about how to build sticky headers for UICollectionView which mainly concerns when to pick up clues to transform each header element, and working out the minimal and maximum offset extents you can use to shift each element.

I recently had to implement sticky headers for a Web-based UI which was fraught with much peril. It is my hope that the following notes will be of your use.

General Idea

A ‘sticky header’ discussed in this article is something in a long, usually scrollable list that stays fixed to the top edge of its clipping view. In lists with multiple segments, each segment may have its own sticky header, and when the end user scrolls they will expect the upcoming header to push an existing header ‘out of sight’.

Usually, you will end up implementing a repositioning subroutine which is invoked by some kind of event callback. On iOS this can happen in -layoutSubviews on your scroll view subclass where you query the offset, on the Web it may happen when your onScroll event handler is fired.

Within your callback, you will usually take the following steps:

Find all headers that need to be repositioned. This is usually done by first inferring where all headers normally rest if they were not to be sticky. The usual resting position for each header and each section is then considered, and you will end up with a collection of sections that are currently visible in the clipping view.

Each currently visible section should have its header visible, and if its header is currently out of bounds, it will need to be repositioned. The second step is to determine, for each header view, how much transformation needs to be applied so the header remains visible. At this step, you will compute the minimum and maximum extents that the header view can be in, and through your own derivation you will end up with a precise offset.

After all the offsets are computed you will apply the transformations to each header view.

Shifting Elements Properly

In order to write the subroutine mentioned above that makes your headers ‘sticky’ you will need to figure out:

Where each header element is at

Where each group holding its header element is at

Where the clipping view is at, and what its offsets are

After all that information is gathered you can compute the proper offsets, and you will need to figure out:

A way to quickly shift each header element that needs shifting to the right position, and reset each header element, that does not need shifting, back to its usual position.

Let’s dissect each step. First of all there is more than one way to figure out where an element sits, but the best way to know about an element’s position is by querying getBoundingClientRect() (MDN, MSDN, eJohn, CanIUse). You may find that offsetTop is faster, though, and it may work for your use case. The thing about getBoundingClientRect() is that it also considers all transforms applied, which you may want to note for later.

By using getBoundingClientRect(), you get measurements and fulfill prerequisites 1, 2 and 3. Afterwards, you can write a subroutine as follows:

If a group is entirely out of view, nothing in it requires shifting.

Otherwise the header of that group should be sticky and the offset can probably be computed as:

desiredY = Math.min(Math.max(accordionRect.top, containerRect.top), accordionRect.bottom - headerRect.height)

This allows you to make the header sticky but not protruding. Note that by using getBoundingClientRect(), all measurements are relative to the viewport, meaning that you will not need to do any kind of transformations yourself: all of that work is normalized for you, at a very reasonable performance budget.

Now you know how much to shift each element, by comparing the desired position with its current position.

currentY = headerRect.top offsetY = desiredY - currentY

There is more than one way to shift an element, but there is only one way which is the fastest: translate3d() (CanIUse, TreeHouse). You can apply this as a transformation like this:

headerElement.style.transform = "translate3d(0, #{offsetY}px, 0)"

Just be careful that, as previously mentioned, any transformation will be taken into account when computing getBoundingClientRect(), there are two solutions around this:

Explicitly empty all transformations on all elements that could possibly have been transformed — giving them translate3d(0,0,0).

Note down previous offsets and deduct these offsets when computing resting rects.

I went with the first approach as it is surprisingly fast, anyway.

Scrolling, Event Callbacks and Compositor Behavior

Now you have a repositioning routine going, it is time to look at when to apply it. If you simply ran this code in scroll you may be surprised to find that it is quite jittery in Safari, or that it is inconsistently jittery on a couple devices and not other devices.

The main problem with approaching this for the Web is that most things can now be hardware accelerated, and the true meaning of a scroll event has changed. In the old days, the scroll event is always fired and the entire clipped area is repainted, so when you get the scroll event, you are sure that the underlying content has shifted by an integral number of points and that you can rely on it for your calculation. You are also sure that your layout changes will be committed, and most importantly your code will be called on every single frame that the browser has to compute.

Since repainting is too expensive and end users want to flick through an entire document quickly, the solution was to utilize the compositor and scroll asynchronously. In other words, the scroll event can be fired asynchronously and may not fire on every frame that the compositor does work in. For example, in Safari on El Capitan, the Scroll event is fired every other frame whilst the compositor works on every frame.

In plain English, this means that if you shift your headers around in a scroll event callback and rely on the browser to call your code on every frame, it won’t work. Your headers will jump around and look jittery. Your product starts to look like it’s made by sad elves.

If you’re not particularly resourceful to possess multiple different configurations to test your product on, you may not even discover this issue. (For example, the latest Chrome works fine and calls scroll on every frame, but when the same version of Chrome is run on the new MacBook it does something different.)

I personally think that this change was, ironically, only required because people were doing too much stuff in their scroll handlers, so in order to achieve smooth scrolling, the browser engineers had to make certain things asynchronous.

WebKit #45631: Scroll event should be fired asynchronously

You may think of using the wheel event but it’s even worse. The root cause is that your code is not in control of layout on every frame. The solution, therefore, is to regain control without sacrificing the smoothness afforded by hardware-accelerated scrolling.

There is more than one way to reposition a couple elements in a scrollable content area. Earlier in this article we determined that using CSS transformation is the best way to reposition a header; you may be tempted to make these elements position: fixed and reposition them manually, but that is pure madness because now you have to do an insane amount of housekeeping to ensure that the underlying layout flow still works, and it is still not repositioning on every frame. If you have another application framework on top of this (like Ember, or AngularJS) which generates elements this approach is nearly impossible.

There is only one solution left, which is to circumvent the browser and build your own scrolling handler ‘properly’. That is a lot of responsibility.

Controlling Every Frame

You must be in control of every frame. The solution to header repositioning jitter, introduced by idiosyncracies, is to implement hardware-accelerated scrolling, mutated by your own event listeners only, with deterministic guarantee that each offset phase invokes your repositioning code.

In other words, the scrolling wrapper should only update its offset when your header repositioning callback has worked out how to shift visible headers so they remain sticky.

This is not a new concept. Back in the old days, Cappuccino used a transparent element, covering the entire viewport, to accept events and to proxy them, so first responder support and other Cocoa-like event handling logic can be ported properly. To build your own scroller, you will need to catch all kinds of events that usually result in the wrapper scrolling its content, and offset your child (or children) accordingly.

Fortunately, you can do this quite easily with iScroll as long as you use the Probe Edition. (The author of iScroll is quite proud of the many ‘faces’ of iScroll.) You can initialize an iScroll element as such:

iScroll = new IScroll $element[0], mouseWheel: true scrollbars: true fadeScrollbars: true probeType: 3 disableTouch: true disableMouse: true disablePointer: true iScroll.on 'scroll', -> float() # your code

This more or less guarantees deterministic behavior. (Some additional tweaks were added by me in order to make it behave with jQuery UI Sortable.)

You can read more about the probeType option here:

This regulates the probe aggressiveness or the frequency at which the scroll event is fired. Valid values are: 1, 2, 3. The higher the number the more aggressive the probe. The more aggressive the probe the higher the impact on the CPU. probeType: 1 has no impact on performance. The scroll event is fired only when the scroller is not busy doing its stuff. probeType: 2 always executes the scroll event except during momentum and bounce. This resembles the native onScroll event. probeType: 3 emits the scroll event with a to-the-pixel precision. Note that the scrolling is forced to requestAnimationFrame (ie: useTransition:false).

This basically means that probeType 3 is precisely what we want.

Nevertheless, there is one hidden problem you may then discover if you do a lot of testing: iScroll may not resize itself properly when its content size changes (#596, #642). After learning more about flow events (a, b), I decided to not do it that way. Instead I went with Resize Sensor which is part of CSS Element Queries.

You can then attach a Resize Sensor to the wrapper, and a Resize Sensor to the content, and call iScroll’s refresh() method accordingly.

AngularJS + iScroll

This article will not be complete without a functional AngularJS directive and sample code so here it is.

You may find it annoying that iScroll only translates the first element in its wrapper. Certain directives out in the wild (a, b) have required use of nested elements, but I think that is not elegant enough.

By utilizing AngularJS Transclusion (teropa), you can generate the wrapper for your child elements automatically. For example, the following markup triggers an esHeaderFloater directive:

%div{ 'es-header-floater' => '.panel-heading' } %div{ 'ng-if' => 'currentStudent' } = render 'shared/portfolio_content' …

If you use transclusion properly like this…

es_app.directive 'esHeaderFloater', ['$rootScope', ($rootScope) -> template: '<div ng-transclude></div>', transclude: true, link: ($scope, $element, attrs) -> $scope.iScroll = new IScroll($element[0], …) ]

…then at execution time, the content wrapper will be generated automatically for you, without you having to litter wrapper elements everywhere in your codebase. This is also good for future-proofing when at a certain time in the future, you can eliminate use of iScroll as a clutch without having to modify all calling sites again.

This may not be the fastest solution, but I am confident that it is quite understandable.

Given the following table definition:

Table "public.subscription_prices" Column | Type | Modifiers ------------+-----------------------------+------------------------------------- id | uuid | not null default uuid_generate_v4() status | integer | not null default 0 usd_cents | integer | not null default 0 validity | tsrange | not null created_at | timestamp without time zone | not null updated_at | timestamp without time zone | not null Indexes: "subscription_prices_pkey" PRIMARY KEY, btree (id) "subscription_prices_no_overlap" EXCLUDE USING gist (validity WITH &&) Check constraints: "subscription_prices_boundedness" CHECK (lower(validity) IS NOT NULL AND upper(validity) IS NOT NULL AND lower(validity) >= '-infinity'::timestamp without time zone AND upper(validity) <= 'infinity'::timestamp without time zone AND lower_inc(validity) AND NOT upper_inc(validity)) "subscription_prices_greater_or_equal_to_zero_cents" CHECK (usd_cents >= 0)

The following query identifies gaps between each definition.

with lut as ( select current.validity, (before.id is null) as missing_before, (after.id is null) as missing_after from subscription_prices current left join subscription_prices before on lower(current.validity) = upper(before.validity) left join subscription_prices after on upper(current.validity) = lower(after.validity) where before.id is null or after.id is null ), missing as ( select 'after' as missing, '-infinity' as timestamp union all select 'before' as missing, lower(validity) as timestamp from lut where missing_before is true union all select 'after' as missing, upper(validity) as timestamp from lut where missing_after is true union all select 'before' as missing, 'infinity' as timestamp ), ranked as ( select *, rank() over (order by timestamp asc) from missing order by timestamp asc ), missing_ranges as ( select tsrange(lhs.timestamp, rhs.timestamp, '[)') as missing_range, lhs.timestamp as from, rhs.timestamp as to from ranked lhs left join ranked rhs on rhs.missing = 'before' and rhs.rank = lhs.rank + 1 where lhs.missing = 'after' and rhs.timestamp is not null ) select * from missing_ranges;

With the number of rows low, this query finishes quickly, but once you hit a higher number of rows, you might want to further optimize.

Visit a country that Twilio carries SMS-capable numbers for.

Provision a Twilio Number with SMS capability.

Configure Geographic Permissions for your account, so you can text any country. (It appears that in some countries, Apple’s SMS gateway appears to be using an ‘out-of-network’ number.) For more information, see International Permissions.

Install the Twilio Ruby Gem.

Send the reservation request (SMS) with irb using Twilio::REST::Client.new.messages.create.

Check inbound reservation confirmations on Twilio’s Web UI.

The Use Case is Complete. Cost: ~$10.

Notes

Twilio Numbers are billed monthly with no prorating on early cancellation.

OS X El Capitan broke Pow and the following steps bring it back, so you can be productive again.

Update: https://github.com/basecamp/pow/pull/510/files fixes the issue for El Capitan (10.11) Public Beta 1.

Steps to Fix

Ensure that Pow 0.5.0 is installed. The instructions may be out-of-date if you attempt to do this against a different Pow version.

Terminate Pow. If you have powder, run powder down. Otherwise, use launchctl to shut it down:

$ launchctl unload ~/Library/LaunchAgents/cx.pow.powd.plist $ sudo launchctl unload ~/Library/LaunchDaemons/cx.pow.firewall.plist

Patch Pow. If you installed Pow via curl | sh the default installation will be inside ~/Library/Application Support/Pow/Current. Find dns_server.js and change the following line:

res.addRR(q.name, NS_T_A, NS_C_IN, 600, “127.0.0.1”);

to:

res.addRR(q.name, NS_T_A, NS_C_IN, 600, “127.127.127.127”);

Restart Pow. Run powder up if you have Powder, otherwise follow the community’s advice or use launchctl:

$ launchctl load ~/Library/LaunchAgents/cx.pow.powd.plist $ sudo launchctl load ~/Library/LaunchDaemons/cx.pow.firewall.plist

Verify ports that Pow is serving from. Usually Pow listens on a TCP port for HTTP traffic and an UDP port for DNS traffic. In certain cases, the port information written to /etc/resolver/dev during installation can be incorrect.

Run the following command to see which PID Pow is running under:

$ ps aux | grep pow evadne 7976 0.0 0.2 3102620 27632 ?? S 12:30AM 0:00.28 pow

Afterwards, run the following command to know where Pow is listening from:

$ lsof -i | grep node node 6243 evadne 12u IPv4 0x89a98d03be85df97 0t0 TCP *:20559 (LISTEN) node 6243 evadne 13u IPv4 0x89a98d03b6501eef 0t0 UDP *:20560

In the case above, Pow is running with PID 7976, listening on port 20559 for TCP traffic (HTTP), and on port 20560 for UDP traffic (DNS).

Verify resolver configuration.

Within /etc/resolver/dev you should see the following content:

$ cat /etc/resolver/dev # Lovingly generated by Pow nameserver 127.0.0.1 port 20560

Pay attention to the port used here as there were reported cases where this port number was off by one. If the port number is incorrect, correct it as root and save.

Refresh and verify DNS configuration.

In certain cases updating /etc/resolver/dev does not immediately take effect. You should consider running the following command to flush the DNS cache using dscacheutil.

$ dscacheutil -flushcache

After flushing DNS cache all name resolution should now operate normally, and things like curl foo.dev should start to work. To verify, you can run the following command:

$ scutil —dns

This prints out all DNS resolvers that the OS will use. If, however, the port number under the dev resolver is still incorrect after flushing DNS cache, you can consider flipping the DNS config for your active network interface temporarily to some other value then flipping it back.

If the hack above still does not work after the fifth try, you may want to consider pursuing an alternate career as programmer oracle.

Use Pow

After taking the steps above you should be able to try…

$ curl foobar.dev

…and get a Error page, unless you have configured ~/.pow/foobar. This means that you have fully restored Pow 0.5.0 to full functionality.

Reference

“10.11 El Capitan Beta 1 Upgrade Breaks Pow”

I have seen more than countless data models that were built in such a disrespectful manner — they do not respect the passage of time and are trapped in their own linear existence. At the end of the day, if your data model has not been carefully built, you will be unable to track new changes over time, and expect full resolution and granularity, if the metrics were not introduced on day one.

For example, if you’re tracking conversion in your CRM and forgot that customers may churn and come back at a later time, your data model will only reflect the present, you will have another data store littered with snapshots, and you will be unable to really know when the events occurred at a level more granular than the granularity of your snapshots.

The only true solution that I know of is to record raw events at their natural granularity and ensure that you can transform the entire dataset into whatever view you may fancy at any point in time, even well after both the original events and those that conflict with them have occurred. (In other words, record the segue itself, not the ingress and egress.) This approach opens up various possibilities — it lets you fork the universe and backtest everything in your ‘what if’ scenarios.

You should really think of this as a more generic way to A/B test your data handling strategies… your world view… even your approach to life and work, in a totally safe manner with very little risk. On a more concrete level, the ‘what if’ scenarios include but are not limited to:

If we were to price our products differently in 6 months, how will our ARR look?

If I were to import this huge file, what’s going to happen to my students in the system?

I’d like to make a global change, but what will happen if I click ‘Save’? The representatives assured me that this is reversible…

At the end of the day, 1) seeing is believing, and 2) the ability to see how the possibilities play out naturally affords possibility to make a conscious choice to proceed or to bail, hence 3) ability to respect time and change on an engineering and intuitive level (because this discipline must be internalized by every single engineer building the solution) allows customers total control over their data, potentially hosted by a foreign entity.

There are interesting extrapolations as well:

If you’ve built your applications in the correct way, and if you’re building your data storage layer the correct way, you will be able to fork a formation of data storage services as easily as you spawn additional logic-area-specific containers, run your tentative transformation in the new formation, and redirect the customer to the new, tentative dataset, which they can then explore using all functionalities you already provide in your application suite.

You can actually turn point-in-time recovery from something that is only used sparingly by operations folks into something that works harmoniously within your application suite as a feature, which translates to reduced support costs and turnaround time, because everything is forkable. More importantly, if you build the entire system and the accompanying recovery process in a logical and humane manner, you will be able to outperform all major players in your space by a magnitude of 10 or more: “Other folks charge $10k and make you wait a week, but with us you can do this yourself in five minutes”.

You will be immune to your past mistakes building certain components that analyze information and store their conclusions for presentation.

I personally believe that most software engineering (or even product engineering) can either be done with a very small group of highly proficient individuals or a larger team composed of redundant workers who have undergone an unified training plan.

Specialist teams specializing in rapidly bootstrapping a new product and reducing uncertainty (therefore reducing value-at-risk) may not be suitable for maintaining business-as-usual applications that must now work harmoniously with reality. Conversely, it will be inappropriate to task teams built in an uniform manner with massive creation of new value thru introduction (or delivery) of new technology at an appropriate time and location.

The risk profile for any software project also changes over time as more code is written and the chasm between specification and deliverable is gradually bridged or exposed. On day one, it may be equally possible that you’ll either be doomed or reveled, but on day ∞ you can probably pass the Most Doomed Individual label around freely.

The easiest way to gauge the type of your team is by measuring the median pain somebody may bring if they were to quit unexpectedly on the next day. If you have a well-knit team comprised of highly efficient individuals, you must tolerate occasional volatility.

It seems that the best solution for all is to build both kinds of teams and allow open-allocation. There are two other options in the matrix — a large team of highly-proficient and effective individuals, and a small team of duds. While the latter is certainly unpalatable, the former is possible, but I also believe that there is a natural limit on the size of the team, before communication grinds down to a halt and turns professionals into unwilling slackers.

There is a built-in way to create installation media from the over-the-air installers which still works today. No more fiddling with InstallESD.dmg.

$ sudo Installer.app/Contents/Resources/createinstallmedia

Usage

createinstallmedia --volume <path to volume to convert> --applicationpath <path to Install OS X Mavericks.app> [--force]

Arguments

--volume, A path to a volume that can be unmounted and erased to create the install media.

--applicationpath, A path to copy of the OS installer application to create the bootable media from.

--nointeraction, Erase the disk pointed to by volume without prompting for confirmation.

Example

There are a few human interface tips for creating successful value confirmation components:

The error must be highly visible. Applying red outlines on invalid fields is a good idea.

Validation must occur in real-time as the human attmepts to correct the discrepancy between the original value used for commiting data, and the replica value used for validation. Validation must occur as the human types and re-types in either field, presenting or dismissing visible error markers.

The following snippet implements password validation with a popular JavaScript component h5Validate with the following assumptions:

The app uses $ for jQuery.

Whenever two password fields are contained in .set-user-password, a validation link is anticipated.

One thing to note is the escapeRegex function appropriated from jQuery UI. This ensures that special characters are matched properly, avoiding situations like when a . placed in the source string allowed any character to be matched (which defeats the purpose of validation).

On the day I gave a three months’ notice, the CTO shared his art of quitting a software engineering job:

Individual contributors should give a two weeks’ notice.

Find your replacements and train them well.

Write documentation in the successor’s working language (especially important if it differs from your working language).

Ensure that your colleagues are able to see the things you see.

Quit on extremely good terms, so you may be invited back for a talk, or consulted about affairs irrelevant to your previous work product.

Be honest about the technical debt you know about and share ways to clean it up.

Once upon a time, the man quit the job in the morning, went to an interview in the noon, discovered more bozos than estimated, and re-assumed his old position in the evening. I found his recount otherworldly and was not quite sure how this would work for any person less stellar.

Presented without critique.

Three Felonies A Day: How the Feds Target the Innocent

The Silicon Valley of Dreams: Environmental Injustice, Immigrant Workers, and the High-Tech Global Economy

Nickel and Dimed: On (Not) Getting By in America

The Art of Capacity Planning: Scaling Web Resources

PostgreSQL 9.0 High Performance

Unwritten Laws of Engineering

Making it Big in Software: Get the Job. Work the Org. Become Great.

Marketing High Technology