Smack

Appa

April 9, 2021 We had been expecting this day for a while. Yet, when it finally came, it all felt so hurried. As the long night drew closer, I was growing worried, that after this last mile, I would never remember your smile. That my feelings will forever lay burried beneath our strained relationship, muddied. Will I only remember you as a cynic? Will I still be your harshest critic? Will I only lean on you, for what not to do? Will you fade away misunderstood? Like I was to you, no good.

I couldn't trust my memories to be a true reflection of the man he was. In the ~months~ years leading up to his death, he had become increasingly reclusive and distant from the rest of us -- or, at least from me. There were increasingly more negative interactions, both first and second-handed, than positive ones. And, even during the last month, when I got to spend more one on one time with him in Houston, there were more confrontations and disagreements than consensus and trust.

I had referred to him as "The Petulant Parent", "Imsai Arasan 24th Pulikesi", etc. albeit jokingly. I even got so frustrated at one point that I had settled on "In Service of Assholes" as the title for my non-existent autobiography. (Please save the Sir Mixalot jokes 😎)

Near the end, we had gotten into many arguments about our politics, and our ethics. We were almost polar opposites in how we viewed the world, and our headbuttings only became more frequent as we both knew we were running short on time to convince each other of our ways. I was worried I won't be left with any good memories to carry forward in life.

But all those worries faded away after he passed away.

I realized that though we were different, there was so much that we had in common -- From our unhealthy disregard for authority, to our unbounded curiosity and urge to understand how the world really works beneath the way it presents itself. We are both humble in certain aspects of our lives, and elitist smarter/holier-than-thou dickheads in others, though, there isn't much overlap between us in the things we were humble about and not.

And, where he was being difficult towards me, he pushed me to take a step back and gain a deeper understanding about what I'm doing wrong, or why I'm certain about being right.

That is the part I miss the most -- having to live up to the unattainable demands of a petulant parent. Watching him hold on to his convictions and view points, no matter how wrong they were, or seemed to be, had forced me to be aware of my own mistaken views of the world And if I have to be honest, that is the part that worries me the most -- that without his push, my life will become stagnant.

My grandfather taught me how to lead life with purpose, and to find purpose in service to others. But my dad imparted in me the importance of acting and doing, even when there isn't a clear purpose.

I hope I am lucky enough to always find purposeful avenues to express myself. But when I don't, I hope that I can follow my father's spirit and push through blindly.

All along until now, I was placing the utmost importance on the legacy I will be leaving behind. On how I will be remembered. How I could become immortal.

But his passing's biggest lesson has been this: Our legacies are the concerns of those who come after us, not ours. We may be wrong, but we should stake our position and act. Our time here is short. We're mortal.

I'm mortal.

Happy Father's Day!

Mortality vs. Case Fatality

First, we need to understand the metric we are discussing here. The Center for Disease Control (CDC) as well as the World Health Organization (WHO) both define mortality rate as the number of people dying from an infection divided by the total number of people infected.

Mortality Rate = Number of Deaths / Number of Infected.

For COVID-19, or more precisely, for SARS-nCOV2, the WHO estimates mortality rate for the virus to be around 3.4% (and climbing). The CDC estimates it to be around 1 - 2%. And everyone in the media have been comparing it with Influenza, which has a mortality rate of about 0.1%, and claiming that this Coronavirus infection is 10 - 15x as deadly.

For a pessimistic outlook, talk to any statistician (or me) and they will point out that the numerator isn’t fully determined as you are dividing deaths from just closed cases over the total number of cases including both active and closed cases. And just based on that, assuming active cases close with the same percentage of fatalities, you would end up with double the numerator and around a 7% mortality rate by WHO’s measure. Of course, that is assuming that new cases are growing linearly. But we know, we are on an exponential curve up... so, this metric will in fact land somewhere higher.

For instance, consider a hypothetical scenario where an epidemic lasts 20 weeks, and the average duration from infection to recovery or death is around 4 weeks. Assuming a 50% mortality rate, the weekly metrics for deaths and new cases would look something like the following:

In the graph above, red represents the count of new infections, and blue represents the number of deaths per week. And, the cumulative graph would look like this:

If you calculate the mortality rate the way the WHO and CDC calculate today, then up until week 4, it would be 0%, and it will always be less than the true mortality rate of 50% slowly inching it’s way up to the true value at the end of the epidemic. This is why the WHO’s 3.4% and CDC’s 1-2% estimates are artificially optimistic. Sure, they sound nicer to our ears, but it’s only gonna get worse. Unless of course, we can find a vaccine or a cure before the bell curve peaks, which is highly doubtful.

For an optimistic outlook, talk to any doctor, and they will correctly point out that these estimates are meaningless as we don’t know how big the denominator is, because we aren’t testing everyone for the virus, and only test those with severe symptoms. So far, South Korea and the Diamond Princess Cruise are the only places where they tested people regardless of whether or not they showed symptoms, and thus those populations provide a more accurate estimate of what the mortality rate will be for the virus. And in those populations, the mortality rate was shown to be around 0.9%.

Those populations also showed that 82% of the infected either were asymptomatic (i.e, showed no symptoms) or were mildly symptomatic (i.e, a cold, or very mild fever, or a sore throat) and the symptoms went away.

This is in stark contrast to Influenza, where if you contract the virus, then you end up being symptomatic. And, this makes comparing mortality rates between the two diseases ever more meaningless. If I catch the flu, I’m gonna need hospitalization, and the 0.1% probability of dying from it applies to me. If I catch nCoV2, neither the 3.4% nor the 0.9% applies to me. If I’m over 80, then 21.9% applies to me. If I’m young and asymptomatic, almost 0% applies to me. But if I’m symptomatic enough to be admitted to the hospital, then my prognosis depends on a much smaller denominator containing just the population of those infected and had to be hospitalized. And this is where the Case Fatality Rate/Ratio (CFR) applies to me.

Case Fatality Rate

CFR = Number of Deaths / Number of Hospitalized Patients.

For Influenza, this is almost synonymous with the mortality rate. Because, if you catch the virus, then you are going to end up with the flu, and need to be hospitalized. Thus total number infected approximately equals the total number hospitalized.

But for Coronavirus, this would be vastly different from mortality rate as the number that need hospitalization is only around 17% of those infected. And since we can assume that if you are severely symptomatic, then you would end up hospitalized (similar to the flu), you can get at this number by looking at the closed cases thus far. And the numbers thus far paint a much direr picture than the 10-15x comparisons around the mortality rates.

While the flu has a CFR of around 0.1% (the same as its mortality rate), COVID-19′s CFR is around 12% and growing. (When I started writing this post, it was around 11%, and by the time I’m done writing it, it’s at 14%... Go figure! I think it’s because of the systemic failure with the Italian Healthcare System). This is about 100x that for Influenza. While this is alarming, we need to remember that while the flu affects everyone almost uniformly regardless of age, COVID-19 is especially lethal to the elderly. And a majority of those that require hospitalization are elderly and thus the numbers will get skewed towards the 21.9% mortality rate for above 80-somethings than the 0.2% mortality rate for the below 20s. And the elderly are already filling up our hospital beds, where they are most likely to contract it as well as get tested for it. So, there is definitely a bias in the overall CFR towards that of the elderly.

If you’re like me, you might think “I’m close to 40, how does this CFR affect me?” It is still worth noting that the virus is much more deadlier than the flu, and affects older population more severely because of co-morbid conditions. But it doesn’t affect old people exclusively. It also affects those with heart and pulmonary diseases, those with cancer, as well as those with diabetes more adversely than others. A lot of these conditions arise with old age. But, they also arise with obesity. And in the US, where a majority of the population is now overweight or obese, we don’t know how these conditions will affect the case outcomes for this disease. Of course, it also affects fit young people as well (I know of one 33 year old personally that had to be hospitalized for a week).

So, what do we do now?

Take this sh** seriously. Avoid contracting it as much as possible. Don’t take social distancing lightly, and be extra careful about visiting your parents / grandparents. Facetime/Zoom/Duo/WhatsApp your family. Buy them groceries and leave it at their doorstep to show that you care. But, please don’t hug and KOD your parents.

Most importantly, if you have a fever, or sore throat, DO NOT rush to the ER!!! At least, not until your symptoms turn out to be severe and you really need in-patient care, because:

1. If you have COVID-19, you are most likely to spread it to others in the ER unnecessarily, when you are just mildly symptomatic and don’t need to be hospitalized.

2. If your symptoms are not due to COVID, you are more likely to contract the virus in the ER.

You are most likely better off waiting for your symptoms to wean off while quarantining yourself at home. Also, you can’t create videos like this at the hospital... So, stay home! Stay safe! Stay creative! :)

View this post on Instagram

Interfaces & Dependency Injection:

The four basic interfaces I came up with are:

IBuilding

IElevatorShaft

IElevator

IElevatorCallController

To make it testable, and easy to come up with various configurations, I also had abstract factories for each of those defined.

IBuildingFactory

createBuilding(options: IBuildingOptions, eccf: IElevatorCallControllerFactory, esf: IElevatorShaftFactory, ef: IElevatorFactory)

IBuildingOption {numFloors: number, numShafts: number)

IElevatorShaftFactory

createElevatorShaft(options: IElevatorShaftOptions, ef: IElevatorFactory)

IElevatorFactory

createElevator(options: IElevatorOptions)

IElevatorCallControllerFactory

createElevatorCallController(shafts: IElevatorShaft[])

You can see how dependency injection is used here to provide all of the dependencies at each level. There is a bit of balance to be struck here in terms of code readability and testability. As you can tell from above, the ElevatorFactory needs to be passed in through all the layers (Building -> ElevatorShaft -> Elevator), and this could get out of hand. For a good (i.e, bad) example, look at any Guice based project's codebase, and you will come across plenty of nested factories. DI Frameworks such as Spring (and Guice as well) help you avoid this issue and preserve readability in your code through annotations and autmagically injecting the dependencies through reflection. They acheive this primarily by employing Factory Provider and Service Locator patterns, but that's for a future post.

Let's take a look at what actually is handled by each interface...

Separation of Concerns:

IElevator

IElevator is only concerned with the properties and actions of a single elevator. What might these involve?

requestFloor(floor: number) - Pressing a floor

openDoor(): void - Opening the elevator door. Mind you, this does not include the shaft door.

closeDoor(): void - Closing the elevator door.

addPerson(): void - A person entered the elevator.

removePerson(): void - A person exited the elevator.

Additional elevator attributes such as rider capacity, current rider count, etc.

NOTE: This is just a basic interface. If we wanted, we could also specify which person entered, and who exited for finer tracking abilities. But for the sake of simplicity, I'm leaving those details out.

IElevatorShaft

IElevatorShaft is only concerned with the properties and actions of a single elevator shaft.

moveElevator(floor: number) - move elevator to floor X.

openDoor(floor: number) - open the shaft door at floor X (and ideally instruct the elevator to open its door as well)

closeDoor(floor: number)

additional shaft attributes such as range of floors serviced by this shaft, elevator’s current direction of travel within the shaft, etc.

getElevator() - of course, the elevator in the shaft.

Let’s pause here to notice that the actual motor drive system for both the elevator cable as well as the doors themselves is abstracted away here. Those are additional details that can be handled by the implementations of IElevator and IElevatorShaft.

IElevatorCallController

IElevatorCallController is only concerned with figuring out some way of scheduling an elevator to handle a call at any given floor.

scheduleElevatorForCall(floor: number, direction: UP|DOWN)

IBuilding

IBuilding is only concerned with the properties and actions of the building. To keep it simple, for now let’s just say this is the only action provided by the interface.

callElevator(floor: number, direction: UP|DOWN): void

getShafts(): IElevatorShaft[]

Communicating through Delegates

So far, we have isolated the basic actions and properties within each interface. However, these concepts don't exist in the void by themselves, and have a need to interact with each other. For instance, when an ElevatorShaft instructs an Elevator to open it's door, it may need to notify back to the shaft after the action was performed. Or, an Elevator might have to notify to its ElevatorShaft that its running at full capacity.

Or, if we want to dive deeper, when a rider presses a floor, the Elevator might have a need to check whether that floor is serviced by its particular shaft before it lights up the button. We see that an ElevatorShaft contains an Elevator, so that communication is take care of. And for the callbacks, we can have the Elevator contain a ElevatorShaft reference as well, and we are done, right? What's the big deal?

Nope! If we did that, 1. we would introduce a circular dependency and tightly couple the interfaces. 2. ElevatorShaft composes an Elevator, thus needs to know about IElevator. However, Elevator need not know about IElevatorShaft.

We still want the callbacks to happen but have the objects be loosely coupled. One way to achieve the necessary communication and still maintain decoupling between the interfaces is through the Delegate Pattern. i.e, IElevator would communicate with an IElevatorDelegate for its back and forth communication needs. In the concrete implementation, the same object (the concrete ElevatorShaft instance) will satisfy the needs of both the IElevatorShaft and IElevatorDelegate, but the interfaces are not coupled. This might not seem that important, but it helps a lot with testability where we can pass different fake and mock those methods independently.

To make it more clear, let's go one level deeper and look at IElevator and its buttons IButton. IElevator needs to know about the IButton interface since it composes of a bunch on buttons. But IButton need not be coupled with IElevator. Instead, it can just rely on implementers of IButtonDelegate. That way, the same button can be used in an Elevator, or in a Keyboard, or on a Wall, etc.

interface IElevatorDelegate { // Basic handleFloorKeyPress(floor: number, sender: IElevator): boolean; // return true if allowed, false otherwise. handleDoorOpened(sender: IElevator): void; // Door opened handleDoorClosed(sender: IElevator): void; // Door closed // Advanced handleCallKeyPress(sender: IElevator): boolean; handleCancelKeyPress(sender: IElevator): boolean; handleDoorOpenKeyPress(sender: IElevator): boolean; handleDoorCloseKeyPress(sender: IElevator): boolean; }

Putting it all together

I'm not gonna cover implementing concrete objects for all interfaces in this post. But with the above interfaces, let’s consider how two buildings might be constructed.

Building A - A wooden 4-story building with a single elevator made by Schindler Lifts.

Building B - A concrete 50-story building with 4 elevators made by Otis Lifts.

class Building implements IBuilding { ecc: IElevatorCallController; shafts: IElevatorShafts[]; numFloors: number; constructor(numFloors: number, ecc: IElevatorCallController, shafts: IElevatorShaft[]) { this.numFloors = numFloors; this.ecc = ecc; this.shafts = shafts; } callElevator(floor: number, direction: UP|DOWN): void { this.callController.scheduleElevatorForVCall(floor, direction); } } class WoodenBuilding extends Building implements IElevatorShaftDelegate{ // ... all wood building specific props constructor(shafts: IElevatorShaft[]) { for ( var shaft: IElevatorShaft in shafts ) { shaft.setDelegate(this); // Set self as delegate } } } class ConcreteBuilding extends Building { // ... all concrete building specific stuff } class WoodBuildingFactory implements IBuildingFactory{ createBuilding(options: IBuildingOptions, eccf: IElevatorCallControllerFactory, esf: IElevatorShaftFactory, ef: IElevatorFactory) { let ecc = null; let shafts = []; if (options.numShafts > 0) { for ( var i = 0; i < numShafts; i++) { let es = esf.createElevatorShaft(ef); shafts.push(es); } if (eccf != null) { ecc = eccf.createElevatorCallController(shafts); } } return new WoodBuilding(shafts); } } class ConcreteBuildingFactory implements IBuildingFactory{ // ... similar stuff goes here. } // A generic dumb call controller that round robins through all available shafts. class GenericCallController implements IElevatorCallController { shafts: IElevatorShaft[]; currShaft: number; constructor(shafts: IElevatorShaft[]) { this.shafts = shafts; } scheduleElevatorForCall(floor: number, direction: UP|DOWN) { shafts[currShaft]. currShaft = currShaft++ % len(shafts); } } // A more advanced call controller that might use specialized algorithms to maximize people flow. class OtisMultiShaftCallController implements IElevatorCallController { //... } class OtisElevatorShaft implements IElevatorShaft, IElevatorDelegate { //... } //... //... //... and so on.

and to instantiate the buildings, you would simply code something along the lines of...

func main() { //... //... let buildingA = WoodenBuildingFactory.createBuilding( {numFloors: 4, numShafts: 1}, SchindlerCallControllerFactory.getInstance(), SchindlerElevatorShaftFactory.getInstance(), SchindlerElevatorFactory.getInstance()); let buildingB = ConcreteBuildingFactory.createBuilding( {numFloors: 50, numShafts: 4}, OtisCallControllerFactory.getInstance(), OtisElevatorShaftFactory.getInstance(), OtisElevatorFactory.getInstance()); //... //... }

A few things to note above.

Even though I haven't included concrete implementations of the classes above, you can see from whatever little bit of code I have included, that we are primarily dealing with interfaces. This is very important in terms of testability, as now we can easily swap in mock and fake implementations while testing, thus making writing unit tests a breeze. It also makes the code very reusable and extensible.

I am callig singleton instances of the factories in the main function. Ideally, these would be provided by a DI framework such as Guice or Dagger or Spring through the use of simple annotations.

You start at an initial model state m0. Client A performs an Operation A and arrives at model state mA. Client B simultaneously performs Operation B to arrive at model state mB. Now, given the pair of operations A and B, we need to find complimentary operations A’ and B’ such that mB + A’, and mA + B’ both converge back to a single model that both clients can sync back to. For example, Client A’s operation of inserting ‘!’ at the 5th index-0 position get’s transformed to inserting ‘!’ at the 4th index-0 position, after B’s operation of deleting the first character is performed. And B’s operation undergoes no transformation because A’s operation doesn’t affect it’s index.

The important thing to note here is that regardless of what the model was, those two operations will lead to the same exact transformed operation. For instance, if the initial model state was “namaste”, and we performed the same operations as above. We would arrive at the same synced state of “amas!te”.

If we can define transformations A’ and B’ for every possible pair of operations A and B, then we can achieve our goal of syncing changes without syncing documents.

Of course, figuring out the useful transformations for all the different combinations of A and B could get quite complicated, but once they’ve been figured out, OT will guarantee that the clients will arrive at a synced state.

I emphasize “useful” because one could easily define A’ and B’ to simply delete the entire model for any and all combinations of A and B. In that case, your clients would technically arrive at a synced state and satisfy OT’s requirements. But it would be an empty document, and that does nobody any good :/

So, let’s take our initial example of taking the string “hello!”, and the two clients A and B, with A replacing ‘h’ with ‘H’, and B inserting “ world” before the exclamation mark. These state changes and all other text manipulation can be achieved via a minimal set of 2 operations:

1. deleteCharacter(atPosition p) aka DELETE (index) 2. insertCharacter(character c, atPosition p) aka INSERT (char, index)

This implies there are 4 possible pairs of inputs for our OT diamond.

DELETE / DELETE

DELETE / INSERT

INSERT / DELETE

INSERT / INSERT

And we could define our transformation logic as follows:

Operation A Operation B Transform A' Transform B' DELETE (indexA) DELETE (indexB) (indexB == indexA) ? NO_OP : (indexB < indexA) ? DELETE(indexA - 1) : DELETE(indexA) (indexB == indexA) ? NO_OP : (indexA < indexB) ? DELETE(indexB - 1) : DELETE(indexB) INSERT (charA, indexA) DELETE (indexB) (index B < indexA) ? INSERT(charA, indexA - 1) : INSERT(charA, indexA) (index A <= indexB) ? DELETE(indexB + 1) : DELETE(indexB)</td> DELETE (indexA) INSERT (charB, indexB) (index B <= indexA) ? DELETE(indexA + 1) : DELETE(indexA)</td> (index A < indexB) ? INSERT(charB, indexB - 1) : INSERT(charB, indexB) INSERT (charA, indexA) INSERT (charB, indexB) (index A > indexB) ? INSERT(charA, indexA + 1) : INSERT(charA, indexA) (index B >= indexA) ? INSERT(charB, indexB + 1) : INSERT(charB, indexB)

There are a couple of interesting things to notice with the transforms we have defined.

First, we defined a new NO_OP operation to return for the case where both clients perform the same delete operation. This is to avoid double-deleting the wrong character.

And second, the >= in the last pair prioritize A's inserts over B's insert if they both insert characters at the same index. So if A inserted " Bob" and B inserted " World", the output will always result in " Bob World".

Note: If both added " World", we would end up with " World World", but in a real-time collaborative environment, this is acceptable. If we really care about avoiding duplicates, we could define higher level operations that look at word inserts and compare whole words/phrases for duplicates.

This transformation model works well for single character operation. But how do we handle a string of operations by each client? For instance, replacing 'h' with 'H' itself takes a minimum of two operations (a DELETE followed by an INSERT) and inserting " world" requires 6 INSERT operations.

Handling Multiple Operations

Once you’ve figured out the transformation logic for a single pair of operations, handling multiple operations becomes a no-brainer. It’s simply the same OT diamond cascading transformed operations as input to the next OT diamond.

Take the following example where A diverges by 3 operations, while B diverges by 2. To compute the transformed operations A1′, A2′, and A3′ to apply on client B, as well as B1′ and B2′ to apply on client A, you simply have to cascade your transformations down. Note that there are intermediate states and intermediate operations shown in the diagram below, but there is no need to store or even care about the intermediate models as OT works independent of model state as shown above with that really lame “amas!te” example :)

In the above case, we had to perform transformations on 6 different OT diamonds. Basically, OT becomes a O(m*n) operation, where m & n are the number of operations the clients have diverged by. So, it is preferable to sync frequently keeping the count of operations to perform OT on pretty low.

Luckily, this suits really well for real-time collaborative use cases where we are syncing almost continuously after each and every edit.

Handling Multiple Clients

Syncing multiple clients is just as easy. We simply sync them sequentially and additively. Assume that you have 3 clients A, B, and C that have each performed 3, 2, and 2 operations respectively. To bring them all in sync, we first sync A (operations A1, A2, and A3) and B (B1 and B2) to determine A’ (A1′, A2′, A3′) and B’ (B1′, B2′). Now both A + B’ and B + A’ lead to the same synced state mAB for both clients.

We can now take one of these equivalent paths (either A1, A2, A3, B1′, B2′ or B1, B2, A1′, A2′, A3′), and sync that with C to determine C1′ and C2′, as well as the transformed operations to apply on client C. Let’s take the first path A1, A2, A3, B1′, B2′ for illustrating how this works. The sync would work as shown in the following diagram.

It’s not shown clearly in the diagram, but C1′ and C2′ are operations that get applied on both A and B, which are currently both at state mAB, to bring all three clients to the same synced state mABC. And we can follow this same technique and extend it to as many clients as we need to sync.

I strongly believe that creativity gets better through collaboration, and the current set of available tools don’t enable frictionless collaboration. (I’m well aware of the irony intrinsic to me working solo on this project.)

I am passionate about filmmaking and storytelling, and not bought into the hype surrounding #blockchain yet :/ Definitely not bitcoin, which is literally fool’s gold.

I wasn’t happy about the work I did at Google. Don’t get me wrong! I love the company, and think it’s a great place to work. But, I felt unhappy with the scope of my work, and that feeling Steve Jobs talked about in his Stanford commencement speech kept creeping up on me.

The reason I want to create a collaborative video editor is simple: There’s increasing demand for video content on the web. And, to keep up with the demand, media sites and software companies are focused on speeding up the video creation process by sacrificing on quality and putting out junk content. There are companies like Wibbitz out there that are focusing on using machine learning to automate the creation of more junk videos.

I think there is a better way to keep up with the demand, and still maintain the quality of content being put online. I want to focus on enabling teams to streamline their workflow and unleash their creativity through collaboration. I want there to be more videos like this, and less like its subject matter :)

How?

Two words: “Operational Transforms”

Actually, it’s a lot more than that, but’s that’s the basic tech behind making the project state sync in real time. That’s how Google Docs syncs documents across user sessions. That’s how Dropbox Paper syncs its paper docs . And pretty soon, how Eclipse Che editors will sync their source code.

Words beyond “Operational Transforms” which I will get into detail in future posts:

React Components for UI.

Canvas + WebGL for rendering preview.

Ffmpeg.js for exporting.

Web Workers + Socket IO for communication.

gRPC + Spring Boot for sync backend.

Spring Boot + MySQL for backend services.

Dockers + Kubernetes for deployment (potentially on GCP or AWS).

Google Cloud Store or S3 for asset storage requirements, which is currently satisfied by my dev Macbook’s local filesystem.

What’s the status?

I spent a lot time initially to get the OT syncing right, and have a working gRPC service for syncing operations on the shared document between clients, which, as an engineer looking at it from an architecture perspective, is where the meat of the project is.

I’m currently working on building the editor front-end which, as far as the rest of the world is concerned from a value perspective, is where the meat of the project is.

And, it currently looks like this :/ #DontJudgeABookByItsCover #DontJudgeAProjectByItsLooks

Sometime near the beginning of this year, I suddenly stopped being happy about a lot of things that I used to be ecstatic about... Stuff that others would kill for! I fell in love in NYC, and with NYC. Then I wasn’t happy about either of that. I was in love with Google. Then I wasn’t happy about that either. I initially blamed oxygen deprivation from Kilimanjaro and exhaustion from running a marathon for messing up my brain. But then after a lot of shower thoughts, I realized it’s actually because of something much worse!!! It was because of muthafuckin’ Tony Robbins! And my buddy that channeled him by asking me blunt piercing questions during his visit to NYC in February. (You know who you are!!!) Things just weren’t the same after that.

I started feeling the same sense of stagnation that had originally forced me to move out of San Francisco, and made me look for a job in NYC. It was like the plot to INCEPTION. The thought was seeded in my mind that I’m not living up to my full potential, and it made me unhappy about everything that I did. There was a cloud of apathy hovering over me with whatever I did. And it affected everything! Both in life, and at work. And it didn’t go unnoticed, especially at work. Apathy + Google’s work demands just don’t mix. So, after months of feeling like a millennial craving for “impact” and feeling like I’m stagnating, and not particularly caring about things perf (Googler speak) wanted me to care about... I quit and decided to start on my passion project!

What is it? Why now? And How? I’ll get into those details in follow up posts. I have loftier goals. But for now, it has to do with online video editing and real-time collaboration, and it's gonna stretch Operational Transforms, the basic tech behind Google Docs, beyond its limits. I’ll get into the technical details of OT in a subsequent blog post as well. But, here’s a sneak peak of what it will look like visually:

I took most of December off to destress from everything, and to take care of some personal finance stuff back home in California. (I need to save in every which way I can to extend my runway for this project.)

And, in the last week of actually working full time on the project, I’ve gotten much farther that I thought I would with the technical design. (And of course, the visual design as well.) I have come up with a generic algorithm and a set of operations to perform OT efficiently on any large complex hierarchical model. (Uhhhh, say for instance... a video editing project?) And more importantly, I think I’ve found a way to sync the same document across multiple servers, which will allow for a lot more large-scale collaborative applications beyond just video editing. This one is a hard problem to solve for, especially when it comes to handling network and server failures. There’s a reason why Google limits the number of people that can simultaneously edit the same doc. I even have the front-end Javascript, and the video rendering pipeline figured out.

It’s all coming together! My confidence in my project’s feasibility is now at 100% 💯. My confidence in my execution... uhhhhh, not so much. 😕 J/k. My goal is to have a working prototype that looks like that visual design by March.

Aggressive? Yes.

Achievable? Absolutely! (I think...)

But not if I simply keep blogging about it, and don’t write any code. Peace!

A Note on Porters

In my previous super-dramatic post about not dying on a mountain, I mentioned about writing a follow-up post on the colonial era concept of porters. I did start writing about it, and the stains of a racially biased colonial past that are still evident in today’s tourism industry in Africa and the third world. But, as I was drafting, I realized that it wasn’t any different that any of us working for the man, and that it was only a matter of economics and access to education. And in time, porters as a profession will be extinct.

We, as consumers and tourists should just be cognizant of the human cost involved in the profession, and be empathetic to how much we demand of them. Sure, you are paying them to carry your shit... but don’t be a dick about it by stuffing your bag with unnecessary stuff that you don’t need. So, that’s that.

Here’s the original post (sans a chart) from last November with the title “Pause || Resume >”:

I originally wanted to write a post about how my 2016 NYC Marathon run went, and about how badly the Nike app on my new Nike+ Apple Watch failed in tracking my run. But, that happened on a Sunday, a whole 2 days before more pressing matters took over people’s attention and fears, and the whole world seemed to turn upside down for many when Trump swept in his victory on election night.

Suddenly, a lot of emotions started pouring out of those around me, and most of it negative. Fear. Anger. Despair. Sorrow. And along with emotions poured out snap judgements, and misguided inferences. “A lot of people must’ve been closet Trump supporters that voted for him, though they supported Hillary in public.” “I don’t feel safe in this country anymore!” “I can’t believe how racist this country has become to have elected Trump!” “Let’s sign this petition to ask the Electoral College to vote their conscience!”

Alright, pause right there!

TL,DR; Don’t trip chocolate chip! It’s not as bad as you think it is.

First of all, the nation is no more xenophobic/racist/ignorant than before. I’m not saying those elements don’t exist. But that’s not the reason why we lost this election to a misogynist. It’s just that not enough democrats came out to vote for Hillary this time, and DWS and the DNC are to blame for that. And partially, Bernie as well. I love Bernie and I know the scales were unfairly tilted against him during the primaries, but Hillary would have won the nomination even if the primaries were fair. And he should have acknowledged that earlier, instead of fueling the anger and rage felt by the berners that eventually turned into disillusionment and apathy.

Here’s the voter turnout for the last 3 elections. Republicans had more or less the same number of people voting their way. Democrats didn’t.

Second, not all trump supporters are racist ignorant tribal xenophobes. Okay, I take that back. They are all pretty much ignorant for thinking he actually cares about their issues, and tribal for voting just because of the (R). Regardless of whether they just feel patriotic and want to make America great “again”, or whether they are just selfish ultra capitalists that hope to benefit from Tax cuts for the rich and service cuts for the poor, a Trump presidency is not going to be beneficial for any of them. Even the racist xenophobes aren’t gonna benefit from his presidency. Mexico is not going to build that wall, and those Nazi saluters at trump rallies aren’t gonna become mainstream. Although, there is a huge chance that race relations will stop progressing.

The only people that are gonna benefit from this presidency are gonna be Trump’s cronies, and the spineless GOP politicians that stood in line to kiss King Trump’s feet... I’m staring at you Ted Cruz!

But, that’s not a bad thing at all. Sure, the nation will suffer from gross mismanagement for 4 years. But, the ignorant public will hopefully wake up and realize the big con Trump and the GOP pulled on them, and vote them out of office in the upcoming mid-term elections. And hopefully, Trump will be a single term president.

So, I had to visit the “toilet tent” a few more times than the others... So what? Big deal! Even after hitting the loo twice before 8am, I brushed it off as just a case of indigestion and went onwards with our 5-hour hike to the next camp at Moir Hut. This camp was at an elevation of around 4200m, the highest I had ever been up until that point. Once there, I was still feeling okay, and even decided to do the optional acclimation hike that evening up to a towering rock formation around 4670m (approx. 15,200ft), the highest elevation I have ever climbed to.

In hindsight, this was probably a mistake and I should’ve probably rested at camp like my campmate who was recovering from her sickness from day 1. But, I had a huge case of FOMO, and I had no idea how much worse it could get. It was daunting, but I just kept saying to myself “sip by sip, step by step!” and made sure I stayed hydrated. And it was totally worth the hurt. I got to see one of the most beautiful sunsets ever.

Sadly, it will also be the highest I will ever get to on this trip. When we returned to camp that night, we had to go through our routine checks for our pulse and blood oxygen saturation levels. Everyone else registered a healthy reading in the range of 89%-92%, slightly lower than the 95%-100% readings we got the day before. But that drop was expected due to the decrease in atmospheric oxygen at high altitudes. I registered a mere 73% – way below the 80% threshold the guides were comfortable with on the mountain.

Before I proceed any further, let me explain a little bit about blood oxygen saturation. It’s basically the percentage of red blood cells that are oxygenated in your body. Usually, 95 - 100% of your RBCs are oxygenated. And as you go higher up in elevation, the lack of atmospheric oxygen causes a lesser amount of blood passing by your lungs to get oxygenated, thus causing your saturation level to drop. Below 80%, the main organs like your brain and heart begin to get affected. And to compensate, your heart starts to beat faster to push more oxygenated blood to your organs, resulting in higher blood pressure, a higher pulse, and a throbbing headache.

Here’s a little video that explains a bit more about the effects of Hypoxia.

The video focuses on the drastic effects above 25,000ft and how it affects your mind in a matter of minutes. We obviously weren’t that high, but we had to deal with it for days, not minutes. And symptoms aren’t the same for everyone affected by hypoxia. For some, it’s diminished cognitive skills. Others start to slur while they speak. For me, it was fatigue and headache. Both of which could be attributed to just dehydration.

The only sure way to keep track of the effects of hypoxia was to measure your blood oxygen saturation. On the mountain, the guides recorded our pulse and saturation levels twice daily – once after breakfast, and once before bedtime. Your saturation level is expected to drop to 80 - 90% on the third day until your body acclimatizes, at which point it creeps back up to settle around 85 - 95%. My 73% reading that night was definitely a cause for concern.

We attributed my low reading to the stomach bug, and decided to check again the next morning, thinking may be my body had to prioritize my sickness over altitude. That night I had to use the toilet tent four additional times, and when morning came, my reading hadn’t changed by much. In fact, I woke up with an even lower reading of 69%. The highest reading I could get even with my sleeping bag acting as a temporary hyperbaric chamber was 77%. We did register a reading of 86% at some point in the morning, and the guides decided to officially record me at that. They didn’t want me to take further readings that day.

They wanted me to continue on the mountain and successfully summit just as much as I did. We decided to give it another day, and headed off to the next camp, Pofu Camp, which is at a slightly lower elevation of around 3850m. I thought I was feeling better that morning, but I couldn’t have been more wrong. This hike was brutal. I was still sick, and I had to hike 12km in 6 hours over rolling hills through a burning fever. Mind over matter right? Well, somehow I made it through that day.

That second night at altitude, my oxygen saturation was still stuck at 77%. My fever had begun to subside, and I was starting to feel a lot better. The guides though were weary with doubtful faces, but still tried to be optimistic about it. And I was too. Maybe things will get better once we get to the next camp at Third Cave, which is at around the same elevation as Pofu camp. “Don’t worry, tomorrow is gonna be an easy hike. It’s only 3 to 4km, and after about 20 minutes of uphill, it’s all downhill. We will take a reading once we get to Third Cave and then decide if you should continue or not. But I think you will be fine!”

The Northern Circuit is pretty much designed to give the body as much time as necessary to acclimatize. After the third day at 4200m, it’s basically two days of camping at lower elevations to let the body acclimatize before getting up to base camp on the 6th day around 4600m at School Hut, all ready for the summit and descent on the 7th day.

On the morning of the fifth day, my fever was gone, and I was upbeat about my health. The pulse oximeter still showed a reading of 77%. “It’s fine, we’ll check along the way to the next camp, and hopefully it will go up as the day progresses.” We checked along the way, and it didn’t waver much. It still showed 77% wherever we stopped to take water breaks. By the time we made it to camp, I was exhausted. It was only 3km, and it was all downhill, but I was done. I was ready to quit.

It was no longer a matter of mind vs. body. The night prior, I slept in the cold for a good half of it because I woke up to go take a leak, then decided not to, but then, forgot to zip up my sleeping bag. I kept waking up cold after that, and noticed every single time that my sleeping bag was open. But I kept forgetting to zip up. This happened not just once or twice, but three times! And earlier in the night, I was shivering for a bit in my fully zipped up warm sleeping bag. I was not thinking straight. I had already been feeling fatigue and constant headache for the past two days. And now, add reduced cognitive skills to my list of symptoms.

My body was telling me to GTFO, and this time, my mind was in agreement.

When we arrived at Third Cave, everyone went to take pictures with the camp signs. I went straight to a tent to rest. The lead guide came and checked my levels, and looked at me with a serious face. “You need to descend immediately! One of the guides and a porter will accompany you, but you should eat your lunch, or pack a boxed lunch, and descend right away. You can come back and climb again another day, but not today. Next year, next month, may be even next week if you want. This mountain will still be here. You’re still too young. You still have a lot of life left in you. Don’t die on this mountain today! Go!”

Words can’t express my gratitude towards Msafiri and Twalib, the guide and porter who left the mountain with me and helped carry my stuff. It took us 5 hours to descend via the scenic Rongai Route, and I already started feeling better half-way down the trail. I still had some symptoms of altitude sickness begin to show up even after I got back to the hotel that night, like a little bit of fluid in my lungs. But, they would have been worse had I decided to stay up on the mountain another night. I had had enough with altitude sickness after the throbbing headaches and shivers. I didn’t need to cough up blood to realize how bad it could get.

I have always loved nature, but now I have a new found respect for it.

I will be back to climb the mountain again, and the next time I will be better prepared for it. I will go climb other high elevation peaks to train for it, and not just rely on running with my scorpion-looking altitude training mask. I will put in the work the mountain demands, and give it the respect the mountain deserves. I only ask that the mountain returns the favor and respects my effort.

Notice that we adjusted the rotation of the hands inside the groupings to be -90º. This is to make the math simpler when we calculate rotation based on the current time, so that 0º would be 12 o’clock, and 90º 3 o’clock.

Now that we know what the SVG that we need looks like, let’s create the same using javascript & D3, and get references to the graphics context / grouping elements that we actually care about... the ones that define the rotation of each hand in degrees.

DOM

<div id="d3vis">

Script

// From Part 1 of this Blog Post var dim = 300; var svg = d3.select("#d3vis") .append("svg") .attr("width",dim) .attr("height",dim); var gHours = svg.append("g") .attr("transform", "translate(" + [dim/2, dim/2] + ")"); var gSeconds = svg.append("g") .attr("transform", "translate(" + [dim/2, dim/2] + ")"); var hourMarkers = gHours.each(function(d) { d3.select(this).call(clockFace, 12, dim/2); }).selectAll("circle"); var secondMarkers = gSeconds.each(function(d) { d3.select(this).call(clockFace, 60, dim/2); }).selectAll("circle"); secondMarkers.attr("fill","#aaa"); hourMarkers.attr("fill",function(d,i){return (i == 0)?"#666":(i % 3 == 0)?"#666":"#aaa";}); function clockFace(selection, numDots, size) { offset = 0; radius = numDots * size / (numDots + 2); deltaY = (numDots % 2 == 0)?0:(radius / 2) * (1 - Math.cos(Math.PI / numDots)); selection.selectAll("g") .data(d3.range(numDots)) .enter().append("g") .attr("transform",function(d) { var angle = d * 360 / numDots + offset; return "translate(0,"+deltaY+")rotate("+angle+")translate("+radius+")rotate("+ -angle + ")"; }) .append("circle").attr("r",0.9*(size)/(numDots)); } // All the NEW STUFF for Part 2. svg.append("defs") .append("symbol").attr("id","hand") .append("path").attr("d","M 0 10 L 20 0 L 120 10 L 20 20 z"); var secondRot = svg.append("g").attr("transform","translate(150,150)") .append("g").attr("transform","rotate(10)"); var secondHand = secondRot.append("use") .attr("xlink:href","#hand") .attr("fill","#e33") .attr( "transform", "rotate(-90),scale(1.5,0.5),translate(-20,-10)"); var minuteRot = svg.append("g").attr("transform","translate(150,150)") .append("g").attr("transform","rotate(30)"); var minuteHand = minuteRot.append("use") .attr("xlink:href","#hand") .attr("fill","#999") .attr( "transform", "rotate(-90),translate(-20,-10)"); var hourRot = svg.append("g").attr("transform","translate(150,150)") .append("g").attr("transform","rotate(40)"); var hourHand = hourRot.append("use") .attr("xlink:href","#hand") .attr("fill","#666") .attr( "transform", "rotate(-90),scale(.75,1),translate(-20,-10)");

Demo

// From Part 1 of this Blog Post var dim = 300; var svg1 = d3.select("#d3vis1") .append("svg") .attr("width",dim) .attr("height",dim); var gHours1 = svg1.append("g") .attr("transform", "translate(" + [dim/2, dim/2] + ")"); var gSeconds1 = svg1.append("g") .attr("transform", "translate(" + [dim/2, dim/2] + ")"); var hourMarkers1 = gHours1.each(function(d) { d3.select(this).call(clockFace, 12, dim/2); }).selectAll("circle"); var secondMarkers1 = gSeconds1.each(function(d) { d3.select(this).call(clockFace, 60, dim/2); }).selectAll("circle"); secondMarkers1.attr("fill","#aaa"); hourMarkers1.attr("fill",function(d,i){return (i == 0)?"#666":(i % 3 == 0)?"#666":"#aaa";}); function clockFace(selection, numDots, size) { offset = 0; radius = numDots * size / (numDots + 2); deltaY = (numDots % 2 == 0)?0:(radius / 2) * (1 - Math.cos(Math.PI / numDots)); selection.selectAll("g") .data(d3.range(numDots)) .enter().append("g") .attr("transform",function(d) { var angle = d * 360 / numDots + offset; return "translate(0,"+deltaY+")rotate("+angle+")translate("+radius+")rotate("+ -angle + ")"; }) .append("circle").attr("r",0.9*(size)/(numDots)); } // This is the new stuff for Part 2 svg1.append("defs") .append("symbol").attr("id","hand") .append("path").attr("d","M 0 10 L 20 0 L 120 10 L 20 20 z"); var secondRot1 = svg1.append("g").attr("transform","translate(150,150)") .append("g").attr("transform","rotate(10)"); var secondHand1 = secondRot1.append("use") .attr("xlink:href","#hand") .attr("fill","#e33") .attr( "transform", "rotate(-90),scale(1.5,0.5),translate(-20,-10)"); var minuteRot1 = svg1.append("g").attr("transform","translate(150,150)") .append("g").attr("transform","rotate(30)"); var minuteHand1 = minuteRot1.append("use") .attr("xlink:href","#hand") .attr("fill","#999") .attr( "transform", "rotate(-90),translate(-20,-10)"); var hourRot1 = svg1.append("g").attr("transform","translate(150,150)") .append("g").attr("transform","rotate(40)"); var hourHand1 = hourRot1.append("use") .attr("xlink:href","#hand") .attr("fill","#666") .attr( "transform", "rotate(-90),scale(.75,1),translate(-20,-10)");

Moving the Clock Hands

Okay, time to update the rotation angles based on the current time, and voila!

Script

currentTime = function() { var date = new Date(); var secondAngle = (date.getSeconds()/60) * 360; var minuteAngle = (date.getMinutes()/60) * 360 + (secondAngle/360)*6; var hourAngle = ((date.getHours() % 12)/12) * 360 + (minuteAngle/360)*30; hourRot.attr("transform","rotate("+hourAngle+")"); minuteRot.attr("transform","rotate("+minuteAngle+")"); secondRot.attr("transform","rotate("+secondAngle+")"); } resetTime = function() { hourRot.attr("transform","rotate(0)"); minuteRot.attr("transform","rotate(0)"); secondRot.attr("transform","rotate(0)"); }

Demo

var svg2 = d3.select("#d3vis2") .append("svg") .attr("width",dim) .attr("height",dim); var gHours2 = svg2.append("g") .attr("transform", "translate(" + [dim/2, dim/2] + ")"); var gSeconds2 = svg2.append("g") .attr("transform", "translate(" + [dim/2, dim/2] + ")"); var hourMarkers2 = gHours2.each(function(d) { d3.select(this).call(clockFace, 12, dim/2); }).selectAll("circle"); var secondMarkers2 = gSeconds2.each(function(d) { d3.select(this).call(clockFace, 60, dim/2); }).selectAll("circle"); secondMarkers2.attr("fill","#aaa"); hourMarkers2.attr("fill",function(d,i){return (i == 0)?"#666":(i % 3 == 0)?"#666":"#aaa";}); svg2.append("defs") .append("symbol").attr("id","hand") .append("path").attr("d","M 0 10 L 20 0 L 120 10 L 20 20 z"); var secondRot2 = svg2.append("g").attr("transform","translate(150,150)") .append("g").attr("transform","rotate(0)"); var secondHand2 = secondRot2.append("use") .attr("xlink:href","#hand") .attr("fill","#e33") .attr( "transform", "rotate(-90),scale(1.5,0.5),translate(-20,-10)"); var minuteRot2 = svg2.append("g").attr("transform","translate(150,150)") .append("g").attr("transform","rotate(0)"); var minuteHand2 = minuteRot2.append("use") .attr("xlink:href","#hand") .attr("fill","#999") .attr( "transform", "rotate(-90),translate(-20,-10)"); var hourRot2 = svg2.append("g").attr("transform","translate(150,150)") .append("g").attr("transform","rotate(0)"); var hourHand2 = hourRot2.append("use") .attr("xlink:href","#hand") .attr("fill","#666") .attr( "transform", "rotate(-90),scale(.75,1),translate(-20,-10)"); currentTime = function() { var date = new Date(); var secondAngle = (date.getSeconds()/60) * 360; var minuteAngle = (date.getMinutes()/60) * 360 + (secondAngle/360)*6; var hourAngle = ((date.getHours() % 12)/12) * 360 + (minuteAngle/360)*30; hourRot2.attr("transform","rotate("+hourAngle+")"); minuteRot2.attr("transform","rotate("+minuteAngle+")"); secondRot2.attr("transform","rotate("+secondAngle+")"); } resetTime = function() { hourRot2.attr("transform","rotate(0)"); minuteRot2.attr("transform","rotate(0)"); secondRot2.attr("transform","rotate(0)"); } That's cool, but ours is an analog clock, and analog clocks don't move like that! Let's throw in a splash of D3 transition animation. The D3.js framework includes support for smooth transitions and animations with the help of the transition() construct. Inside a transition() block, all numeric and color attributes (which end up with a numeric representation anyway) get interpolated to their new values over a short duration (I think default is 300ms, but it can easily be changed). For instance, to make all our second markers pulse, (with a staggered delay for a cool ring effect) we can define a function such as:

Script

pulse = function(){ secondMarkers .transition() .delay(function(d,i){return i*5;}) .attr("r",function() { return d3.select(this).attr("r") * 2; }) .each("end",function(){ d3.select(this).transition().attr("r", d3.select(this).attr("r")*.5); }); }

Demo

var svg3 = d3.select("#d3vis3") .append("svg") .attr("width",dim) .attr("height",dim); var gHours3 = svg3.append("g") .attr("transform", "translate(" + [dim/2, dim/2] + ")"); var gSeconds3 = svg3.append("g") .attr("transform", "translate(" + [dim/2, dim/2] + ")"); var hourMarkers3 = gHours3.each(function(d) { d3.select(this).call(clockFace, 12, dim/2); }).selectAll("circle"); var secondMarkers3 = gSeconds3.each(function(d) { d3.select(this).call(clockFace, 60, dim/2); }).selectAll("circle"); secondMarkers3.attr("fill","#aaa"); hourMarkers3.attr("fill",function(d,i){return (i == 0)?"#666":(i % 3 == 0)?"#666":"#aaa";}); svg3.append("defs") .append("symbol").attr("id","hand") .append("path").attr("d","M 0 10 L 20 0 L 120 10 L 20 20 z"); var secondRot3 = svg3.append("g").attr("transform","translate(150,150)") .append("g").attr("transform","rotate(0)"); var secondHand3 = secondRot3.append("use") .attr("xlink:href","#hand") .attr("fill","#e33") .attr( "transform", "rotate(-90),scale(1.5,0.5),translate(-20,-10)"); var minuteRot3 = svg3.append("g").attr("transform","translate(150,150)") .append("g").attr("transform","rotate(0)"); var minuteHand3 = minuteRot3.append("use") .attr("xlink:href","#hand") .attr("fill","#999") .attr( "transform", "rotate(-90),translate(-20,-10)"); var hourRot3 = svg3.append("g").attr("transform","translate(150,150)") .append("g").attr("transform","rotate(0)"); var hourHand3 = hourRot3.append("use") .attr("xlink:href","#hand") .attr("fill","#666") .attr( "transform", "rotate(-90),scale(.75,1),translate(-20,-10)"); pulse = function(){ secondMarkers3 .transition() .delay(function(d,i){return i*5;}) .attr("r",function() { return d3.select(this).attr("r") * 2; }) .each("end",function(){ d3.select(this).transition().attr("r", d3.select(this).attr("r")*.5); }); }

The code above doubles the radius attribute ("r") of each second marker inside a transition, and at the "end" of the transition, once again transitions it back to its original size.

However, the rotation of the clock hand is defined as a string inside a transform attribute, and it's not as simple to interpolate as a numeric field. Luckily, D3 provides many interpolation functions that we can employ, and one such function is the interpolateString function that finds numeric values inside a string, and interpolates them individually to new values found in the replacement string.

So, in our case, if we change transform attribute from "rotate(0)" to "rotate(30)", interpolateString function will take care of automagically tweening between 0 and 30 within the String. So, rewriting our time update function above...

Script

hourRot.transition() .attrTween("transform", function() { return d3.interpolateString(d3.select(this).attr("transform"), "rotate("+hourAngle+")"); }); minuteRot.transition() .attrTween("transform", function() { return d3.interpolateString(d3.select(this).attr("transform"), "rotate("+minuteAngle+")"); }); secondRot.transition() .attrTween("transform", function() { return d3.interpolateString(d3.select(this).attr("transform"), "rotate("+secondAngle+")"); });

Demo

var svg4 = d3.select("#d3vis4") .append("svg") .attr("width",dim) .attr("height",dim); var gHours4 = svg4.append("g") .attr("transform", "translate(" + [dim/2, dim/2] + ")"); var gSeconds4 = svg4.append("g") .attr("transform", "translate(" + [dim/2, dim/2] + ")"); var hourMarkers4 = gHours4.each(function(d) { d3.select(this).call(clockFace, 12, dim/2); }).selectAll("circle"); var secondMarkers4 = gSeconds4.each(function(d) { d3.select(this).call(clockFace, 60, dim/2); }).selectAll("circle"); secondMarkers4.attr("fill","#aaa"); hourMarkers4.attr("fill",function(d,i){return (i == 0)?"#666":(i % 3 == 0)?"#666":"#aaa";}); svg4.append("defs") .append("symbol").attr("id","hand") .append("path").attr("d","M 0 10 L 20 0 L 120 10 L 20 20 z"); var secondRot4 = svg4.append("g").attr("transform","translate(150,150)") .append("g").attr("transform","rotate(0)"); var secondHand4 = secondRot4.append("use") .attr("xlink:href","#hand") .attr("fill","#e33") .attr( "transform", "rotate(-90),scale(1.5,0.5),translate(-20,-10)"); var minuteRot4 = svg4.append("g").attr("transform","translate(150,150)") .append("g").attr("transform","rotate(0)"); var minuteHand4 = minuteRot4.append("use") .attr("xlink:href","#hand") .attr("fill","#999") .attr( "transform", "rotate(-90),translate(-20,-10)"); var hourRot4 = svg4.append("g").attr("transform","translate(150,150)") .append("g").attr("transform","rotate(0)"); var hourHand4 = hourRot4.append("use") .attr("xlink:href","#hand") .attr("fill","#666") .attr( "transform", "rotate(-90),scale(.75,1),translate(-20,-10)"); currentTimeAnimated = function() { var date = new Date(); var secondAngle = (date.getSeconds()/60) * 360; var minuteAngle = (date.getMinutes()/60) * 360 + (secondAngle/360)*6 + ((date.getHours() % 12)*360); var hourAngle = ((date.getHours() % 12)/12) * 360 + ((minuteAngle % 360)/360)*30; hourRot4.transition().duration(2000) .attrTween("transform", function() { return d3.interpolateString(d3.select(this).attr("transform"), "rotate("+hourAngle+")"); }); minuteRot4.transition().duration(2000) .attrTween("transform", function() { return d3.interpolateString(d3.select(this).attr("transform"), "rotate("+minuteAngle+")"); }).each("end", function() { secondRot4.transition().duration(300) .attrTween("transform", function() { return d3.interpolateString(d3.select(this).attr("transform"), "rotate("+secondAngle+")"); }); }); } resetTimeAnimated = function() { hourRot4.attr("transform","rotate(0)"); minuteRot4.attr("transform","rotate(0)"); secondRot4.attr("transform","rotate(0)"); }

We need to modify it a little bit so that the hands don't spin counterclockwise when going from 11 to 12, or from 59 to 0. So, with minor modifications, and a Javascript timer function, we have our functioning clock widget!

Script

updateTime = function() { var date = new Date(); var secondAngle = (date.getSeconds()/60) * 360; var minuteAngle = (date.getMinutes()/60) * 360 + (secondAngle/360)*6; var hourAngle = ((date.getHours() % 12)/12) * 360 + (minuteAngle/360)*30; animateClock(hourAngle,minuteAngle,secondAngle); } var currHourAngle = 0; var currSecondAngle = 0; var currMinuteAngle = 0; animateClock = function(hourAngle,minuteAngle,secondAngle) { hourRot.transition().duration(300) .attrTween("transform", function() { return d3.interpolateString("rotate("+currHourAngle+")", "rotate("+((currHourAngle > hourAngle)?(360+hourAngle):hourAngle)+")"); }).each("end",function(){currHourAngle = hourAngle;}); minuteRot.transition().duration(300) .attrTween("transform", function() { return d3.interpolateString("rotate("+currMinuteAngle+")", "rotate("+((currMinuteAngle > minuteAngle)?(360+minuteAngle):minuteAngle)+")"); }).each("end",function(){currMinuteAngle = minuteAngle;}); secondRot.transition().duration(300) .attrTween("transform", function() { return d3.interpolateString("rotate("+currSecondAngle+")", "rotate("+((currSecondAngle > secondAngle)?(360+secondAngle):secondAngle)+")"); }).each("end",function(){currSecondAngle = secondAngle;});; } window.setInterval(updateTime, 500); // Just to be safe, schedule it 2 times per second.

Demo

var svg = d3.select("#d3vis") .append("svg") .attr("width",dim) .attr("height",dim); var gHours = svg.append("g") .attr("transform", "translate(" + [dim/2, dim/2] + ")"); var gSeconds = svg.append("g") .attr("transform", "translate(" + [dim/2, dim/2] + ")"); var hourMarkers = gHours.each(function(d) { d3.select(this).call(clockFace, 12, dim/2); }).selectAll("circle"); var secondMarkers = gSeconds.each(function(d) { d3.select(this).call(clockFace, 60, dim/2); }).selectAll("circle"); secondMarkers.attr("fill","#aaa"); hourMarkers.attr("fill",function(d,i){return (i == 0)?"#666":(i % 3 == 0)?"#666":"#aaa";}); svg.append("defs") .append("symbol").attr("id","hand") .append("path").attr("d","M 0 10 L 20 0 L 120 10 L 20 20 z"); var secondRot = svg.append("g").attr("transform","translate(150,150)") .append("g").attr("transform","rotate(0)"); var secondHand = secondRot.append("use") .attr("xlink:href","#hand") .attr("fill","#e33") .attr( "transform", "rotate(-90),scale(1.5,0.5),translate(-20,-10)"); var minuteRot = svg.append("g").attr("transform","translate(150,150)") .append("g").attr("transform","rotate(0)"); var minuteHand = minuteRot.append("use") .attr("xlink:href","#hand") .attr("fill","#999") .attr( "transform", "rotate(-90),translate(-20,-10)"); var hourRot = svg.append("g").attr("transform","translate(150,150)") .append("g").attr("transform","rotate(0)"); var hourHand = hourRot.append("use") .attr("xlink:href","#hand") .attr("fill","#666") .attr( "transform", "rotate(-90),scale(.75,1),translate(-20,-10)"); updateTime = function() { var date = new Date(); var secondAngle = (date.getSeconds()/60) * 360; var minuteAngle = (date.getMinutes()/60) * 360 + (secondAngle/360)*6; var hourAngle = ((date.getHours() % 12)/12) * 360 + (minuteAngle/360)*30; animateClock(hourAngle,minuteAngle,secondAngle); } var currHourAngle = 0; var currSecondAngle = 0; var currMinuteAngle = 0; animateClock = function(hourAngle,minuteAngle,secondAngle) { hourRot.transition().duration(300) .attrTween("transform", function() { return d3.interpolateString("rotate("+currHourAngle+")", "rotate("+((currHourAngle > hourAngle)?(360+hourAngle):hourAngle)+")"); }).each("end",function(){currHourAngle = hourAngle;}); minuteRot.transition().duration(300) .attrTween("transform", function() { return d3.interpolateString("rotate("+currMinuteAngle+")", "rotate("+((currMinuteAngle > minuteAngle)?(360+minuteAngle):minuteAngle)+")"); }).each("end",function(){currMinuteAngle = minuteAngle;}); secondRot.transition().duration(300) .attrTween("transform", function() { return d3.interpolateString("rotate("+currSecondAngle+")", "rotate("+((currSecondAngle > secondAngle)?(360+secondAngle):secondAngle)+")"); }).each("end",function(){currSecondAngle = secondAngle;});; } window.setInterval(updateTime, 500); // Just to be safe, schedule it 2 times per second.

Thoughts on D3.

My original understanding was that D3 would allow view elements to be dynamically bound to data objects, and somehow we would be able to automagically rebind and refresh the view. But instead, all it provides is a mechanism to statically bind data (array elements) to selections, and a way to create/remove/update elements based on data provided. Don’t get me wrong, this is still a powerful library that helps simplify creation of amazing visualizations. And it comes with a rich set of helper functions around plot axis management, geo mapping, time slicing, interpolating string/colors/hues, etc. Just look at all visualizations its creator made while working for the New York Times. http://bost.ocks.org/mike/

Selectors

If you are familiar with JQuery or CSS Selectors, you already know how selectors work, and how powerful they can be in quickly applying a function on a list of elements. D3 works the same way, except the API is designed in such a way that every function returns a reference to the Selection it was applied on as well, so that you can chain calls gracefully like "attr()" above. The only exception is when the function creates a new selection, in which case that selection is returned.

So, in the above code, d3.select("#d3vis1") returns the selection for the div element, following which, append("svg") returns the selection containing the newly created svg element.

Okay, it's kinda cool that I don't have to do document.getElementById() all over the place, and the having every function return the selector it was applied on makes chaining calls easy... But I could do that with JQuery, and I don't see what's the hype about D3. Let's explore something a bit more complex to understand D3's most powerful feature...

Bindings.

Now instead of just one circle, let's draw 12 circles.

First, the iterative way.

var svg = d3.select("#vis") .append("svg") .attr("width",dim) .attr("height",dim); for (i = 0; i < 12; i++) { svg1.append("g") .attr("transform","translate("+[i*dim/12 + dim/24,dim/2]+")") .append("circle") .attr("r", (dim/2) * .9); }

and the D3 way...

var svg = d3.select("#vis") .append("svg") .attr("width",dim) .attr("height",dim); var circles = svg.selectAll("g") .data(d3.range(12)) .enter().append("g") .attr("transform",function(d){ return "translate("+[(d)*dim/12 + dim/24,dim/2]+")"; }) .append("circle") .attr("r", (dim/2) * .9 / 12);

Demo 2:

var svg2 = d3.select("#d3vis2") .append("svg") .attr("width",dim) .attr("height",dim); var circles2 = svg2.selectAll("g") .data(d3.range(12)) .enter().append("g") .attr("transform",function(d){ return "translate("+[(d)*dim/12 + dim/24,dim/2]+")"; }) .append("circle") .attr("r", (dim/2) * .9 / 12);

It may be hard to see the benefits of this approach right away, but notice how we were able to bind specific values of "d" (which is a range going from 0..11), and create elements for each of those values automatically, and in the end, automatically get the selection of all "circles" that were created in one step, even though they are a bunch of second-level elements from the node where data binding was applied. It may take a bit of effort in the beginning, but once you start wrapping you mind around it, coding with D3 becomes a thing of beauty. (Nerdgasm!)

Let's make things a bit more complicated and create a clock face. Let's draw the 12 inside a circle for the hour markers. And instead of having a for loop, lets do a simple binding operation on an array containing the numbers 1 through 12 to generate our circles.

Script 3:

var svg = d3.select("#vis") .append("svg") .attr("width",dim) .attr("height",dim); var g = svg.append("g") .attr("transform", "translate(" + [dim/2, dim/2] + ")"); var circles = g.each(function(d) { d3.select(this).call(clockFace, 12, dim/2); }).selectAll("circle"); function clockFace(selection, numDots, size) { offset = 0; radius = numDots * size / (numDots + 2); deltaY = (numDots % 2 == 0)?0:(radius / 2) * (1 - Math.cos(Math.PI / numDots)); selection.selectAll("g") .data(d3.range(numDots)) .enter().append("g") .attr("transform",function(d) { var angle = d * 360 / numDots + offset; return "translate(0,"+deltaY+")rotate("+angle+")translate("+radius+")rotate("+ -angle + ")"; }) .append("circle").attr("r",0.9*(size)/(numDots)); }

Demo 3:

var svg3 = d3.select("#d3vis3") .append("svg") .attr("width",dim) .attr("height",dim); var g3 = svg3.append("g") .attr("transform", "translate(" + [dim/2, dim/2] + ")"); var circles3 = g3.each(function(d) { d3.select(this).call(clockFace, 12, dim/2); }).selectAll("circle"); function clockFace(selection, numDots, size) { offset = 0; radius = numDots * size / (numDots + 2); deltaY = (numDots % 2 == 0)?0:(radius / 2) * (1 - Math.cos(Math.PI / numDots)); selection.selectAll("g") .data(d3.range(numDots)) .enter().append("g") .attr("transform",function(d) { var angle = d * 360 / numDots + offset; return "translate(0,"+deltaY+")rotate("+angle+")translate("+radius+")rotate("+ -angle + ")"; }) .append("circle").attr("r",0.9*(size)/(numDots)); }

Notice how easy it was to get a reference to all the circles you just created. Anyway, to draw the second/minute markers, we just need to call the same clockFace function with 60... Right?

Script 4:

var hourMarkers = g.each(function(d) { d3.select(this).call(clockFace, 12, dim/2); }).selectAll("circle"); var secondMarkers = g.each(function(d) { d3.select(this).call(clockFace, 60, dim/2); }).selectAll("circle");

Demo 4:

var svg4 = d3.select("#d3vis4") .append("svg") .attr("width",dim) .attr("height",dim); var g4 = svg4.append("g") .attr("transform", "translate(" + [dim/2, dim/2] + ")"); g4.each(function(d) { d3.select(this).call(clockFace, 12, dim/2); }).selectAll("circle"); g4.each(function(d) { d3.select(this).call(clockFace, 60, dim/2); }).selectAll("circle");

Wait a minute! What happened there? Why are there only 48 second dots, instead of 60?

That's because we are using the enter() data-binding function in our code, which defines how to create objects for additional data-points that don't have backing DOM elements from the results of the selection. So, in our case, there were already 12 circles created which get skipped.

One simple way to fix this is to have two separate top-level graphics contexts created, and draw into each one, so you won't find the 12 existing ones during the second call.

Script 5:

var gHours = svg.append("g") .attr("transform", "translate(" + [dim/2, dim/2] + ")"); var gSeconds = svg.append("g") .attr("transform", "translate(" + [dim/2, dim/2] + ")"); var hourMarkers = gHours.each(function(d) { d3.select(this).call(clockFace, 12, dim/2); }).selectAll("circle"); var secondMarkers = gSeconds.each(function(d) { d3.select(this).call(clockFace, 60, dim/2); }).selectAll("circle");

Demo 4:

var svg5 = d3.select("#d3vis5") .append("svg") .attr("width",dim) .attr("height",dim); var gHours = svg5.append("g") .attr("transform", "translate(" + [dim/2, dim/2] + ")"); var gSeconds = svg5.append("g") .attr("transform", "translate(" + [dim/2, dim/2] + ")"); var hourMarkers = gHours.each(function(d) { d3.select(this).call(clockFace, 12, dim/2); }).selectAll("circle"); var secondMarkers = gSeconds.each(function(d) { d3.select(this).call(clockFace, 60, dim/2); }).selectAll("circle");

Let me also introduce you to the function(d,i) signature for biding data to functions. So, function(d) signature can be used on attr() as well as call() functions to bind each array element in data d to each sekection element. You can also use the function(d,i) signature to get the array index passed in as well. And if there's no data bound to the selection, you can still use function(d,i) to get the index of the element in your selection. (Basically, you have bound null data to the operation)

So, let's try and make every 3rd hour marker a different color than the rest.

Script 6:

secondMarkers.attr("fill","#aaa"); hourMarkers.attr("fill",function(d,i){return (i == 0)?"#666":(i % 3 == 0)?"#666":"#aaa";});

Demo 6:

var svg6 = d3.select("#d3vis6") .append("svg") .attr("width",dim) .attr("height",dim); var gHours6 = svg6.append("g") .attr("transform", "translate(" + [dim/2, dim/2] + ")"); var gSeconds6 = svg6.append("g") .attr("transform", "translate(" + [dim/2, dim/2] + ")"); var hourMarkers6 = gHours6.each(function(d) { d3.select(this).call(clockFace, 12, dim/2); }).selectAll("circle"); var secondMarkers6 = gSeconds6.each(function(d) { d3.select(this).call(clockFace, 60, dim/2); }).selectAll("circle"); secondMarkers6.attr("fill","#aaa"); hourMarkers6.attr("fill",function(d,i){return (i % 3 == 0)?"#666":"#aaa";});

Evaporative Cooling

Having been a kid in the 80s growing up in India, I know a few things about cooling water through evaporation. How do earthen clay pots keep water cool in the hot arid climate? How do mud huts stay cool on hot sunny days? (Well, after splashing water on them) How do canvas pouches keep water cool in the desert?

Not going into too much detail, it’s because they are all permeable membranes with water evaporating on the outer surface extracting heat required for the state change (from liquid to vapor) from the water/air inside the membrane (pot, canvas, mud wall).

Air Coolers expand upon the same idea, and use that cold water to extract heat out of air to make it cool before blowing through a fan. They also help moisturize the air, which is a necessity in the hot dry playa. But, for what they are, they are quite pricey and quite power hungry. So, let’s figure out how to build a DIY Air Cooler out of dirt cheap materials for under $40.

Home Depot 5-gal Homer Bucket......... $3.00 120mm 12V DC Fan @ $7.50x2............ $15.00 12V DC Submersible Pump............... $5.00 Tube for drip system (enough for 2)... $3.50 Dura-Foam............................. $12.00

The setup goes something like this...

You basically drill holes along the top half of the bucket. You then line it with foam, and fill the bottom half with water. You then lay a drip system that pumps water on to the top of the foam, which slowly evaporates through the holes you just drilled, thus cooling the air/water/stuff inside the bucket. Oh, and most importantly, you blow this cooled air using fans from the top of the bucket. Simple stuff! Right?

Here’s what the setup looks like...

Please don’t cut the entire way down the bucket!!! I just drew a sectional diagram for illustrative purposes.

And here’s a simple wiring diagram, if you aren’t sure how to connect the wires...

A quick note about the battery: Use a Marine/Boat Battery to power your setup instead of a Car Battery. They have a more steady discharge curve, and are better for powering low power devices over long periods of time, unlike Car Batteries which are designed to discharge quickly to power the high-torque needed for the engine starter. Your headlamp lights mainly run off of the alternator, and use the battery only when the engine is turned off.

Staying Cool

Just blowing cold air inside your yurt may sound like a quick way to cool it. But now, we have to find ways to get it to stay cool, and this is where the R-Value (aka Thermal Resistance) of your build material matters. It specifies how slowly heat can transfer through the surface (or in other words, coolness can be lost), and the higher the R-Value, the better the insulation.

The rate at which this happens depends on three things:

How much insulation the walls provide. (aka the R-Value of the panels)

How much colder the inside is relative to the outside. (the higher the temperature difference, the larger the amount of heat transferred) 

How much surface area there is for heat transfer to take place. (Did you know? It’s better to pack cold food/ice in one large cooler than 2 smaller half-sized ones.)

The unit of R-Value is defined as h·ft2·°F/Btu.

That’s... time in hour * area in sq.feet * temperature difference in Farenheit / heat transferred in Btus (British thermal units). Gosh, I love imperial units!!! They make the math calculations so much simpler... said nobody! (Actually, it isn’t that bad if we stick to those units all along, but a lot of science literature is in S.I., as it should be.) Anyway, if it isn’t obvious from that...

The more time it takes to transfer the same amount of heat across the same surface area, the higher the R-Value. i.e, R-value is directly proportional to time. So, get the higest value you can afford in money and space, so that you can run your coolers less frequently. Similarly, the more surface area required to transfer the same amount of heat over the same amount of time, the higher the R-Value. Also, the more the temperature difference required to transfer same amount of heat... you get the point.

In addition to that, you need to counteract all the other sources of heat, such as your own body heat, Sun's radiation, etc.

Are you guys ready for some R-Value Mathematics?

Let’s assume we have a closed Yurt without any ventilation, that we’ve somehow managed to cool it down by 20ºF, and that we have 4 persons inside it.

Let’s first figure out how much heat is being added to our yurt’s interior. First, there’s body heat:

330 Btu/hr per person x 4 persons = 1320 Btu/hr.

Next, there’s Conductive Transfer of heat through the yurt’s walls from the hot air outside. i.e, R-Value part.

In our case, we can’t change the Surface Area of our Yurt (well, we could if we redesign it to be a geodesic dome), and the temperature difference is what we want to achieve... so, we will keep that target as a constant as well. That leaves us with rate of heat transferred per hour being the variable that’s directly affected by the R-Value. (Technically, inversely affected).

We know from before that the unit of R-Value is defined as h·ft2·°F/Btu, and our R-9 construction material implies that we have an R-Value, R = 9h·ft2·°F/Btu.

Given that the inside of our yurt is 20ºF cooler than outside, and our Yurt's walls and roof have a total combined surface area of 384ft2 (12 panels that are 8′x4′ each.), we could easily solve for the hourly rate (x) of Heat Transfer by solving:

9 hr·ft2·°F/Btu = 1hr * 384ft2 * 20ºF / (x) Btu

=> (x) = 384*20/9

=> x = 853.33 Btu/hr.

Then there’s radiative heat from the sun, but the reflective surface of the R-Max panels, and the fact that the inside is well shaded and dark kinda makes it negligible.

The same applies for radiative heat from the earth. Desert sand cools down pretty quickly in the shade, and the reflective tarp that we used in Yurt construction also helps make any radiative surface heat negligible

So, in total there’s 2173.33 Btu added to the system inside. To maintain a constant temperature inside, the same amount of heat must be extracted out of the system through some rate of water vaporization! This is on top of the what you need to cool it down to the desired temperature (about 20ºF cooler than outside) in the first place. If you’re smart, you can see from above that it’s way faster to cool the yurt with you not inside it as you will be the primary generator of heat :)

Let's now take a look at the other side of math to balance this out.

Enthalpy of Evaporation

Water is a great liquid for cooling purposes, primarily because of its strong Hydrogen Bonds holding its molecules together, that cause it to have a high enthalpy of vaporization – a.k.a latent heat capacity. (Did you know? It takes 5 times as much heat to vaporize boiling water at 100ºC/212ºF as it takes to heat it from  0ºC/32ºF to 100ºC/212ºF in the first place.)

For water, this value is 2257 kJ/kg. (kilojoules / kilogram)... 1 kg = 1 liter, and 1 kJ = 0.94 Btu. So, it’s about 2121 Btu/l. So, it takes just a bit over a liter of water per hour to reach equilibrium and maintain one of our yurts at 20ºF cooler than the outside with you sitting inside it. But, we also need to remember that the water evaporating from the foam surface in our bucket doesn’t just extract heat from inside the bucket, but also from the air outside... so to be conservative, let’s double the amount of water needed. So, if your home depot bucket setup can actually get about 2 liters (approx. 1/2 gallon) to evaporate in an hour, you can indeed maintain a yurt interior that’s cooler by 20ºF with 4 of you present inside.

Even if all the heat needed for evaporation is extracted from the air inside, it still takes about a liter/hr to maintain that temperature. That’s not a realistic expectation to set, even though you could do it with a large enough evaporator setup :/

Luckily, all that math above does not apply in our case! Say WHAAAAAAT??? Yep! if you look above, the very first assumption is that we are looking at a Yurt without ventilation. And, all that math still applies to that closed system :) But, we have Fans and Ventilation!!! 

Ventilation!

The hole on top of our roof for exhaust and the holes in our bucket for intake mean that we don’t have to cool and maintain the equilibrium for the entire volume of air inside the yurt. Instead, we just need to evaporate enough water to cool the smaller volume of air inside the bucket, before pushing it into our Yurt where it get’s heated up and pushed out of the exhaust. The cool air enters the yurt through the intake placed closer to the feeling, and pushes out warmer air up through the exhaust vent in the roof... thus rapidly bringing the temperature of the yurt down close to the temperature of the cold air inside your bucket air cooler :) This is why ventilation in yurts (and in buildings) is important!

You should also note that most of the heat added to the system (both conductive, as well as body heat) also escapes through the roof vent, thus reducing the cooling requirements of your evaporator.

Sorry about those unnecessary equations, and I wont bore you with more math about cooling the air inside the bucket ;) Just know that it doesn’t take much water to cool the air inside the bucket before blowing it through the fan. Though, I’m glad to do it if you guys are interested. But the main reason was to show you the importance of ventilation in any construction. Also, it’s more efficient as the heat extracted for vaporization comes from air that’s being sucked in through the holes in the bucket, which means none of the water is wasted to cool unused outside air.