Fred Wu's Blog

So about a month ago, I picked up a project I started a year ago but gave up shortly after: Crawler. At the time GenStage was just announced and I was interested in incorporating it into my project as I thought it'd be a great fit. But due to varies reasons - mostly not having a firm grasp of the GenStage concept and implementation, as well as taking on a CTO role at a startup, I couldn't find enough time and patience to make it work so I had to let it go.

Until now.

Crawler, on Steroids

I'd realised that the way I was trying to incorporate GenStage into my project was never going to work. Not because of GenStage itself, but because of the way I approached learning it. At the time I was so eager to make use of GenStage, and coming off the back of my good streak of releasing the aforementioned machine learning libraries, I thought I could take shortcuts and things would all work out perfectly.

No.

So I licked the wounds, learnt my mistakes and changed tactics. This time, I scoped out and encapsulated my learnings (just as one would in designing a software system), and eventually I came up with another library - OPQ: One Pooled Queue.

OPQ: One Pooled Queue

I knew Crawler could technically work without any queueing system, or GenStage. So I kept on building Crawler, until I felt productive in writing Elixir again, and touched enough areas and concepts that I knew exactly what I needed from GenStage.

And luckily for me too, GenStage over the past year has matured and more importantly, has better documentation with more code examples. Upon closer investigation of the code examples, I found that their GenEvent and RateLimiter examples were almost exactly what I needed. It was an epiphany moment for me after reading and understanding these examples, all of a sudden I "get it".

If you take a look at the source code of OPQ you'll notice that the heavy lifting logic was mostly inspired (or even copy-pasted) from those examples.

Open Sourcing is Contributing and Caring

Up until this point, as far as writing open source Elixir code goes, it had mostly been me writing my own code for my own projects. But open sourcing is much more than just writing one's own code and publishing them on Github.

If you've followed my work you'll know that I'm a big fan of contributing to other projects, some of which are well-known ones like Rails and Slim.

It just so happens, that over the weekend I ran into situations where I needed to contribute to other Elixir projects - and spoiler alert: one of which is Elixir itself.

ElixirRetry

One of the features I set out to add to Crawler, was to allow failed crawls to retry before giving up. Naturally, the first thing I did was to try find an existing package to support the retry functionality.

After some googling and digging into some source code, I've found and settled on ElixirRetry - a neat library that was cleverly built.

Soon I found out that since its last release, it offered both retry/2 and retry/3, the latter of which was to support an extra argument that specifies which exceptions to allow as part of the retry flow. It was a great addition, but it doesn't effect me as Crawler doesn't need it.

As a developer who cares deeply about code quality and clarity, I immediately thought about how the interfaces for retry/2 and retry/3 could be improved, by simply combining them and making the extra argument in retry/3 an option.

So I did, and issued a pull request here: https://github.com/safwank/ElixirRetry/pull/12 You'll notice that I've also made some improvements to the test suite while I was at it. ;)

Elixir Typespec

One issue I ran into while building Crawler, was the static analysis via Dialyzer - some code that actually ran and passed tests functionally, failed the type checking.

As someone who isn't as experienced in Elixir, I first opened an issue and phrased it more as a question seeking validity of the issue. I then jumped on Elixir's Slack group and asked people in the group about this issue.

Fortunately, Ben Wilson immediately came to aid by verifying my issue and validating my suspicion of it being an issue in Elixir's typespec documentation.

And so, a pull request was created and approved shortly after.

Sharing is Caring

The bulk of Crawler as well as the entirety of OPQ were built in the past month or so. I hope some people will benefit from having these libraries around. And, I hope people also enjoyed me sharing my experience, and perhaps be inspired to start sharing more too.

Learnings

The preface is to give a bit of background of when and how I started learning Elixir, now, let me talk about one of my favourite features of Elixir, and how it helps me write better code not just in Elixir, but in virtually any other language.

Introducing the topic of today, a really simple feature, and in fact it has been part of Python for years - the doctest.

Doctest

In short, a doctest is pieces of code examples that run as part of the test suite, and show up as part of the documentation. For example:

defmodule Greeting do @doc """ ## Examples iex> hello("world") "hello world" iex> hello("dear") "hello dear" """ def hello(input) do "hello #{input}" end end

And then all you need is in your corresponding test file, to enable doctest:

defmodule GreetingTest do use ExUnit.Case doctest Greeting end

Even though I knew about Python's doctest for years, I'd never realised how impactful it can be given its simplistic nature, probably due to the fact that I've never used Python for anything serious.

Until Elixir.

There are three things I find the most impactful as I write more doctests: clarity, scope, and design.

Clarity

As a ruby programmer, I appreciate greatly the beauty of not just the main code base, but also its test suite. Hence I've always preferred to use RSpec in larger code bases. However, as the application gets more complex, and as the number of test files grows, the cognitive overhead of reading and processing all the files and lines becomes higher and higher.

Doctest solves this perfectly - no longer do we have to crawl through the right file and the right line for a particular test case, all the test cases are neatly presented right in front of you as you read the function itself.

You might think this as trivial, but just like many organisations spend time and effort to optimise for effective communication by studying proxemics, proxemics between different components of a software code base also plays a role in improving the code clarity, and ultimately the code quality.

Scope

Doctest is purposely simple, and is designed for unit tests. There have been many times when I found myself realising my function was too dependant on external states, or are doing too many things because it was hard to write simple doctests. In a way, the constraints of doctests have forced me to rethink the scope of my function, and that would often lead to an overall better designed system.

Design

As much as I'd like to think about the SOLID principles all the time, it is often too easy to dig deep wholes in the midst of building things.

Every now and then I find myself extracting a piece of logic to a private function and calling it a day. In Elixir, only public functions can have doctests - again, this constraint pushes you to think about the importance and the role of a particular function, perhaps it is better to be moved to another module as a public function therefore can have its own doctests. Here is an example when I did some refactorings on Crawler.

Closing

I wanted to write this article for a while now - as I truely love and appreciate Elixir's asthetics and features. Many developers might find functional programming as a barrier, but I can assure you that with Elixir's tooling and ecosystem, and of course doctest (wink), building software feels like a breeze.

Learning Elixir

Given the strong influence of Ruby in Elixir, I can’t help but get nostalgic about learning it by reading another great book by Dave Thomas: Programming Elixir.

Dave Thomas’ original Programming Ruby was instrumental in the success of the Ruby programming language and its communities in the west, and it certainly has helped me to greatly widen my exposure to not only the wonderful world of Ruby but object-oriented programming in general as a PHP developer.

Learning Elixir has been really fun, not only does it share the same developer-friendliness championed by Ruby, it also does so with remarkable strength in concurrency thanks to BEAM (Erlang’s VM).

Many people are drawn to Elixir due to its Ruby influence, its functional and immutability nature, and its Actor-based concurrency model. I would however like to call out one of my favourite features of Elixir that sometimes gets overlooked, and that is how you could write unit tests using ExUnit.DocTest.

Elixir Makes Writing Unit Tests Effortless

Python programmers have been writing doctests for years, but Ruby due to not having an equivalent standard library, has never had writing doctests taken off in its community. I’m glad Elixir has.

Take a look at the code snippet below:

defmodule Stemmer.Step0 do @doc """ ## Examples iex> Stemmer.Step0.trim_apostrophes("'ok") "ok" iex> Stemmer.Step0.trim_apostrophes("o'k") "o'k" iex> Stemmer.Step0.trim_apostrophes("'o'k'") "o'k" """ def trim_apostrophes(word) do word |> String.replace_prefix("'", "") |> String.replace_suffix("'", "") end end

The three test cases given will actually be tested by ExUnit, provided you ask it to do so as below:

defmodule Stemmer.Step0Test do use ExUnit.Case, async: true doctest Stemmer.Step0 end

You may think having doctests is not a big deal, what I can say is that I am really enjoying the fact that unit tests can be written with minimal friction and high visibility (as they sit with the implementation, rather than within a big test suite with a sea of files). Besides, not all of us practice TDD religiously or 100% of the time, and when I don’t or can’t do TDD, having doctests ensures I don’t forget adding test cases. :)

Toy Robot in Elixir

The best way to learn something new is to practice it as you learn it. Instead of diving straight into a complicated system, or to write a hello world program, I thought I would dig out a code test and re-implement it in Elixir.

The code test is the rather infamous Toy Robot test. As an interviewer, I must have reviewed a few hundreds of these tests, most of which in Ruby. I know what a good OO solution looks like, so naturally I was looking forward to “rethink” the problem and have a crack at it using Elixir.

Here is the result: https://github.com/fredwu/toy-robot-elixir

As a learner, I wanted to practice as many language features as possible, so included in the Toy Robot test code I tried:

Different data structures such as Struct and List

Pattern matching

Data piping (|>)

Agent for managing states

Macros for validation rules

I kept the test code small and nimble on purpose to illustrate the readability of a highly expressive language. I may not have exercised all the language features (such as supervisors and protocols), but at that point I felt I could start the Elixir journey with a big smile on my face already.

Learning Phoenix

I have to admit, a big part of the reason that got me started on learning Elixir is to build a side project. Initially I wanted to simply use Ruby and Rails because I can be very productive using them, but in the end I decided on learning Elixir and Phoenix, because most side projects fail, and when they do, how much learning can you extract out of those experiences? My answer to this question, is to learn a new language, a new framework and most importantly a new programming paradigm to drastically increase the amount of learning I could gain.

But please, before you jump on Phoenix, learn Elixir properly first! Over the years I’ve seen far too many cases of people jumping on Rails without understanding the language features of Ruby…

To learn Phoenix, the Programming Phoenix book is pretty much the bible on this subject aside from the official guide, and it is written by Chris McCord, the author of Phoenix, Bruce Tate, and Jose Valim, the author of Elixir.

As an experienced Rails developer, it only took me a day or two to go through the book (I largely skimmed the chapters on Channels as I don’t intend to build a real-time app).

Now, as a Ruby developer, the biggest pain point when working on Rails or other sizeable Ruby MVC projects for me, is ActiveRecord. I got so frustrated to the status quo I even attempted to fix it.

Fortunately, in Elixir and Phoenix we have Ecto. One of my favourite features of Ecto is the concept of Changeset.

Take a look at the code snippet below:

defmodule MySecretApp.User do use MySecretApp.Web, :model schema "users" do field :username, :string field :email, :string field :password, :string, virtual: true field :encrypted_password, :string has_one :profile, MySecretApp.Profile timestamps() end def changeset(struct, params \\ %{}) do struct |> cast(params, [:username, :email]) |> validate_required([:username, :email]) |> validate_length(:username, min: 3, max: 20) |> validate_format(:username, ~r/\A[a-zA-Z0-9_]+\z/, message: "alphanumeric and underscores only") |> validate_format(:email, ~r/@/) |> unique_constraint(:username) |> unique_constraint(:email) end def creation_changeset(struct, params \\ %{}) do struct |> changeset(params) |> password_changeset(params) end defp password_changeset(struct, params) do struct |> cast(params, [:password]) |> validate_required([:password]) |> validate_length(:password, min: 8, max: 200) |> encrypt_password end defp encrypt_password(changeset) do case changeset do %Ecto.Changeset{valid?: true, changes: %{password: password}} -> put_change(changeset, :encrypted_password, Comeonin.Bcrypt.hashpwsalt(password)) _ -> changeset end end end

The changset/2 function can be used when a user needs to be updated, whereas the creation_changeset/2 is to be used only when a user is first created. Sure, you can achieve similar result in Rails by using custom validators, but the fact that this practice is enforced by the library and the framework, is encouraging.

One other thing that I’ve seen in larger Rails apps, is the leaky abstraction of view-level logic, they typically sit in controllers, helpers (which are globally available) or worse, models. Phoenix in this case follows what Hanami, Trailblazer and many other frameworks do: it introduces a “view model” layer.

Something along the lines of:

defmodule MySecretApp.UserView do use MySecretApp.Web, :view def full_name do "#{title} #{first_name} #{last_name}" end end

In a nutshell, Phoenix is just like Rails, but less magical (in a good way) and faster. :) Like the Elixir eco-system, most libraries and concepts feel like their ruby/rails counterparts, but more refined.

There are a few popular Hex packages if you’re familiar with their ruby counterparts:

Elixir Ruby Code analysis credo rubocop Testing espec rspec Browser testing wallaby capybara Test coverage excoveralls simplecov Test factory ex_machina factory_girl Authentication guardian devise

And the list goes on and on… Be sure to check out Awesome Elixir for more community curated Elixir libraries.

Learning Machine Learning

As I started building the foundation of my side project, you know, the usual user management and session management, etc, etc, by chance I came across mentions of Bayesian inference.

One thing led to another, very soon I started looking into varies different machine learning algorithms such as Naive Bayes and Random Forest. These are all subjects I had never come across or even heard of before, as I have no strong mathematics, statistics or computer science background. I was however, intrigued and inspired nonetheless, and wanted to employ some machine learning in my side project.

And the best way to understand and learn about machine learning algorithms? You guessed, is to write one! After some research, I came to the conclusion that Naive Bayes is one of the simplest to implement, is great for text classification which is useful for my side project, and has good accuracy given its simplicity and fast speed.

Introducing Simple Bayes

So, after a few train rides to and from work, on my newly purchased Macbook (yes, the one with only a single USB-C port), I built a Naive Bayes library: Simple Bayes.

As of writing, this is the only Naive Bayes library in Elixir that supports the following features:

Naive Bayes algorithm with different models

Multinomial

Binarized (boolean) multinomial

Bernoulli

No external dependencies

Ignores stop words

Additive smoothing

TF-IDF

Optional keywords weighting

Optional word stemming via Stemmer

Let’s see it in action shall we? :)

SimpleBayes.init |> SimpleBayes.train(:ruby, "I enjoyed using Rails and ActiveRecord for the most part.") |> SimpleBayes.train(:ruby, "The Ruby community is awesome.") |> SimpleBayes.train(:ruby, "There is a new framework called Hanami that's promising.") |> SimpleBayes.train(:ruby, "Please learn Ruby before you learn Rails.") |> SimpleBayes.train(:ruby, "We use Rails at work.") |> SimpleBayes.train(:elixir, "It has Phoenix which is a Rails-like framework.") |> SimpleBayes.train(:elixir, "Its author is a Rails core member, Jose Valim.") |> SimpleBayes.train(:elixir, "Phoenix and Rails are on many levels, comparable.") |> SimpleBayes.train(:elixir, "Phoenix has great performance.") |> SimpleBayes.train(:elixir, "I love Elixir.") |> SimpleBayes.train(:php, "I haven't written any PHP in years.") |> SimpleBayes.train(:php, "The PHP framework Laravel is inspired by Rails.") |> SimpleBayes.classify("I wrote some Rails code at work today.") # => [ # ruby: 0.20761437345986136, # elixir: 0.08101868169313056, # php: 0.019047884912605735 # ]

As you can see, provided with reasonable training data, Naive Bayes can work extremely well.

Introducing Stemmer

Something I discovered as I was building Simple Bayes, is something called stemming. Let’s see another example:

SimpleBayes.init(stem: false) |> SimpleBayes.train(:apple, "buying apple") |> SimpleBayes.train(:banana, "buy banana") |> SimpleBayes.classify("buy apple") # => [ # banana: 0.057143654737817205, # apple: 0.057143654737817205 # ]

Oops, the probabilities of the sentence “buy apple” of being apple and banana are the same, that’s not good, as we know “buying” and “buy” should mean the same thing. This is where stemming comes in handy. Let’s enable stemming and run the example again:

SimpleBayes.init(stem: true) |> SimpleBayes.train(:apple, "buying apple") |> SimpleBayes.train(:banana, "buy banana") |> SimpleBayes.classify("buy apple") # => [ # apple: 0.18096114003107086, # banana: 0.1505149978319906 # ]

Stemming in this case correctly identified “buy” and “buying” as the same thing (they both have the stemmed root of “buy”).

How did I include stemming in Simple Bayes you wonder? Let me introduce you to Stemmer - as of writing this is the only Porter2 stemmer available in Elixir. :)

Elixir and BEAM may not be known for their raw performance, but compared to a similar Porter2 stemmer implemented in pure ruby, my Elixir version runs about five times faster (under 5s to stem 29000+ words, compared to 25s with the ruby version).

The Learning Continues…

I’ve only been writing Elixir for a month, and there are still heaps to learn and to practice. So far though, I have to say my experience with both Elixir and Phoenix has been fantastic - I get the same satisfying and pumped feeling I got when I was learning Ruby and Rails years ago.

One morning I came to work, noticed a few server alerts that required attention. This was pre AWS and the whole Devops automation movement days so I needed to log onto the server. For some reason my access was revoked so I had to log into the hosting dashboard to reset the root password. The senior dev on the team turned up, and when asked about my access, casually told me: "oh you did something on the servers the other day, so I removed your access."

CTO asked me to help building a new mission critical PHP and MySQL service, upon discovering an existing service already built on Ruby and MongoDB by three very senior developers and consultants, CTO said the reason for the rebuild was because “ActiveRecord is too slow.” Let me remind you that the initial Ruby solution was built using MongoDB. At the time the company had about 30-40 experienced Ruby developers, and only 2 junior to mid level PHP developers who were specifically hired to build the PHP/MySQL service which turned out to be a disaster.

As the team lead who was responsible for evaluating performance and setting salary, I was told by one of my team members that they had gotten a pay rise. Surprised, I asked one of the founders of the company who is non-technical what was going on, during the conversation I discovered another pay rise was given to one other member of my team - all without consulting or even informing me.

Excited to have had my overseas conference talk proposal accepted, I asked the new head of our department who had joined our company for only a month or two for travel approval. Fully expected a pad on the back, I was surprised to have been told that I cannot go, with the reason being “it is just not a good time”.

One of our servers needed to be rebuilt, so trying to be a team player and help, I started installing some basic packages. The senior dev on the team turned to me, straight faced and enunciated in a deep and cold voice: “don’t touch anything on it, this is my server!"

The new development manager walked into our meeting, pulled me out, guided me to another meeting room where the new CTO was waiting. The new CTO opened with “so it’s time for us to go separate ways”, and implied the reason being something that I could definitely have pushed for unfair dismissal. Trying to be professional, I walked back into my original meeting, but the new development manager quickly pulled me out and said “you need to go right now”. He then stood behind me, saw me formatted my work laptop, then literally escorted me out of the office. The new development manager and CTO soon drove the company to the ground and left the country.

After submitting the salary increases for the all my direct reports, I was delighted that the CEO was happy with them all. Given I have had no pay increases in over one and half years I asked for a modest pay increase for myself. I was told “no, we have to discuss this after your project delivery.” My direct reports and I were in the same delivery team.

It was near the end of the working day, around 5pm, as the person coordinating the developer recruitment in our area I ping’ed our Slack channel with a lighthearted message to encourage our developers to start reviewing some code tests from job candidates if they're free. One person replied: “My end of day activity is doing the stuff I should have been doing all day instead of the other things that came up."

One day I got really really sick - difficult to walk or even stand still, dizzy and having fever. Upon informing one of the founders, I was told to come into the office, even though there was no urgent tasks need to be done. Reluctantly obliged, I came into the office, went out for lunch with colleagues but I was so sick I could not eat anything and can barely sit straight. I then went home, and was sick for the rest of the week...

My manager, who was the general manager, wanted to fire two of my developers. His tactic for firing the senior developer was to make the senior developer role redundant and offering the dev a junior developer role instead.

I was tasked to investigate setting up a Cassandra database cluster. Upon discovering certain network limitations, I approached the then head of network operations, when asked about the inability to access the Cassandra cluster, he replied confidently that “it works”. Later on I discovered that he successfully telnet’ed to the ports and declared it “working".

The head of HR who is female and a big advocate for workplace diversity walked into my exit interview. The opening line was “Ordinarily I only do exit interviews with female employees..."

New to the big corporation environment, after asking around a few people and could not figure out how to provision an iPad for my team (but knowing some other teams do have company iPads), in an email reply by the head of enterprise IT: “Hey Gents, Fred has escalated this through multiple channels today. Awesome effort on his behalf, to seems like he didn’t like our response.” To which a colleague had to gently remind him that it wasn’t my fault there was no documented process for provisioning hardware.

After having some discussions on our MySQL database, the dev who had recently joined my team all of a sudden raised his voice in an extremely arrogant and dismissive tone: “you clearly don’t understand this feature, do you?"

A senior software architect who I used to respect walked close by in a meetup. We worked for the same company a while back so I smiled, said hi and was about to start a conversation, he quickly cut me off with “I need to get a drink” without looking at me and wondered off.

The general manager who is non-technical, asked me to investigate options to uplift our ageing bespoke ecommerce solution. Upon delivering my findings, I was told that “your findings are biased.” Later it was clear to me that he wanted confirmation of his agenda from the technical perspective and his mind had already been made on the approach we should take.

A new big shot executive joined the company as our new CTO. He had no agile background and our company was transforming and pushing for a lot of agile principles at the time. Two weeks into his new appointment, the new CTO published an internal document titled “Controlled Chaos”. After reading the document everyone immediately realised that he was describing waterfall. The document was shared as a Google Doc and was open for comments, so people started asking hard questions. Weeks later, many of us who were vocal about his document were let go.

It is very tempting to rely on the wonderful and powerful Excel formulas to help record and analyse data points and generate metrics such as cycle time. However, punching things into a spreadsheet is tedious, error-prone, time consuming and violates the DRY principle.

The spreadsheet I used to use for tracking cards.

Introducing Amaze Hands

As someone who strives to keep writing code even in a non-technical role, I started building a tool called Amaze Hands to help reduce the amount of waste I accumulate as a delivery lead.

Amaze Hands' simple Web UI.

Analyse Cards Like Poker Hands

I used to play a bit of online poker and one thing you do in online poker is to look at your hand history to understand the game and your opponents.

If you think about it, agile boards are just like poker games - there is history to what has happened in the past, and in order to optimise for future gains, we need to understand what went wrong and what to improve on a case-by-case basis.

One of the teams I am apart of uses LeanKit, whilst it is a good tool, its reporting functionalities are very limited and its XML export function is completely broken. As a result I started building Amaze Hands to parse the copy-pasted card history from LeanKit, and to eventually generate the metrics I care about.

The LeanKit strategy which consists of a parser and a transformer is able to parse the copy-pasted text from a card as shown below:

And below is the high level architecture of Amaze Hands:

+---------------------+ | Text | <- Raw text input. +----------+----------+ | +--------------v--------------+ | Strategies | +-----------------------------+ | +---------------------+ | | | Parser | | <- Parses text into an AST. | +----------+----------+ | | | | | +----------v----------+ | | | Transformer | | <- Transforms the AST into a common AST. | +---------------------+ | +--------------+--------------+ | +----------v----------+ | Builder | <- Builds the dataset from the common AST. +----------+----------+ | +----------v----------+ | Reducer | <- Filters the dataset. +----------+----------+ | +----------v----------+ | Analyser | <- Analyses the dataset for metrics. +----------+----------+ | +----------v----------+ | Producer | <- Produces metrics. +----------+----------+ | +----------v----------+ | Presenter | <- Presents metrics. +---------------------+

Incremental Analysis - Zero In On Metrics That Matter

No projects are equal, and no project teams are equal - the goal of Amaze Hands is to incrementally add intelligence to our agile process that matters to a particular project and its delivery team.

By incrementally adding and/or filtering data points for analysis, we will be able to zero in on the problematic areas of our agile process. The following is a list of potential areas we could perform analysis on:

cycle time

wait time

blocked time

knocked-back time

context switch (between different streams of work)

other factors such as meetings, attrition, etc

As of the time of writing, Amaze Hands supports the following common metrics:

cycle time (mean and median)

cycle time rolling average (mean and median)

wait time (mean and median)

wait time rolling average (mean and median)

standard deviation rolling average

cycle time scatter

It's Just the Beginning

Amaze Hands started off as an REA Hackday project - on the technical level (hey I still see myself as a developer!), the tool was built in a way that it isn't over-engineered (a.k.a. slow to get it out the door and validate its usefulness) but at the same time has multiple layers as shown in the architecture diagram above so I could refactor and optimise each layer independently when necessary.

It is still early stage, but I thought I'd share what I have right now to gather some feedback and perhaps inspire fellow project leaders to look into optimising your own workflow.

When I started Amaze Hands I was only leading one project team that uses LeanKit, but since last week I started leading another team that uses a physical wall - I can't wait to adapt Amaze Hands to support the new input stream.

"Probably no one." I shrugged, and decided to simply suck it up and sleep it off, hoping solutions to the problems I had would come to the light of the day.

I got home, opened up my laptop and started reading unread emails. "Software Lead Weekly, ah I haven't read the latest issue yet," I thought to myself, "I wonder who the curator is."

Discovered "Leading Snowflakes"

And that had lead me to discover Oren Ellenbogen's book Leading Snowflakes.

On the book's landing page it says:

This isn't another 350-page book you'll agonize yourself over not reading. It's short, concise and pragmatic. You're busy, I get that.

"This is great!" I thought, with a sense of uncontrollable excitement. After all, these days I found myself hard to justify spending hours and hours on books that talk about subjects in a hand-wavy manner. A short, concise and pragmatic book on leadership was exactly what I needed.

There was a free sample chapter for download, so I clicked on it without hesitation. To my surprise, after downloading the sample chapter, Oren reached out to me with a personal email, and that had quickly lead to a chain of emails of me writing about the things at work that I found challenging, and Oren providing sound advice and suggestions.

I felt not only grateful, but lucky. Over the years I had worked at varies places where I was fortunate enough to have met quite a few people who I consider my mentor. But to run into one who I had never met before, on the same night my spirit was crushed, and who were willing to not only listen to my rant but also to walk me through my problems and offer advice - it was like finding a needle in a haystack.

My decision to purchase the book was made half way through our email exchanges, as I was browsing through the links Oren provided on his book landing page (under the section of "Sharing my lessons learned"), I found them to be very insightful.

Convinced that Oren's book would offer even more value, I quickly bought the complete package with the book and all 10 interviews.

Leading Snowflakes - A Book Every Software Team Manager Should Read

Fast forward to today, I have finally finished reading the book, and I could confidently say that this is a book every software team manager should read.

There is no hand-wavy theories or lectures, what the book offers is exactly what is said on the landing page - short, concise and pragmatic.

The book contains a lot of sound advice, practical tools and techniques as well as a ton of useful links. Reading through it, I can recall many moments where I thought to myself, "hey, that indeed is a very cool technique, I can totally see how I could use it and be better at leading."

I am now ready to implement those techniques and I am hopeful that I would become a better team leader and my team would become a better team as a result. :)

If you are a software team manager, or aspire to be one, I highly recommend this book!

Breakfast in North Melbourne:

Conference:

One Week is a Major Commitment for the Candidate

In Australia, a full time employee typically gets four weeks of annual leave, with one or two weeks of which used up for the Christmas / New year down time. We are looking at asking candidates to spend 33-50% of their vacation time to commit to a trial week for one company - a terrible ROI (Return On Investment) from the candidate's perspective if you ask me.

Candidates who are currently employed, with multiple offers from other organisations are more likely to skip the trial week - from experience, this is often the higher quality candidate pool.

Side Effects

Increases the likelihood of burnout due to the reduced vacation time

Shrinks the candidate pool

Misses top talents who are unable to make the one-week commitment

As a result, the overall quality of the candidate pool drops

Paints an image of "not-caring (enough) about the employee's well being"

Of course, since the trial week is paid for, the employee could always take unpaid leave from their current employer.

Side Effect

Raises alarm bells at current workplace since one week of unpaid leave is significant

One Week is a Major Commitment for the Team

Given the trial only lasts a week - we better make it count! That means one or more current developers need to be assigned to take care of the trial developer - pairing and walking through existing systems, etc. This is assuming we are going to act responsibly, and not simply just direct the trial developers to their desks and ask them to "go for it".

Side Effects

Higher pressure for the team

More difficult to act on other priority tasks

Developer Productivity Curve (One Week is Not Enough)

From my experience of on-boarding new developers, it typically takes 4-8 weeks for a developer to become productive and effective in a new work environment.

According to Weebly, candidates are assigned with a project that is small enough to do in a week, but still resembles what the candidate would be doing if hired. It sounds great if it works, but for many organisations this is unfeasible, for instance:

There is no small projects to assign, unless invented

Navigating documentation and source code would take days, if not weeks

Either way, with one week of trial, the candidate is unlikely to have enough time to contribute as well as to be integrated into the team culture.

Side Effects

Higher chance of misjudging the candidate's ability and productivity

Significantly higher chance of creating solutions misaligned with the team and/or the organisation

Higher maintenance cost should the team decides to keep the solutions created

66% Hire Rate Suggests Deeper Hiring Issue

Weebly at the end of their blog post writes:

Our hire rate out of trial week is around 66%, which feels like the right level.

I respectfully disagree. A 66% hire rate from the trial week is a 34% failure rate on the pre-trial week recruitment process, and this is significant.

Which brings us to...

More Effective Ways to Vet a Candidate

Where I work, we have a simple, three-step recruitment process:

Complete a small and fun code challenge, in your own time and with your own pace. The code challenge usually takes 2-4 hours to complete.

Invited to our office to chat with our developers and founders, optionally done via video chat. This usually takes an hour or so.

Pairing session, usually takes 30-60min.

Step 2 and 3 are sometimes swapped. And we also check out the candidate's Github account if available, and their past projects if public.

In the code challenge we vet the candidate's problem-solving ability, software design sense, code quality, code style and ethics (it's easy to tell whether they cheated).

During the chat we vet the candidate's project experience, depth of knowledge, breadth of knowledge, communication skill and culture fit.

In the pairing session we vet the candidate's development practice, thought process and the ability to articulate.

By the end of the three steps we are usually pretty confident on +1 or -1 to hire the candidate. If we aren't, it's a -1.

But hold on, didn't I mention one week is not enough for a candidate to be productive and effective? Yes! And that's why most places have a three-month probation.

The difference between the long probation period and the short trial period, is not only in duration, but more importantly in commitment. In my opinion, only when both parties are committed can you achieve great result.

So, let's hear your say, what do you think? :)

RSpec.configure do |config| config.before :suite do DatabaseCleaner.clean_with :truncation DatabaseCleaner.strategy = :transaction end config.before do if example.metadata[:js] || example.metadata[:type] == :feature DatabaseCleaner.strategy = :deletion else DatabaseCleaner.strategy = :transaction DatabaseCleaner.start end end config.after do DatabaseCleaner.clean end end

Essentially, we want to truncate the DB only once before the whole suite runs to ensure a clean slate DB, then we only want to use deletion on Capybara tests, everything else should just use transaction which is the fastest strategy.

Now, as a bonus, I have just discovered @amatsuda's DatabaseRewinder which is a lightweight alternative that supports only ActiveRecord. It offers comparable performance with a much similar API.

RSpec.configure do |config| config.before :suite do DatabaseRewinder.clean_all end config.after do DatabaseRewinder.clean end end

By the way, we also use parallel_tests to scale our test suite to multiple processes, even on Travis CI and Wercker.

it does not replicate implementation details;

it does not provide false sense of security;

it runs reasonably quickly;

it does not slow down the development significantly;

it guides the programmer towards a better architecture;

and, it does not make you sigh every time you want to modify your tests.

Art and Science, TDD or Not TDD

Just like writing production code, writing tests is also a combined form of art and science. It takes not only experience, but also intuition to write sensible tests. You have to remember that not all projects and programmers are equal - take what you get, practise, and reflect on your findings.

Many times I had come across seasoned programmers practising TDD, only to find themselves cornered into a bad design that ultimately had to be thrown away. TDD does not save you from writing bad code, this article is not about TDD, it's about testing in general.

I am most comfortable with using RSpec, FactoryGirl, Capybara and Turnip, so I'm going to use these tools in the code. The principles however apply to any testing framework.

Test as Little as Possible to Reach a Given Level of Confidence

Kent Beck, the inventor (or more correctly, 'rediscoverer') of TDD once said:

I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence.

I used to prefer testing almost everything, but over the recent years I find myself increasingly look for key areas of the system that need the test coverage the most. Typically, our systems would have:

unit and functional tests for model behaviours

unit and functional tests for services

integration tests for controller actions

request tests for API endpoints

isolated JavaScript tests

high level integration/acceptance tests in Gherkin

Model and service level tests are arguably the most important ones so we make sure we have really good test coverage for those. For controller tests we rely heavily on reusable production and test code for maintainability and sanity. For API endpoints we mostly test presented data structure - as business logic and data integrity should have been covered in model, service and controller layers. Isolated JavaScript tests take care of both presentational business logic and tricky UI tasks. And finally, acceptance tests handle happy-path user interactions.

Do Not Test Framework and Library Code

Writing application-specific business logic is difficult enough, you really should not test functionalities provided by the framework or libraries. Below is an example of such bad tests:

describe ApprovalStakeholder do it { should belong_to(:approval) } it { should_not validate_presence_of(:approval) } end

Similarly to how you would add useful comments, i.e. describe why instead of what, these tests should be replaced by tests that cover actual functionalities, for instance the reason why an ApprovalStakeholder doesn't need an Approval to be presence should be demonstrated in the tests:

shared_examples_for "non-approval specific stakeholder" do its(:action_that_does_not_care_about_approval) { should be_true } end describe ApprovalStakeholder do let(:approval) { create(:approval) } let(:user) { create(:user) } let(:role) { create(:role) } subject do build(:approval_stakeholder, :user_id => user.id, :role_id => role.id ) end context "with an approval" do before { subject.approval = approval } it_behaves_like "non-approval specific stakeholder" its(:action_that_does_care_about_approval) { should be_true } end context "without an approval" do it_behaves_like "non-approval specific stakeholder" its(:action_that_does_care_about_approval) { should be_false } end end

Ensure What You are Testing Makes Sense

The test case below showcases the original developer's lack of attention and awareness on designing a functional and secure system. It actually tests the reference keys for the ApprovalStakeholder object are allowed to be mass assignable, which is a recipe for disaster.

describe ApprovalStakeholder do it { should allow_mass_assignment_of(:user_id) } it { should allow_mass_assignment_of(:role_id) } end

De-Duplicate Test Cases

Looking at the example below, the first thing you'd notice is the amount of duplication.

describe ApprovalStakeholder do it "#traveller" do stakeholder = create(:approval_stakeholder, :approval => approval, :user_id => traveller.id ) stakeholder.stub(:user).and_return(traveller) approval.stub(:stakeholders_as).and_return([stakeholder]) approval.traveller.should == traveller end it "#authoriser" do stakeholder = create(:approval_stakeholder, :approval => approval, :user_id => authoriser.id ) stakeholder.stub(:user).and_return(authoriser) approval.stub(:stakeholders_as).and_return([stakeholder]) approval.authoriser.should == authoriser end end

It's true that tests act as a form of specification therefore should be optimised for clarity, in this case however, we could still maintain the clarity with significantly reduced duplication:

describe ApprovalStakeholder do let(:stakeholder) do create(:approval_stakeholder, :approval => approval, :user_id => user.id ) end subject { approval } before do stakeholder.stub(:user).and_return(user) approval.stub(:stakeholders_as).and_return([stakeholder]) end describe "#traveller" do let(:user) { traveller } its(:traveller) { should == traveller } end describe "#authoriser" do let(:user) { authoriser } its(:authoriser) { should == authoriser } end end

Do Not Replicate Implementation Details

I am often surprised to see many seasoned developers "enjoy" writing tests that essentially replicate the production code logic without much benefit. See below:

describe ApprovalStakeholder do it "references a user" do approval_stakeholder = build :approval_stakeholder, :user_id => 1 User.should_receive(:find).with(1) approval_stakeholder.user end it "references a role" do approval_stakeholder = build :approval_stakeholder, :role_id => 1 Role.should_receive(:find).with(1) approval_stakeholder.role end end

Rather than creating noisy tests, tests with actual assertions seem much more meaningful and readable:

describe ApprovalStakeholder do subject do build(:approval_stakeholder, :approval => approval, :user_id => user.id, :role_id => role.id, ) end its(:name) { should == "#{user.first_name} #{user.last_name}" } its(:role_name) { should == role.name } end

Reduce the Reliance on Mocks and Stubs

This is a difficult and often-debated subject. In my experience, having too many mocks and stubs even though speeds up the test suite, usually leaves too many holes in your tests and makes the test suite less accurate and effective. Fortunately, by using more service objects (described below), mocking and stubbing become more manageable as you use them mostly on external objects and interfaces.

Take Apart the System, One Service at a Time

If you're a Rails developer, you are already familiar with MVC. But just relying on MVC to hold your application architecture is probably not going to be sufficient for an average modern day web application. Many people like Service-oriented architecture, so do I.

Services are unassociated, loosely coupled units of functionality that are self-contained.

In my experience, as long as you are disciplined in having services do one and only one thing really well, testing becomes much easier.

For instance, we have a Bouncer service that is responsible for safeguarding resources - ensuring read-only attributes don't get overridden.

module Services class Bouncer def self.guard(resource, options = {}) if options[:existing_resource] resource.readonly_attributes.each do |attr_name| resource.send("#{attr_name}=", options[:existing_resource].send(attr_name)) end end resource end end end

The corresponding tests for this service are both fast and self-contained:

describe Services::Bouncer do class BouncerDude include Mos::Entity set_readonly_attributes :age, :gender attribute :name attribute :age attribute :gender end let(:resource) { BouncerDude.new(name: 'Penny', age: 28, gender: 'female') } let(:existing_resource) { BouncerDude.new(name: 'Sheldon Cooper', age: 34, gender: 'male') } subject { Services::Bouncer.guard(resource, existing_resource: existing_resource) } describe "#guard" do its(:name) { should == 'Penny' } its(:age) { should == 34 } its(:gender) { should == 'male' } end end

Recognise Common Patterns and Refactor Them into Services

One of the reasons why service-oriented architecture is so popular is because things are broken down into smaller, more manageable and more testable pieces. It is especially helpful for TDD practitioners as it significantly reduces the amount of coupling between your production code and your tests due to having simpler internals per test subject.

Take a look at the below example, which is hard to read, hard to test and error-prone:

module ApplicationHelper def branch_logo_options(branch) BranchLogo.where(branch_id: branch.id).map { |logo| [logo.file, logo.id] } end def branch_options(agency) BranchRepository.find(agency_id: agency.id, archived: false).map do |b| [b.name, b.id] end end def agency_user_options(agency, filtered_users) filtered_user_ids = filtered_users.compact.map(&:id) || [] AgencyUserRepository.find(agency_id: agency.id, archived: false).select do |u| !filtered_user_ids.include?(u.id) end.map { |u| [u.full_name, u.id] } end def current_agency_user_options(filtered_users = []) agency_user_options(current_agency, filtered_users) end def current_agency_trust_bank_account_options BankAccountRepository.find( agency_id: current_agency.id, archived: false, account_type: BankAccount::TRUST_ACCOUNT).map do |b| [b.account_name, b.id] end end def code_options_for(klass) klass.all.map { |cc| ["#{cc.code} - #{cc.name}", cc.id] }.sort end end

Let's refactor it into something more manageable, by introducing a service ShowGirl for fetching and presenting data collections:

module CollectionOptionsHelper def branch_logo_options(branch) Services::ShowGirl.present(branch, from: BranchLogo, show: :file) end def branch_options Services::ShowGirl.present(current_agency, from: BranchRepository) end def consultant_options(excluded_users = []) Services::ShowGirl.present( current_agency, from: AgencyUserRepository, show: :full_name ) do |collection| collection.reject { |user| user.id.in?(Array.wrap(excluded_users).map(&:id)) } end end def trust_bank_account_options Services::ShowGirl.present( current_agency, from: BankAccountRepository, show: :account_name, filters: { account_type: BankAccount::TRUST_ACCOUNT }, ) end def code_options_for(name) Services::ShowGirl.present( current_agency, from: Admin::Configurations::Essential.descendants.find { |d| d.name =~ /::#{name.to_s.classify}/ }, show: -> (item) { "#{item.code} - #{item.name}" } ) end end

Better yet, we can clean it up even further by introducing another service, BusBoy for just serving the data, and leaving ShowGirl for only presenting the data:

module CollectionOptionsHelper def branch_logo_options(branch) Services::ShowGirl.present( Services::BusBoy.serve(:branch_logos, branch: branch) ) end def branch_options Services::ShowGirl.present( Services::BusBoy.serve(:branches, agency: current_agency) ) end def consultant_options(excluded_users = []) Services::ShowGirl.present( Services::BusBoy.serve(:consultants, agency: current_agency), show: :full_name ) do |collection| collection.reject { |user| user.id.in?(Array.wrap(excluded_users).map(&:id)) } end end def trust_bank_account_options(account_type) Services::ShowGirl.present( Services::BusBoy.serve(:bank_accounts, { agency: current_agency, BankAccount::TRUST_ACCOUNT } ), show: :account_name ) end def code_options_for(name, options = {}) Services::ShowGirl.present( Services::BusBoy.serve(name, agency: current_agency), options ) end end

Basic Controller CRUD Actions

In one of our projects we have lots and lots of forms. Consequently we have lots and lots of CRUD actions. In order to keep our sanity as well as to make basic CRUD controllers maintainable, we have a custom DSL to make CRUD actions portable and testable:

module Profiles class TravellersController < BaseController authorize_resource class: Traveller datamappify_resources entity: Traveller, repository: TravellerRepository, filter_by: :agency_id, filter_value: -> { current_user.agency_id } end end

Most of our controller tests look like this:

require 'spec_helper' describe Profiles::AccountsController do let(:existing_resources) { [] } let(:create_resource) { Mos::Data.create_account } let(:create_resources) { Mos::Data.create_accounts(2) } let(:a_resource) { assigns(:resource) } let(:invalid_param) { { name: '' } } let(:params_key) { :account } let(:redirect_path) { profiles_accounts_path } it_behaves_like 'datamappify resources controller' it_behaves_like 'searchable resources controller', :name, :profile_id, :branch_id, :activated describe "permission" do context 'as a manager' do before do sign_in_as :manager end it_behaves_like 'with write access' it_behaves_like 'with read access' it_behaves_like 'with index access' end context 'as a consultant' do before do sign_in_as :consultant end it_behaves_like 'without write access' it_behaves_like 'with read access' it_behaves_like 'with index access' end end end

API Endpoint Tests

One of our projects at work is an API service that is essential to our platform. Naturally, we not only need to test the models, services and controllers, we also need to ensure the API endpoints do what they are supposed to do - mostly exposing the correct data structure.

During the early stage of the development, I had come up with ApiTaster - a super useful gem for visually testing our Rails application's APIs. Later on, as we continued to grow our API endpoints, we started utilising ApiTaster for our automated test suite too.

In essence, we have one API spec file responsible for describing which endpoints are tested and missed according to the information given by ApiTaster:

describe "API" do load 'db/seeds.rb' load 'spec/api_endpoints.rb' ApiTaster::Route.map_routes ApiTaster::Route.defined_definitions.each do |route| it "api endpoint #{route[:verb]} #{route[:path]}" do params = ApiTaster::Route.params_for(route).first expectation = ApiTaster::Route.metadata_for(route)[:expectation] setup = ApiTaster::Route.metadata_for(route)[:setup] verb = route[:verb].downcase path = parse_path_with_url_params(route[:path], params[:url_params]) setup.call if setup send verb, path, params[:post_params] response.body.should match_json_expression(expectation) end end # warn about undefined definitions ApiTaster::Route.missing_definitions.each do |route| pending "api endpoint #{route[:verb]} #{route[:path]}" end end

Then, we have a bunch of endpoint test files to do the actual testing, like this:

resource_response = ResponseHash[ :response => { :id => Integer, :name => String, :token => String } ] get '/:version/company', {}, { :expectation => resource_response } post '/:version/companies', { :model => FactoryGirl.attributes_for(:company) }, { :expectation => resource_response } put '/:version/companies/:id', { :id => 1, :model => { :name => 'New Company' } }, { :expectation => resource_response.with(:name => 'New Company') } delete '/:version/companies/:id', { :id => 1 }, { :expectation => resource_response }

Notice that for API endpoint tests we don't test the business logic or data integrity - these should be tested in models, services and controllers. What we do test are correct endpoints are exposed, correct parameters are accepted and correct data structures are returned.

Isolated JavaScript Tests

Many developers prefer to rely on their integration test suite to do JavaScript / UI testing. This approach is fine until you start making lots of front-end changes and constantly need to pinpoint the relevant feature spec.

Having an isolated JavaScript test suite (which should be run as part of your continuous integration process) is extremely beneficial and often saves debugging time.

I like Mocha so we use Konacha in our Rails app. Though Mocha with Chai is really not that different to Jasmine.

Custom JavaScript behaviour is obviously a good candidate for isolated testing:

#= require spec_helper describe "form toggle", -> beforeEach -> $("body").append(JST["templates/form/toggle"]) it "hides the collapsible field by default", -> $(".control-group.branch_deactivation_date").hasClass('in').should.be.false it "does not override if there is already a value", -> value = $("input#agency_deactivation_date").val() $("input#agency_activated").click() $("input#agency_deactivation_date").val().should.equal(value)

Sometimes it's also useful to ensure library code is initiated and triggered correctly, if you have other custom JS interact with it:

#= require spec_helper #= require bootstrap-datepicker describe "form dates", -> beforeEach -> @dateFormat = 'DD/MM/YYYY' $("body").append(JST["templates/form/dates"](dateFormat: @dateFormat)) it "has a placeholder", -> $("input").attr("placeholder").should.equal(@dateFormat) it "defaults to today's date", -> $("input#empty").focus() $("input#empty").focus() $("input#empty").val().should.equal(moment().format(@dateFormat)) it "does not override if there is already a value", -> value = $("input#filled").val() $("input#filled").focus() $("input#filled").val().should.equal(value)

"Real" UI Tests

Isolated JavaScript tests are super fast and useful. However, there are times when having pure JavaScript tests simply isn't enough, due to the complicated nature of DOM interaction and template rendering.

A while ago our calendar widget was broken due to a production and UAT environment issue that was not picked up by our JavaScript test suite. Since then we started adding dedicated UI tests in our acceptance test suite (we use Turnip):

@ui Feature: UI Background: Given I am signed in And I go to agency consultants page And I click on "Add New Consultant" Scenario: Calendar When I click "#agency_user_start_date" And I click ".day.active" within ".datepicker" Then I should see today as part of the date field

Effective Acceptance Tests

Writing acceptance tests - also known to many Rubyists as "Cucumber tests", is a double-edged sword - it's extremely useful, but very few developers can write good, maintainable Gherkin-style acceptance tests.

Here's an example of a badly written feature spec with too much implementation details and noise:

Feature: Session Background: Given I visit "/" And there is a user "admin" "password" Scenario: Sign in with valid credentials When I fill in "Username" with "admin" And I fill in "Password" with "password" And I click "Sign In" Then I should be on "/dashboard" Scenario: Sign in with invalid credentials When I fill in "Username" with "admin" And I fill in "Password" with "invalid_password" And I click "Sign In" Then I should not be on "/dashboard" Scenario: Sign out When I fill in "Username" with "admin" And I fill in "Password" with "password" And I click "Sign In" And I click "Sign Out" Then I should be on "/sign_in"

A much cleaner version with only high level, descriptive steps:

Feature: Session Background: Given I am on the homepage And there is a user "admin" with password "password" Scenario: Sign in with valid credentials When I sign in as "admin" with password "password" Then I should be signed in Scenario: Sign in with invalid credentials When I sign in as "admin" with password "invalid_password" Then I should not be signed in Scenario: Sign out Given I am signed in as "admin" with password "password" When I sign out Then I should be signed out

Final Thoughts

Writing good, sensible tests is hard. These examples and tips are by no means the silver bullet, and you might actually find some of them counter-intuitive in your particular situation. So again, take what you get, practise, and reflect on your findings. For Happiness! :)

Not good:

Post.join(:authors) # or Post.join(Author)

Better:

Post.join(:authors, :id => :author_id)

You would think the version above works - it doesn't. Even worse, the above example will give you incorrect data - the IDs of the Post records will now contain the IDs from their corresponding Author records! This is because upon a join, Sequel merges attributes from both models into a single hash.

The correct version:

Post.join(:authors, :id => :author_id).select(:posts__id, :posts__title, :posts__body) # or Post.join(:authors, :id => :author_id).select_all(:posts)

Gotcha 2: Always call "all" at the end of the chain, or the chain will present data in a different format.

In ActiveRecord, all of the below examples return an ActiveRecord::Relation collection:

Post.where(:title => 'Hello world') Post.joins(:author) Post.includes(:author)

And indeed, calling first on any of them returns an object of class Post (assuming the result collection is not empty).

Post.where(:title => 'Hello world').first.class #=> Post Post.joins(:author).first.class #=> Post Post.includes(:author).first.class #=> Post

In Sequel, the below examples all return a Sequel::DataSet collection:

Post.where(:title => 'Hello world') Post.eager(:author) Post.eager_graph(:author)

But let's see what we get from calling first.class on them:

Post.where(:title => 'Hello world').first.class #=> Post Post.eager(:author).first.class #=> Post Post.eager_graph(:author).first.class #=> Hash

Huh? Last one is a Hash? It turns out, if you call all at the end of chains to convert them to Arrays, then the returned collections are consistent:

Post.where(:title => 'Hello world').all.first.class #=> Post Post.eager(:author).all.first.class #=> Post Post.eager_graph(:author).all.first.class #=> Post