/var/log/bhs

The folks at Hakka Labs were kind enough to record the GoSF talk I gave a few weeks ago. Here's the video:

http://www.hakkalabs.co/articles/distributed-systems-go-good-bad-otherwise#!

Here are the slides (with a few after-the-fact additions/edits):

For the N’th time, I found myself writing another custom unexported type with custom Len(), Swap(), and Less() methods just so I could get the golang sort.Sort() function to do something that should be trivial. And, as usual, I was cursing under my breath.

I’m sure every Go programmer has been in this situation – and I wouldn’t be surprised if others have even written and published equivalent code already – but I decided I couldn’t take it anymore and wrote a reflect-enabled helper that allows the programmer to sort a slice in-place just by providing a generic comparator function.

This is what I came up with:

package slicesorter import ( "reflect" "sort" ) // Sort a slice in-place given the provided less comparator function. // // For example: // // // (FWIW, sort.Ints(...) will not work with a []NotAnInt) // type NotAnInt int // s := []NotAnInt{3, 4, 8, 5, 2} // // slicesorter.SortSlice(s, func(l, r interface{}) bool { // return l.(NotAnInt) > r.(NotAnInt) // }) // // // Prints "[8, 5, 4, 3, 2]": // fmt.Println(s) // func SortSlice(inputSlice interface{}, lessFunc func(l, r interface{}) bool) { sortable := sortableSlice{inputSlice, lessFunc} sort.Sort(sortable) } // The internal type that binds everything together. type sortableSlice struct { inputSlice interface{} lessFunc func(l interface{}, r interface{}) bool } // A stupid helper. func (s sortableSlice) reflectElems(i, j int) (elemI, elemJ reflect.Value) { sliceVal := reflect.ValueOf(s.inputSlice) elemI = sliceVal.Index(i) elemJ = sliceVal.Index(j) return } /////////////////////////////// // The sort.Interface contract: /////////////////////////////// func (s sortableSlice) Len() int { return reflect.ValueOf(s.inputSlice).Len() } func (s sortableSlice) Swap(i, j int) { // Cue the preposterous reflection dance! elemI, elemJ := s.reflectElems(i, j) iInt := elemI.Interface() jInt := elemJ.Interface() // (Thankfully, Set() knows about slice indices) elemI.Set(reflect.ValueOf(jInt)) elemJ.Set(reflect.ValueOf(iInt)) } func (s sortableSlice) Less(i, j int) bool { elemI, elemJ := s.reflectElems(i, j) return s.lessFunc(elemI.Interface(), elemJ.Interface()) }

I put the code on github just in case anybody wants to use it:

https://github.com/bensigelman/slicesorter/blob/master/go/src/github.com/bensigelman/slicesorter/slice_sorter.go

I often switch off between distinct Go projects with distinct GOPATHs. While there were some bits and pieces of reasonable solutions on stack overflow, none fulfilled all of my requirements:

follows me around as my working directory changes

no need to remember a different name for the go toolchain meta-command

sets GOPATH properly even when the go command isn't running (this is important for the proper functioning of my vim setup, which relies heavily on the thoroughly excellent vim-go plugin)

allows for easy overriding if I need to explicitly set GOPATH from time to time.

This is what I added to my ~/.bash_profile:

export GOPATH="... a default path ..." function cd() { builtin cd $@ && export GOPATH="$( ( while [[ $PWD != / && $(basename $PWD) != go ]]; do cd .. done if [[ $PWD == / ]]; then echo $GOPATH else echo $PWD fi ))" }

Surely everyone who cares about social media has already heard about Facebook's grand psychology experiment and marveled at the backlash. After all, griping about Facebook has practically become a national pastime, and, over the years, Zuck and co have given us so much great material!

But in this case it's a little more complicated.

Every product in the world – anything from software to bicycles to breakfast cereal – can be assessed from a psychological perspective. Indeed, I'd argue that good products always are: they are successful because they play on a fundamental psychological need. In the best of cases, of course, this just means that they make us happy. In other situations, it's more cynical: e.g., marketers may play at our fears and insecurities in order to make a sale that alleviates our anxiety. But in any case, it all comes down to psychology.

Back to the matter at hand, Facebook claims that they pursued this experiment in order to make their users happier people (and happier with Facebook). From the paper itself (emphasis mine):

In an experiment with people who use Facebook, we test whether emotional contagion occurs outside of in-person interaction between individuals by reducing the amount of emotional content in the News Feed. When positive expressions were reduced, people produced fewer positive posts and more negative posts; when negative expressions were re- duced, the opposite pattern occurred. These results indicate that emotions expressed by others on Facebook influence our own emotions, constituting experimental evidence for massive-scale contagion via social networks.

And from their soundbite-ready press person:

We felt that it was important to investigate the common worry that seeing friends post positive content leads to people feeling negative or left out. At the same time, we were concerned that exposure to friends’ negativity might lead people to avoid visiting Facebook.

Why fault Facebook for trying to understand and, allegedly, improve the psychological state of its userbase?

Because apparently they still don't grok the basic psychological underpinnings of their flagship product.

There are numerous (one, two, three, four, among many others) research articles that illustrate how people don't feel happy when they read about their friends seeming happy on Facebook. In fact, most people read these things when they're alone, so they typically feel isolated, envious, and that the lives of others are richer than their own.

And yet! Facebook continues to focus on posting behavior as the chief indicator of user happiness, even though we know, per the above, that it's a paradoxical psychological signal. There's a more nuanced – and, in my view, more compelling – explanation for why people post more "positive" (sic) things on Facebook when they're exposed to more "positive" items in their respective news feeds: because they don't want to seem like they've been left out. Facebook is all about FOMO: why would it be any different for this study?

Everyone knows that hiring is crucial for a tech startup. Everyone also knows that it's really difficult: partly because good candidates have lots of choices in today's tech labor market, but also because it's sometimes hard to separate the good candidates from the ones that just look like good candidates on paper. This is why we interview people, of course, but anyone who's administered a technical interview knows how hard it can be. Which questions should you ask? How many? How difficult? What topics are fair game? How do you normalize against other candidates? Or against the ideal answer? Interviewing is difficult.

Prologue: the state of tech interviews today (and how we got here)

Google's eng interview process has been emulated by much of the tech world, but it's showing signs of its age. Hiring there spun up just after the 2001 tech crash, and Google was one of the only desirable tech companies that was even growing at the time. As such, there were hordes of skilled candidates eager to work for them, and Google could afford to live with false negatives (i.e., "no-hire" decisions for candidates they should have hired) as long as there weren't many false positives (i.e., "hire" decisions for candidates they should have passed on). Their process still reflects that bias to this day, even as the recruiting climate has changed: Google is no longer the sexiest (or the only!) place to get a job, there are countless companies all trying to hire as quickly as they can find good candidates, and the pool of desirable engineers is not growing quickly enough to match that demand.

So, briefly, false negatives may have been okay in 2003, but they're not okay in 2013. False positives are just as dangerous as ever, though! What can you do to put a dent in false negatives without opening the door to a bunch of B/B- candidates who will gradually erode your company's engineering culture?

I've interviewed many hundreds of candidates by now, and I also spent many years on hiring committees at Google. I've noticed which sorts of feedback and which interview tactics led to good – and bad – hiring decisions. As part of my advisory role at bay-area startup, I recently wrote a 1-pager that condensed the most important things I've learned about effective engineering interviewing over the years. With their permission, I've included it below (with a few tweaks and modifications). I'd love to hear what others think, or which commandments I've left off. Without further ado:

The Ten Commandments of Engineering Interviews

Always treat the candidate like the guest of honor. Even if you don’t happen to like them. Even if they can’t write a for loop. They need to walk out the door feeling as though they were respected, and they need to tell their [possibly employable!] engineering friends that your company is filled with bright, conscientious people.

Ask for whiteboard coding as a last resort. Coding is of paramount importance, but even great programmers can be sloppy on a whiteboard. Talking through algorithms is fine, but trying to write compilable code on a whiteboard is questionable. [I'll make a future post about how I like to evaluate coding skills before making a job offer; suffice it to say, though, that whiteboard coding leads to plentiful false negatives.]

Avoid questions that only have a single subtle answer. These are sometimes referred to as “puzzle” questions (“if you have a rope that takes 10 minutes to burn from end to end, …”, that sort of thing). Answering them correctly suggests true intelligence, but missing them says little about a candidate. I.e., they generate false negatives. Also, good candidates who miss questions like these will be frustrated; see Commandment 1.

Don’t evaluate via trivia questions. A great candidate who gets hired will be able to absorb trivia easily on the job. Testing trivia in an interview is either a poor use of time or a false negative. (E.g., language minutia and/or the ins and outs of various frameworks and libraries)

Coordinate with other interviewers about topics. A good end-to-end interview process covers coding skills, system design, communication, previous work experience, team dynamics, product sense, and personality fit. Coordinate with previous interviewers about topics using some sort of simple, foolproof system (a paper interview sheet that's passed from interviewer to interviewer, for instance). That said, do not discuss your actual assessment with another interviewer until you’ve both written them down: no need to add bias to the process!

Beware of personality issues. Candidates who seem even a little stubborn or combative in an interview setting are often unbearably so in a professional context. Nervousness is another story, of course, but trust your gut. If you don’t like the idea of working on a long, two-person project with the candidate, that’s a red flag. Even if they’re brilliant.

Educate the candidate about the job content. Make sure you sell your company for what it is: [hopefully] a great place to work with excellent people, but not everything to everybody. If a candidate wouldn’t like working there, they should be able to discover that during the interview process (even if you and your company likes them). And if they would like working at your company, the specifics will make you that much more appealing. Look for a great fit, not just a great candidate.

Understand the candidate’s natural role in a group. Try to determine whether the candidate likes to lead, likes to sit in a dark room and code, likes to focus on product, likes infrastructure, and so on. Understand which roles they prefer, which they’re somewhat comfortable with, and which don’t work for them at all.

Never be intimidated by a resume. Titles and prior experiences can be impressive, especially for candidates who’ve been in industry for several decades (and interviewers who have not). Be respectful of accomplishments, but only do a “sell” interview if you’re asked to. Sometimes companies promote the wrong people, and sometimes individual resumes take credit for the work of entire teams.

Several months ago while doing engineering work at a startup, I ran into a problem that demanded an efficient backend that could scale up/out/etc. The existing implementation was part of a Rails app, and we couldn’t reasonably grow within that framework. The company had almost no non-Rails code, so a larger question arose about which language we should use.

While at Google, I spent the better part of a decade building distributed systems from scratch. Those who’ve worked at Google would probably agree that libraries and systems slow development down more than languages per se, but – that said – most of the actual code I or my teams wrote was in C++, typically with bits and pieces of Java and python around the edges. Since Google, I have been [indecently?] exposed to the contemporary menagerie of dynamic languages (ruby, javascript, etc). Somehow, though, none of the above felt right for this new project.

C++: too much rope, hard to maintain, painful to introduce at a company with no prior C++ footprint, frightens junior devs who no longer absorb relevant memory management idioms in school

Java: too verbose, too many FactoryFactories, painful to tune

Python/Ruby/Javascript: my experience is that large systems are difficult to maintain in these languages, as the dynamic typing makes it difficult to refactor large codebases without introducing errors in poorly tested areas of a repository

Rails in particular: Independent of ruby, I see Rails as the emperor with no clothes on. A subject for another post, but I will try my damndest to steer clear of it in the future.

At this point, Go became a viable contender. However, when I was doing my initial research about it, I had trouble finding high-level impressions from recent converts that (a) sounded like they were written by an author with substantial experiences building systems in other languages, and (b) didn’t seem zealous. This post is my attempt to provide the sort of overview I would have valued reading before I started my own investigation into the language.

At the time of this writing, the subsystem I described at the beginning of this post has been written (in Go) and running for many months without issue. It took about two weeks to learn the language, write the necessary code, and deploy it in the startup’s production environment on EC2. The short story is that I’ve been mostly delighted so far. What follows are my impressions.

Good things:

Go feels like a “small" language. It seemed as though I’d absorbed the core language and builtins – with the exception of the reflection APIs, which I haven’t had occasion to use yet – in about a week. (I of course have plenty to learn about the standard libraries that build on that foundation, but that’s to be expected) Contrast this with contemporary C++, Java, or Scala, all of which have plentiful innate complexity.

Goroutines and channels: where have you been all my life? I’ve never had so much fun or spent so little time reading about edge cases while implementing a thread-safe service that uses multiple cores, allows for multiple concurrent readers while excluding writers, etc, etc. Of course we all know how to build these things in other languages, but it’s such a joy that they’re made simple in Go.

Go Interfaces: where have you been all my life? This is the type system I always wanted… it feels a little bit like using python/ruby/javascript/etc in that Go interfaces resemble duck typing, except that in this case there’s a spec for the duck. This helps from a readability/maintainability standpoint, of course, but it also means that the compiler can tell you if you’ve screwed up and fat-fingered a variable name somewhere (rather than finding out at runtime, as is the case with many popular dynamic languages (yes, I know, linters can help here — but the Go compiler is better at it and can’t be forgotten or avoided by lazy team members!)).

`go fmt` ends absurd style debates once and for all. There is a single sanctioned Go style, and `go fmt` (which integrates easily into the editor of your choice) enforces it on any legal go source file. I also appreciate how the formatting tool borrows from the “elastic tabstops" demo (http://nickgravgaard.com/elastictabstops/), though that’s a minor point.

The go toolchain encourages you to organize source files in a sensible manner, as well as external (third-party) dependencies, etc. I wish it didn’t rely on environment variables for critical paths, but that can be hidden via scripts/etc.

Compiling down to a single statically-linked binary makes me feel safe and secure, like I’m back in the warm embrace of Google’s internal systems. It also means that the “deployment yak" is several steps closer… (esp compared with deploying modern Java applications, which is a minor nightmare IMO)

For what it’s worth, my new Go service is fast (fast enough that I have a huge amount of headroom, anyway)… Obviously this undocumented assertion of mine is not equivalent to reporting benchmark results, but suffice it to say that Go passed the sniff test with flying colors (and absolutely blew the doors off of a previous ruby implementation of an analogous service, though that is hardly grounds for bragging!).

Bad things:

The standard library is missing some things I consider essential. I’m thinking in particular about some manner of sorted associative container. There are several open-source projects that provide mature implementations of RBTrees, etc, but they’re missing something important… see the next point:

It’s not true that Go doesn’t have generic types: it does, you just can’t write them yourself. This omission has been beaten to death on the Go mailing list, but it still drives me nuts, especially since there are some core container data types that are missing, leaving the user casting and recasting and recasting as objects enter and leave containers.

Especially while I was in the early learning stages, I was disappointed with the amount of troubleshooting content on the web (StackOverflow threads, mailing lists, etc).

There are a few odds and ends that don’t quite fit right: the `make` builtin feels like a hack, especially since it doesn’t apply to user-defined types; the rules about when you have to derefence pointers and when the language does it for you are still hazy around the edge cases; I think the logging library is too restrictive; and probably other details I’m forgetting. C’est la vie.

I am mostly enamored of Go as a language. Someday I'll devote a post to all of the things I admire about it, but at the moment I'd rather complain about the way the range keyword interacts with slices. The semantics are almost inconceivable to me.

Here's how range works with hashmaps:

m := map[int]string{2: "two", 5: "five", 8: "eight"} for key, value := range m { fmt.Printf("%v: %v\n", key, value) } // This prints // // 2: two // 5: five // 8: eight

Okay, great. How about with a slice of ints?

s := []int{1, 1, 2, 3, 5} // This next line is the crux: for element := range s { fmt.Printf("%v\n", element) } // This prints // // 0 // 1 // 2 // 3 // 4

The range keyword assigns the slice index to element, not the value at that index. Of course, one can write

s := []int{1, 1, 2, 3, 5} // This next line is the crux: for _, element := range s { fmt.Printf("%v\n", element) } // This prints // // 1 // 1 // 2 // 3 // 5

Oh dear... so that's the best practice? Why not this?

... for element, _ := range s { ...

Either of these tuple-valued return values for range would be fine with me, really, as long as it was a compile-time error if the programmer only had a single argument on the LHS of the assignment. But if the single-argument syntax is supported, shouldn't it be the value rather than the index?

In my [recent post](http://blog.bensigelman.org/post/56158760736/golang-impressions) detailing my impressions of Go (the language), I took the following swipe at Rails: > Independent of ruby, I see Rails as the emperor with no clothes on. A subject for another post, but I will try my damndest to steer clear of it in the future. I was called out [on Hacker News](https://news.ycombinator.com/item?id=6090549) for being so vague, and rightly so. This post is my attempt to be fair, objective, and, by consequence, unrelentingly negative about Rails :) Of course I don't literally mean what I'm saying in the title of this post. Rails has its strengths and makes it easy for a total newcomer to glue things together and build an interactive webapp. The reason I think Rails is so dangerous is that it's like balsa wood: very pleasant to work with, but ultimately not solid, and – unlike a service written in any number of other frameworks – tangible performance concerns appear when there are tens of *thousands* of users, not tens of *millions* of users. That is, even small services run into performance problems if they put pressure on the framework (e.g., services for social content or other domains that involve a lot of churn or interactivity).

Edit: this post is getting more attention that I was expecting – great! However, people are not parsing the argument on HN (shocking!), so I will lay it out more succinctly.

Do you need performance? If so, don't use Rails.

You don't need performance. Do you need to maintain this software long-term? If so, don't use Rails.

You don't need performance and you don't need to maintain your software. There are countless other web frameworks written on top of dynamic languages which offer comparable benefits to Rails, yet have favorable performance profiles and a better culture of compatibility. Start there.

Well-Intentioned Developer: "I need to develop a webapp and/or a REST API, and everyone and their dog seems to use Rails. I should, too, right?"

No; Rails is fundamentally – and catastrophically – slow.

This well-known set of webapp benchmarks shows that, for a single-database-query workload, Rails offers less than 1/20th the throughput of, say, Go, 25 times the average latency of Go, and more than 6 times the latency variance of Go. And Go isn't even the most responsive or highest-throughput of the available options. The story is even more alarming for elemental workloads that focus exclusively on the framework internals.

And it's no wonder. Until Rails 4, the core framework didn't enable concurrent request handling by default. And even in Rails 4, concurrent request handling is multiplexed on a single CPU core, leaving Rails unable to take advantage of the parallelism of modern architectures. Don't mistake concurrency for parallelism: any web framework that cannot take advantage of multiple CPU cores will suffer greatly from a throughput standpoint.

At this point it would be reasonable to bring up JRuby. I should admit at this point that I've never used JRuby seriously, as the Rails apps I worked on had dependencies on C libraries which of course eliminated JRuby. Nevertheless, a few thoughts:

Deploying JRuby means entering the dark world of JVM tuning: one of the reasons people want to use frameworks like Rails is to get away from JVM tuning, so it's unfortunate to drag it back into the picture.

That said, JRuby sometimes wildly outperforms cRuby on CPU benchmarks... yet it also sometimes wildly underperforms cRuby for those same benchmarks: read more here.

Although JRuby does allow for parallel processing of requests across CPUs, some gems often (unfortunately) still make the simplifying assumption that there's only ever a single extant request in the Rails process, and thus they are not safe for use in a concurrent environment. One can avoid those gems once identified, but my experience was to find out about them the hard way. Hopefully Rails 4's concurrent-by-default mode will put more pressure on gem maintainers, but this is a rocky place to start from. The Rails community hasn't been thinking about concurrency (much less parallelism) for many years running, and it's a difficult cultural corner to turn.

Independent of JRuby, the best way to make use of a modern multi-core server in Rails is to run a single instance [of Rails] on each CPU core. Then one factors any large caches out of those Rails processes (via memcached or similar) and – ideally – places them on the same machine in order to make proper use of local memory. (That said, it seems that most people run the cache servers "wherever", which is obviously problematic from a latency perspective)

Compare that to a system design where the web framework can use all available CPU cores and use a single in-memory cache; latency is now reduced to the time it takes to grab a read lock and fetch from main memory. It's also a simpler programming model since there are no potential cache-fetch RPC failures to contend with.

Well-Intentioned Developer: "Huh. Well, I am more interested in time-to-market than serving latency, and dynamic languages (and DSLs in those languages) allow me to work more rapidly."

If you never write bugs or refactor your code, you're probably right. Writing code in Ruby and Python and Javascript is fun; sometimes, magically so. Especially when complex ideas can be compressed into a single list comprehension or lambda application.

However, modern dynamic languages (and their incapacity to do the sort of meaningful pre-runtime verification of basic semantic well-being one expects from a compiler) place a heavy burden on test coverage. Of course it is essential in any language to provide test coverage for core algorithms and other subtle aspects of a software module. However, when trying to "move fast" and get to market quickly, one shouldn't have to write tests for every souped-up accessor method or trivial transformation. In a dynamic language, if you don't actually have test coverage for every line, you'll run the risk of trouble with "types and typos." Either because, e.g., you thought you were accepting a list of strings, but some caller accidentally passes in a single string without the []s, or because you fat-fingered a variable name in a rarely-taken conditional branch, or because you were expecting a duck but got a chicken. Aside from not making these [natural] mistakes in the first place, the only cure in modern dynamic languages is utterly comprehensive testing.

These are the situations that concern me in dynamic languages:

u = User.find(...) if almost_always_true(u) u.dynamic_method_name(...) else u.misppeled_dynamic_method_name(...) end

If your tests don't cover that else case, you won't find out about your typo until your code runs the rarely-taken else block in production. These are the sorts of errors that compilers are great at discovering in languages with stronger typing (and of course there are countless such languages: I'm not picking any one in particular here).

In dynamic languages, then, one either spends much of one's dev time writing tests to cover things a compiler could otherwise check for you, XOR one is contending with a long-term maintenance problem.

Well-Intentioned Developer: "well, you could say the same thing about Node.js, Django, or any other framework built on a modern dynamic language. I love dynamic languages, and I am willing to accept the dev cost for the tests (or the maintenance costs for the lack of tests)."

Okay, fine. You still shouldn't be using Rails. Ignoring the general points about maintenance of software written in dynamic languages, you'd be better of with Node.js or Django (or numerous other possibilities). They have both supported concurrent request handling for a while. As such, the communities around those frameworks have written libraries which also work well in a concurrent environment. Node.js is even designed to discourage blocking calls in general, and – thanks to that design decision and the attention paid to javascript runtime performance – it does about as much with a single CPU core as one could hope for in a dynamic language. (ed: okay, okay, HN: Node.js is not fully baked yet. It must be admitted!)

Furthermore, the tools available for profiling Ruby are either wildly inaccurate or have such a profound effect on performance as to be misleading. And trying to make software efficient without a profiler is foolish.

Well-Intentioned Developer: "But hiring is so difficult already, and most of the developers I encounter are familiar with Rails. Aren't people the thing we should optimize for?"

A great engineer will have no trouble learning a new framework. In fact, a great engineer will be excited to learn a new framework. And you should only ever hire great engineers.

Well-Intentioned Developer: "When would Rails be appropriate, then?"