#systems languages

Go reminds me a lot of the late Zune HD.

The Zune was a great piece of technology that was engineered as a direct competitor to Apple’s iPod Touch. The screen was a gorgeous OLED, the interface ventured into flat design years before it was a fad, and under the hood, it had a surprising amount of power. I repeat: it was a great device. And yet, it never caught on.

Similarly, Go seems like a powerful alternative to Python, Ruby, and C++, and yet it doesn’t seem to have the same pervasiveness throughout the programming space. There could be a lot of reasons for this slow adoption (we’ll just leave Roger’s Diffusion of Innovations curve right here.)

Academic jokes aside, this really is a surprising state of affairs, since Golang is highly successful as a language on a number of fronts. As discussed in Rob Pike’s talk, the language is fairly readable for anybody with sufficient programming knowledge; it’s literally built to scale to massive levels - even Google; as a result, it’s highly suited to these scaling problems, and can interface with other languages; finally, Go has a single way of expressing most operations, meaning that tools like gofmt can easily format a document written in Golang, and there’s even interesting work in programmatically rewriting and modernizing old code.

Go is is an important new language that focuses on the society/community built around the product. Go fixes many problems: long compile time, complicated dependency management systems, etc. Disregarding the overhead of C files with files and libraries, Go attempts to make compile time and dependency management at setup SUPER easy! 

Some cool features of Go include their official formatting language. When I first heard of this, I immediately thought that Gofmt was yet another attempts at a widely adopted standard. Instead, this formatting eliminates any confusion rooted from different stylistic choices from each other. This reminds me of certain rules you can set up in the Eclipse IDE to ensure proper formatting of your code on save. This language alleviates the direness of not porperly documenting your code while you’re still working on it. It allows you to focus on the content than the formatting though sometimes the formatting helps you code sometimes!

The Go programming language was developed by Google, for Google.  It was purposefully designed to solve four key problems: readability, scalability, suitability, and toolability.  

Google’s biggest problem is scale - both performance wise and developer wise. Google’s projects have incredibly massive line of code counts and as such optimizing developer performance is actually a big deal.  Developers have to easily be able to look at new code and understand how it works (readability).  In addition, code formatting and tooling is incredibly important to be able to manage large code bases (toolability).  Because it was designed for Google, Go is somewhat like a programming language with a hint of domain-specific language thrown in.  By building a language that solves Google’s problems first and foremost, Go allows Google’s developers to be more productive with their time (suitability).

There are a few interesting design choices that have been pointed out in this week’s blog posts.  First is the syntax choices that differentiate Go from other programming languages such as the use of upper or lower case to denote public vs private scope.  Go also provides tools such as Goget, Gofix, and Gofmt to refactor code easily, even in large codebases.  There are also performance choices such as builtin concurrency in the form of goroutines.  Go also drastically reduces compile-time by reducing fan-out of code, the ratio of bytes of human-written Go code versus bytes sent to the compiler.

Go is a new programming language being developed by Google to solve the problem of scalability.  It’s goal isn’t to be flashy and have all kinds of cool tricks, it keeps things simple and tries to be as readable as possible.  All Go code is formatted automatically to ensure a coding standard, while it isn’t always the style people prefer it ensures the one goal of scalable code.  Every piece of Go looks the same.  Additionally, Go has adopted a few changes from the standard C family languages.  One notable aesthetic change is it uses variable case to distinguish public or private.  

Besides all the fluff around the syntax of Go, it is an incredible language for building massive distributed systems.  From it’s conception it has been designed to work for any size system.  It employs this in many of it’s basic libraries. One clear example of the Go engineers keeping this in mind is the way in which it handles variable names.  All names are unique and are obvious when using external packages.  This ensures that there is no confusion when including several packages and dependencies.  Moreover, when Go compiles, it will not relearn stuff it already knows.  What this means is in a normal C program you may include the String package in several different parts of the project which heavily slows down compilation, while Go recognize the redundancy and not fall into the trap.

Another unique aspect of Go is how it handles scope.  To avoid any confusion it simplifies all the hierarchy down to 4 basic steps: universe, package, file, and function. The engineers felt that this was essential for a distributed computing and scalable language to have.  It also helps with readability in that it’s clear whether a function or some code snippet has access to something it may need.

Another way that Go handles scalability is by allowing developers to easily change code, which can be a problem with larger projects which contain many lines. In these large projects, one change can have far reaching dependencies. Functions in Go, such as Goget and Gofix, handle this problem by using the AST in order to rewrite code simply, even if the changed code appears in millions of lines. An example that was mentioned was Google’s Protocol Buffers.

Go allows programmers to easily read programs and thus improves their efficiency and productivity, as well as handling problems inherent in large coding projects. This design objective was taken into account when designing Go’s memory management. It was observed that in programming languages such as C where garbage collection is done by the user, much of the user’s time was taken up by writing these lines. Go’s design prioritized programmer productivity. Even though the tradeoff is that the binary is slowed, taking memory management off the table by designing a robust garbage collection system greatly boosts a programmer’s output.  

Rob Pike from Google came to speak about the reasons and goals of the Go language, which are somewhat different from what you might traditionally expect from a new language.

Unsurprisingly, scalability is an important design goal, but it is in fact number two on the list! Number one is readability, which Pike maintains is one of the big reasons Go has become quickly successful. His talk focused on these four major design principles for Go, and how Go’s approach is different. 

Readability

Scalability

Suitability

Toolability

Readability

Go’s readability is an integral part of the language. Go syntax looks like standard C-compatible syntax for the most part, but it is actually quite different in many places. This makes Go much easier to figure out than other languages, because most programmers have experience with a brace-based language.

Pike also (rightly so, in our opinion!) was pretty dismissive of the idea of “clever one-liners” and code golf that exists in many environments or that many people enjoy writing. Because programs are more often read than written, Pike stressed that Go’s design was structured to help programmers write readable code.

Interestingly, Pike talked about how readability is important for other parts of the language. Readability improves scalability because code that is clear in what exactly is being done, is easier to understand how that code is being executed. The decisions that Pike discusses for toolability and suitability are also designed to improve readability of Go code.

Scalability

The goal in designing Go was to make a language that was scalable in multiple dimensions on the order of magnitude of the operations of a company the size of Google. The developers wanted to increase efficiency not only in the running of their code, but also it’s development.

At Google, thousands of engineers may be working on a project, pushing code as often as every few seconds. This development environment drove the push for a language that was more manageable in very large code bases. For example, Go compiler makes some interesting changes to improve compile times. The compiler attempts to reduce its workload by looking at dependencies and including the relevant code in object files while it works its way through the dependency tree. This way, it does not have to link the same file multiple times and can drastically reduce fan out.

The goal of increasing scalability is also intrinsically linked to readability. Pike and his colleagues hoped that improving the readability of code would help alleviate the issues behind maintaining massive amounts of code.

Suitability

Pike made the point that Go was developed for Google, by Google. Although Go introduces many new ideas and features that may be appealing to many programmers, it may not be suitable for all purposes. Go was developed specifically to meet the needs of developers at Google and does a very good job of filling those needs. It was not developed to be an all-purpose language that solves everyone’s problems by being readable and scalable. However, it does perhaps serve as an example of how new or existing languages can continue to develop to help solve these same issues in other companies and fields.

Toolability

Tooling is a big part of Go’s design from the get-go. (Sorry.) The idea here is that because tooling is necessity in most projects of a reasonable size, it should be built in to the the environment. The most basic example is called gofmt. This is an automatic stylizer for Go source code that standardizes all the basic stylistic components about a program. This means that all Go code seen in the wild will look the same, again improving readability

As described by Rob Pike, Go is a language built for software engineering. It is a language built to facilitate writing software for large systems. To this end, many of its features were designed just for that. gofmt, gofix, and smart dependency management are just some of many examples of ways that Go has been designed to increase developer productivity and help maintain four important properties: readability, scalability, suitability, and toolability.

One of Go’s biggest advantages over other languages is that it was built with concurrency in mind, providing first-class constructs to handle concurrency. There is only one language that handles concurrency as eloquently as Go: Scala. In many ways, Scala and Go are like yin and yang. Rob Pike intentionally kept Go simple like its spiritual ancestor C, which is useful when you have large codebases with many contributors, each with their own programming styles. On the other hand, Scala gives you much more syntactic sugar to play with. This means that Scala code can be much more concise than similar code in Go, but the ability to complete tasks in so many different ways can be a curse when many developers, each with their own styles, are working in the same codebase.

Although syntax is the most visible trait of any programming language, these philosophical differences can be found in the language implementation as well. On almost every issue that is central to language design, Scala and Go take different stances. At this time, Go does not have support for generics (although the developers are open to adding them if they can find a way to do so easily). Scala, which encourages functional programming, relies on generics heavily. Go is heavily typed, but Scala supports local type inference in some cases. Go does not support type inheritance at all, whereas Scala uses implementation inheritance.

As you can see, the creators of Go and Scala have very different ideas of what a language’s purpose is, and that has influenced the design of their respective languages. Which language is “better” will depend on the size of your organization, the problems you are trying to solve, and your personal philosophies on what a language should be.

In our last class, we heard from Rob Pike, a Google engineer who helped create—and currently works on­—the Go programming language. Go was released a few years ago, and has since received significant attention following its stable 1.0 release in 2012.

During his talk, Pike focused on the core design principles behind Go. In typical Google fashion, Go was conceived as a language willing to depart from established conventions for the sake of scalability. In this case, scalability meant a language could be used in projects with huge numbers of developers and high performance requirements. As a result, Pike’s focuses have been twofold: readability of code, and performance of the language. The Go team looked to occupy a sweet spot in code readability: make the syntax verbose enough that it’s immediately clear to a reader what the code does (unlike, say, Lisp), but not so verbose that it becomes cluttered and tedious to type (like C++). This led the Go team to introduce a few features in the language not seen in most other mainstream languages. First, there are syntactic elements: type declarations must follow the expression, for example. This is a simple change that improves the readability of complicated type declarations—an example taken from the Go website, for example:

int (*(*fp)(int (*)(int, int), int))(int, int)

becomes:

f func(func(int,int) int, int) func(int, int) int

There are also semantics: only capitalized names are exported from a module, and every import is namespaced.

Go also made some important strides in terms of performance. The most obvious (and extremely useful) feature is the concurrency support baked into the language. Whereas in other systems languages concurrency is an afterthought (because many of them were conceived well before the rise of multi-core processors), Go has the concept of goroutines, which are a kind of abstraction over threads, built into the language with the “go” keyword, which runs a function in another thread, and other keywords for joining and managing threads. The Go team also made some major improvements in order to dramatically cut down compile-time in large projects, most notably by reducing the fanout of Go code; that is, the ratio of bytes of human-written Go code versus bytes sent to the compiler. This was achieved by reducing code duplication amongst modules that import the same dependencies.

This week, Rob Pike talked to our class about Go, the programming language developed at Google. It was designed to help combat problems of scale which are frequently encountered at Google, due to a large number of both programs and data, as well as software engineers. Pike began by talking about Go’s features, which were made to address Google’s specific issues, but are also quintessential to many languages. In particular, Pike talked about the “abilities”, philosophies and ideas upon which Go was designed. The first was readability, which addresses Google’s problem of an overwhelmingly large codebase and employee base. That is, the language they developed had to be, more importantly than its ability to be written, able to be read easily and by as many people as possible. Go thus strives to maintain a “balance between clarity and redundancy”. Pike then talked about scalability for the massive amounts of data, computers, and code Google must compile and test daily. He then stressed suitability for which Pike noted that although Go is currently quite popular and used in many applications, its original purpose was to help Google’s software engineering, and thus is definitively not the panacea to the various problems which could plague users with different needs. Finally, he talked about “toolability”, or the ability of a language to have tools written for it, allowing users to write their own programs or fixes to address any problems they have with the language.

When we discuss scalability in the context of programming languages, we must consider several principal dimensions: cores, code, data, speed of compilation, speed of testing, and engineers. Go seeks to solve each of these scalability issues in a concerted effort that no other language has done. Being a product of Google, it is only natural that Go has consistent support for concurrency to take advantage of Google’s seemingly endless machine farms. To leverage the growing number of engineers at Google, Go was designed for large code bases with many developers pushing changes concurrently. For instance, consider dependency management in Go vs. C++. In C++, dependencies can grow exponentially and are often redundant. Go, on the other hand, hoists dependencies resulting in exponentially less data read than with C++’s includes. Because Go was designed with very specific requirements from its initial formation, its syntax has been optimized for readability, clarity, and adaptability, ultimately making it a very scalable language.

Scaling software usually involves concurrency and parallelism. These two things are similar, but they are far from being the same. While parallelism is dividing the work of one task between workers, concurrency is handling several different tasks at once. As Rob Pike put it, “Concurrency is about dealing with lots of things at once. Parallelism is about doing lots of things at once. Concurrency is about structure, parallelism is about execution.” Writing concurrent code is a matter of structural complexity, as the different processes have to be able to communicate with each other so that the code doesn’t crash. Implementation of concurrent programming is a relatively recent development, and as such, older languages like C and Java only have workarounds to it, which can make concurrent programming in those languages complicated. Go, on the other hand, was written with concurrent programming in mind, packing primitives and tools to accommodate it.

Functions in Go run as independent goroutines, which are like threads, but much less expensive because they live on the same address space as the other goroutines. Goroutines can communicate with each other through channels, for example, if you wanted to synchronize them somehow, a channel would allow you to block goroutines until others finish. Go is also able to act depending on which channels are ready to communicate through the use of the select statement, which is like switch, but for channels. With Go’s concurrency paradigms and built-in concurrency tools, designing concurrent code, as well as writing it, becomes much simpler.

Rust is another systems language that was recently developed by Mozilla Research and is aimed at being a memory safe, fast, and concurrent programming language. Rust is a lot newer than Go, and it’s design has a slightly different focus. Much of the design of Rust has been influenced by the shortcomings of previous systems languages like C, whose programs are often plagued with memory leaks. In terms of syntax, Rust resembles C with its braces and similar keywords; however, the language itself contains many semantic differences which make it a lot more flexible such as the explicit distinction between mutable and immutable variables as well as type inference so Rust’s compiler can automatically discern the type of certain variables.

Furthermore, in order to handle memory safety Rust has introduced a new concept of “ownership” which replaces historical models of allocating memory. Whereas in C the developer must keep track of allocating and deallocating memory, in Rust this is handled by creating an “owning handle” whenever memory is allocated. If this handle goes out of scope the memory is deallocated automatically. This means that unlike Go, Rust does not have a garbage collector built into the core language, which has spurred much debate over the performance/safety comparisons of both these languages.