#reborg

This article is excerpted from chapter 1 of the "Clojure Standard Library, An Annotated reference", the Manning book I'm currently working on.

; TLDR

The post shows you why it's important to dedicate time to learn the content of the Clojure standard library. The standard library contains loads of interesting functions and knowing them often leads to a better functional design. An example is given to show how a simple program can be improved by learning what you have available.

The book is available from Manning website along with a few sample chapters.

Tools

Software development is often compared to a craft, despite the fact that it's predominantly an intellectual activity. While software development is abstract in nature there are many craft-oriented aspects to it:

The keyboard requires time and dedication to operate correctly. There are endless discussions on the best keyboard layout for programmers, for example to speed up typing (Dvorak users often claim huge benefits compared to QWERTY users. Here's one comparison, including other kind of layouts: http://lifehacker.com/should-i-use-an-alternative-keyboard-layout-like-dvorak-1447772004).

The development environment is a key aspect of programmers productivity and another source of debate (almost reaching a religious connotation). Mastering a development environment often translates into learning useful key combinations and ways to customize the most common operations.

Libraries, tools and idioms surrounding the language. Almost everything above the pure syntax rules.

Proficiency in several programming languages is definitely a plus in the job marketplace and the way to achieve it is by practicing them on a regular basis including getting familiar with APIs and libraries the language offers.

Many other aspects require specific skills depending on the area of application: teaching, presenting or leadership.

The focus on mastering programming skills is so important that it became one of the key objectives of the Software Craftsmanship Movement. Software Craftsmanship advocates learning through practice and promotes an apprenticeship process similar to other professions.

The standard library is definitely one of the most important tools to master a language. One aspect that characterizes the standard library is the fact that it is already packaged with a language when you first experiment with it. Interestingly, it doesn't get the amount of attention you would expect for such an easy to reach tool.

Why should I care about the Standard Library?

The expressiveness of a language is often described as the speed at which ideas can be translated into working software. Part of the expressiveness comes from the language itself in terms of syntax, but another fundamental part comes from the standard library which is usually provided out of the box. A good standard library liberates the programmer from the most mundane tasks like connecting to data sources, parsing XML, dealing with numbers and a lot more. When the standard library does a good job, developers are free to concentrate on core business aspects of an application, boosting productivity and return of investment.

Consider also that a deep knowledge of the standard library is often what distinguish an average developer from the expert. The expert can solve problems more elegantly and faster than the beginner because, apart from having solved the same problem before, they can compose a complex solution by pulling small pieces together from the standard library

Finally, the standard library contains solutions to common programming problems that have been battle-tested over generations of previous applications. It is certainly the case for Clojure. The robustness and reliability that comes with that kind of stress is difficult to achieve otherwise. There will be possibly just a handful of cases where something in the standard library won't fit your needs and will need to be re-implemented.

What's inside the standard library?

The Clojure standard library is quite comprehensive and can be divided roughly into 3 parts:

What is commonly referred as "core", the content of the single namespace clojure.core. Core contains the functions that have evolved to be the main public API for the language, including basic math operators, functions to create and manipulate other functions, conditionals. Core currently contains around 700 definitions between functions and macros. Functions in core are always available without any explicit reference from any namespace.

Other namespaces other than core that are shipped as part of the Clojure installation. These are usually prefixed with clojure followed by a descriptive name, like clojure.test, clojure.zippers or clojure.string. Functions in these namespaces are sometimes available just prefixing their namespace (like clojure.string/upper-case) but in other cases they need to be imported in the current namespace using require (this is due to the fact that while bootstrapping, Clojure already imports several namespaces that are automatically available for the end user. Very popular tools like nRepl or Cider also load libraries while bootstrapping, which are then available at the prompt. It is good practice to always require what is useful in a namespace explicitly).

Finally the content of the Java SDK which is easily available as part of Clojure Java interoperability features.

The standard library content can be roughly categorized by looking at the major features Clojure introduces and by the most common programming tasks. There are, for example, big groups of functions dedicated to Software Transactional Memory, concurrency and persistent collections. Of course Clojure also adds all the necessary support for common tasks like IO, sequence processing, math operations, XML, strings and many others. Apparently missing from the Clojure standard library are solutions already provided by the Java SDK, for example cryptography, low-level networking, HTTP, 2D graphics and so on. For all practical purposes those features are not missing, but just usable as they are from Java without the need to re-write them in Clojure. Java interoperability is one of the big strength of Clojure, opening the possibility to easily use the Java SDK (Standard Development Kit) from a Clojure program. Here's a broad categorization:

Core support namespaces integrate core with additional functionalities on top of those already present. clojure.string is possibly the best example. Core already contains str but any other useful string functionalities have been moved out into the clojure.string namespace. clojure.template contains a few helpers for macro creation. clojure.set is about the "set" data structure. clojure.pprint contains formatters for almost all Clojure data types so they can print in a nice, human-readable form. Finally clojure.stacktrace contains function to handle Java exceptions manipulation and formatting.

REPL namespaces contain functionalities dedicated to the REPL, the read-evaluation-print-loop Clojure offers. clojure.main includes handling of the main entry point into the Clojure executable and part of the REPL functionalities that have been split into clojure.repl in later time. The latest addition, clojure.core.server implements the server socket functionality.

General support is about additional APIs beyond what core has to offer. The namespaces present here enrich Clojure with new functionalities. clojure.walk and clojure.zip for example are two ways to walk and manipulate tree-like data structure. clojure.xml offers XML parsing capabilities. clojure.test is the unit test framework included with Clojure. clojure.sh contains functions to "shell-out" commands to the operative system. clojure.core.reducers offers a model of parallel computation.

Java are namespaces dedicated to Java interop beyond what core already has to offer. clojure.java.browser and clojure.java.javadoc offer the possibility to open a native browser to display generic web pages or javadoc documentation respectively. clojure.reflect wraps the Java reflection APIs offering an idiomatic Clojure layer on top of it. clojure.java.io offers a sane approach to java.io, removing all the idiosyncrasies that made Java IO so confusing, like knowing the correct combination of constructors to transform a Stream into a Reader and vice-versa. Finally the clojure.inspector offers a simple UI to navigate data structures.

Data Serialization is about ways in which Clojure data can be encoded as string as an exchange format. clojure.edn is the main entry point into (EDN) [https://github.com/edn-format/edn] format serialization. clojure.data contains only one user-dedicated function diff to compute differences between data structures. clojure.instant defines encoding of time related types.

Making Your Development Life Easier

The standard library is not just there to solve the usual recurring programming problems but to offer elegant solutions to new development challenges. "Elegant" in this context translates to composable solutions that are easy to read and maintain. Let's look at the following example.

Suppose that you're given the task to create a report to display information on screen in a human readable form. Information is coming from an external system and a library is already taken care of that communication. All you know is that the input arrives structured as the following XML (here saved as a local balance var definition):

The balance needs to be displayed in a user-friendly way:

Removing any unwanted symbols other than letters (like the colon at the beginning of each key)

Separating the words (using uppercase letters as delimiters)

Formatting the balance as a currency with 2 decimal digits.

You might be tempted to solve the problem like this:

parse takes the XML input string and parses it into a hash-map containing just the necessary keys. parse also converts :currentBalance into a double.

clean-key solves the problem of removing the ":" at the beginning of each attribute name. It checks the beginning of the attribute before removing potentially unwanted characters.

separate-words takes care of searching upper-case letters and pre-pending a space. reduce is used here to store the accumulation of changes so far while we read the original string as the input. up-first was extracted as an handy support to upper-case the first letter.

format-decimals handles floating point numbers format. It searches digits with re-find and then either append (padding zeros) or truncate the decimal digits.

Finally print-balance puts all the transformations together. Again reduce is used to create a new map with the transformations while we read the original one. The reducing function was big enough to suggest an anonymous function in a letfn form. The core of the function is assoc the new formatted attribute with the formatted value in the new map to display.

While being relatively easy to read (the 3 formatting rules are somehow separated into functions) the example shows minimal use of what the standard library has to offer. It contains map, reduce, apply and a few others including XML parsing, which are of course important functions (and usually what beginners learn first). But there are definitely other functions in the standard library that would make the same code more concise and readable.

Let's have a second look at the requirements to see if we can do do a better job. The source of complexity in the code above can be tracked down to the following:

String processing: strings need to be analyzed and de-composed. The clojure.string namespace comes to mind and possibly subs.

Hash-map related computations: both keys and values need specific processing. reduce is used here because we want to gradually mutate both the key and the value at the same time. But zipmap sounds a viable alternative worth exploring.

Formatting rules of the final output: things like string padding of numerals or rounding of decimals. There is an interesting clojure.pprint/cl-format function that might come handy.

Other details like nested forms and IO side effects. In the first case threading macros can be used to improve readability. Finally, macros like with-open removes the need for developers to remember to initialize the correct Java IO type and close it at the end.

By reasoning on the aspect of the problem we need to solve, we listed a few functions or macros that might be helpful. The next step is to verify our assumptions and rewrite the example:

parse now avoids the let block, including the annoying side-effect of having to close the input stream by making use of with-open macro. ->> threading macro has been used to give linear flow to the previously nested XML processing.

subs makes really easy to process sub-strings. We don't need an additional function anymore because turning the first letter to upper-case is now a short single liner.

The key function in the new separate-words version is clojure.string/replace. The regex finds groups of 1 upper-case letter followed by lower-case letters. The last argument conveniently offers the possibility to refer to matching groups. We just need to append a space.

format-decimals delegates almost completely to clojure.pprint/cl-format which does all the job of formatting decimals.

zipmap brings in another dramatic change in the way we process the map. We can isolate changes to the keys (composing words separation and removing the unwanted ":") and changes to the values into two separated map operations. zipmap conveniently combines them back into a new map without the need of reduce or assoc.

The second example shows an important fact about "knowing your tools" (in this case the Clojure standard library): the use of a different set of functions not only cuts the number of lines from 45 to 30, but also opens up the design to completely different decisions. Apart from the case where we delegated entire sub-tasks to other functions (like cl-format for decimals or int to clean a key), the main algorithmic logic took a different approach that does not use reduce or assoc. A solution that is shorter and more expressive is clearly easier to evolve and maintain.

The well kept secret of the Clojure Ninja

Learning about the functions in the standard library is usually a process that starts at the very beginning. It happens when you first approach some tutorial or book, for example when the author shows a beautiful one-liner that solves an apparently big problem.

Usually developers don't pay explicit attention to the functions in the standard library, assuming knowledge will somewhat increase while studying the features of the language. This approach can work up to a certain point but it is unlikely to scale. If you are serious about learning the language consider to allocate explicit time to understand the different nuances of similar functions or the content of some obscure namespace. The proof that this is time well spent can be found reading other's people experience: the web contains many articles describing the process of learning Clojure or documenting discoveries (possibly the best example is Jay Field's blog).

The following is a trick that works wonders to become a true Clojure Master. Along with learning tools like tutorials, books or exercises like the Clojure Koans, consider adding the following:

Select a function from the Clojure standard library every day. It could be lunch or commuting time for example.

Study the details of the function sitting in front of you. Look at the official docs first, try out examples at the REPL, search the web or www.github.com for Clojure projects using it.

Try to find where the function breaks or other special corner cases. Pass nil or unexpected types as arguments and see what happens.

Rinse and repeat the next day.

Don't forget to open up the sources for the function, especially if belonging to the "core" Clojure namespace. By looking at the Clojure sources, you have the unique opportunity to learn from the work of Rich Hickey and the core team. You'll be surprised to see how much design and thinking goes behind a function in the standard library. You could even find the history of a function intriguing, especially if it goes back to the origins of Lisp: apply for example, links directly to the MIT AI labs where Lisp was born in 1958! (eval and apply are at the core of the meta-circular interpreter of Lisp fame. The whole Lisp history is another fascinating reading on its own. See any paper from Herbert Stoyan on that matter). Only by expanding your knowledge about the content of the standard library you'll be able to fully appreciate the power of Clojure.

Summary

The standard library is the collection of functions and macros that comes out of the box by installing Clojure.

The Clojure Standard Library is rich and robust, allowing developers to concentrate on core business aspects of an application.

Information about the Standard Library tends to be fragmented.