Leonomics

It is no secret that software is being used in more and more aspects of society. It should therefore be no surprise that the demand for programming skills has risen over the last few decades. And by the looks of it demand has outsripped supply: I often hear that almost all companies would like to hire more software engineers if only they could find more.

Companies and governments alike have realised that if they want more programmers (and possibly therefore - cheaper programmers) they will need to take active steps in promoting programming teaching and learning. Governments have probably also realised that if they want their nation states to remain or become economically competitive in the long term, then they will need a sizeable population of programmers.

And so initiatives are born to encourage kids and adults to "learn to code". There is, for example, code.org which is "a non-profit dedicated to expanding participation in computer science by making it available in more schools". They have partnered with a host of tech companies. In their mission statement they say that "believe computer science and computer programming should be part of the core curriculum in education". There are many other organisations like this one all over the world. At first glance one would probably think, "o wow, cool, this is amazing..."

Enter the critics. They typically point out that this very modern interest in "coding" is a terribly watered down version of computer science or creativity or some other higher cognitive process. In this article, for example, the author, himself an accomplished professional programmer, argues that although there are "good intentions behind the movement to get people to code, both the basic premise and approach are flawed. The movement sits on the idea that "coding is the new literacy," but that takes a narrow view of what literacy really is". He then argues that programming is an application of the meta skill of modelling...

"Programming as it exists now forces us to model, but it does so in an unnatural way. And while teaching ourselves how to program will help us learn how to break systems down, it does so at the risk of focusing on the wrong things". Yes. We should avoid teaching the craft because people might be too stupid to see the forest for the trees. To me this sounds like plain naysaying. Of course there are risks involved in teaching and of course the approach can be improved. But I find it ridiculous that these risks should prevent people from actually trying to teach others how to "code" - no matter how watered down it may be.

I often wonder what emtions sit behind these critiques. My theory is that programmers feel special. Like they have a special skill that not many people in the world have and they are part of an elite group which happens to be rewarded with above average wages and fantastic job security. And the idea that this skill will become as common as reading and writing freaks them out. I mean, how did the clergy feel when people started reading the Bible for themselves? They were shit scared because they knew they would lose their monopoly on knowledge along with the resultant power and status.

As a self taught professional programmer who did not graduate in computer science, I have on several occasions during the last 3 years felt this insider high and mighty insecurity. "Nah, this task is not for _you_"..."while you're an awesome _almost_ engineer..." they say. I suspect that it is the same haughty attitude that wants to prevent people from learning to code.

Sometimes I stand on the bus, looking at the people around me and suddenly think to myself: "I have nothing in common with you. You are not my people. This is not my country. You mean nothing to me".

I consider myself to be a historically privileged person. The accidents of history have, on the balance, been in my favour. I was born a white South African male, the son of a stable couple living in Stellenbosch - a prosperous, diverse and beautiful university town. Thanks to the prudent choices and consistent encouragement of my parents I had the chance to attend good schools and a good university. Not that I ascribe all the good things in my life to historical accident but I live with an acute awareness that the extent to which my hard work has been rewarded is connected to the historically arbitrary position from where I started.

Through yet another accident of history I find myself currently living and working in Norway. Of course it is not completely arbitrary that I live here but if my girlfriend was French I would be in France now. I didn't decide to come and live here because I want to become a Norwegian. Sure enough Norwegian society and landscape has many great features which I positively enjoy but my main motivation for being here has nothing to do with what the state is willing to offer me if I meet their conditions. I am happy to pay my taxes so I can contribute to redistribution but at the same time I don't expect any welfare support whatsoever in return.

Being a South African I grew up with a deep suspicion of the ability of states and governments to make the correct decisions for society (I mean, just look at South African history). I also learned that the most robust strategy for survival and continued welfare is to rely on skill and private networks. For me the world is basically Hobbesian - not because I want it to be so, but because I think it is a wise strategic standpoint. Taken together this means that I basically expect the state to just be stable enough to ensure rule of law and otherwise leave me the fuck alone so I can take care of myself. Anything more than that is praiseworthy but it is also a bonus which I do not count on.

Norwegians, however, demand much much more from their state and rightly so. Consequently, the state demands much of its immigrants. And if you're from Africa (South Africa is still Africa) the demands are even higher. If I lost my job I would have to leave the country if I did not find one within a few months. Such is life.

I don't think I will lose my job or that I will not be able to find a new one in time even if I did, but I don't want to be so dependent on my job to be allowed into the country. That's just not first prize. And so my aim is to get a permanent residence permit. That would decouple my right to be in the country from my labour market status. Decoupling is good. There are other benefits like getting money from the state if you're unemployed, being allowed to vote in the national election and getting subsidised education and health services. But apart from voting I'm not interested in those. I want the mobility.

And so the conditions emerge! To get the permit I must either do 250 hours of Norwegian language courses and 50 hours of cultural courses or I have to pass a test to prove my language skills. Since I already speak Norwegian with enough competence to make friends in the language I am opting for the test. But still I have to work quite hard on my grammar to pass it. First world problems I know.

Currently I am doing a three week course aimed at helping people pass the test. Once again I consider myself lucky to have enough money to be able to take this course. But even as I sit there, realising my privilege all the way from birth to this very moment I feel a certain amount of resentment. I can't help but feel irritated by the state's requirement that I pass this test or do those hours of courses for me to be apply for permanent residence.

They have bargained a deal: dear immigrant, we give you this permit with these benefits but then you have to be a good potential citizen and prove you can speak and read and write our language. But I am not interested in the package deal - only one very specific part of it. I wish I could say: leave your welfare benefits out of this and just give me the mobility. I just want to be allowed to pay taxes, live and vote here. And then they could just leave me alone so I could continue to learn Norwegian at my own pace.

I understand the position of the state though: it makes sense to require these competencies from immigrants to ensure effective integration and social participation. If you want to cultivate a certain type of democracy you need the right kind of citizens.

I do think it is fair to require me to prove my language capabilities, but when I'm tired after work, learning grammar rules, memorising verbs and I reflect on the historical arbitrariness of my location and how little I actually expect from the state I cannot help but resent the power that it has over me. I mean, even if I had a great argument for why I should be allowed to live here the state has the monopoly on violence. I have to obey and do what I am told and I've never enjoyed doing that. But then again, I'm glad I didn't fall in love with someone from Saudi Arabia.

It is becoming quite commonplace to read headlines like "Will you be replaced by a robot? We reveal the 100 occupations judged most and least at risk of automation". In this case the headline is based on the findings of a paper authored by a group of Oxford researchers entitled, The future of employment: how susceptible are jobs to computerisation. The authors estimate the probability of computerisation for a list of 702 specific occupations, ranking them accordingly. Two of their main findings are that wages and educational attainment have negative relationships to the probability of computerisation. In other words, people with jobs that require high eduction and earn high wages are, everything else being equal, less likely to be "replaced by a robot". Not so sursprising I guess.

I've also found myself in conversations around the dinner table where people discuss how machines will replace people in jobs. It normally goes something like "soon machines will be able to make better diagnoses than doctors" bla bla bla and then some.

While this technical aspect of computerisation is interesting in a limited sense I think the socio-economic reality underlying this transformation does not get the attention it deserves in the press or in casual conversation. Allow me to explain...

The skeptic in me has always been an admirer of Thomas Hobbes and his social contract theory. For Hobbes, the state of nature is a state of war in which all individuals compete with each other. Social institutions like the state arise to help us move beyond this state of war into a state of economic prosperity. Now what does this have to do with the computerisation of work? Let's do a quick review of recent economic history and theory.

Automation is not new. Ever since the advent of the industrial revolution entire economies have steadily embraced automation in almost every imaginable way. As Schumpeter pointed out many years ago, this is typically a process of creative destruction in which old forms of production are replaced by new ones. There are always costs associated with the benefits - some people are losers and others are winners, relatively speaking.

To a great extent this process of creative destruction, in the capitalist world, is made possible by the system of private property we all agree upon - the social contract. People can own things exclusively and when the things they own are included in the production process they are rewarded accordingly.

For example, I own various financial assets which are tied to the economic performance of both companies and countries (equities and bonds) and get rewarded in proportion to my share of ownership. This is classic capitalism. To take a more subtle example though, I also own my time and my skills and sell them to my employer. Given my employer's ability to generate profit in the market, and given the details of my agreement, I get paid my share.

Now let me put this together explicitly: before we agreed upon the system of private property we were competing ruthlessly with each other for scarce resources; in order to turn this competition into something more productive we agreed on a system of private property - the competition is, however, still ruthless; private property allows people to own things like equities, bonds, time, skills and much more recently in economic history: machines, software, algorithms and intellectual property that enable the computerisation of work. The thing to notice is that people own machines.

Machines do not work like humans do. They do not earn wages. When a job is computerised the income once earned by that human is redirected to the owner of the machine, not to the machine itself. In other words, people's jobs are replaced by other people, not by machines. The headlines should read "Will you be replaced by the owner of a robot?" The Hobbesian reality is that people with the knowledge, skill and capital to computerise work are outcompeting other people.

Back in the day when the common man and woman could not read and write the Catholic church was cashing in on their monopoly on literacy. Nowadays when the ability to write code and create software services are limited to a privileged few, tech companies are cashing in like its the middle ages again. And the spin is that people's jobs are being replaced by machines. Nonsense. People are being replaced by other people and always have been. The day a machine replaces another human in the economy is the day we give equal rights to machines to own property and earn income. And that is clearly not what is going on.

Planning on doing in-house analytics are you? Know this:

There are many recurring and entertaining discussions (sometimes fights or heated debates to be more civilised) in the software development industry. One of them, in the domain of architecture, is about how to design and build for scale. To simplify let's take the opposite ends of the spectrum: in the one camp there are people saying that you should design for scale from the start and build accordingly, while the other end says no way forget about it just do what works to get your minimum viable product out of the door and worry about scale later.

People get very hectic and smug about all this, and try their best (in the best of the discussions) to bring empirical evidence to the table. One camp will tout an example from a super lean company that did everything with a single powerpoint slide until they got enough funding to make a demo and so on. Then the other comes along and gives a counterexample of long term vision and careful upfront design for scale.

While entertaining, I think these discussions are often focused on the wrong level of abstraction. A more helpful discussion would be about what kind of things one can do regardless of how many people are using your system at any given time. That is a robust strategy, albeit more abstract. And so I propose the following advice: do things that always scale, from the start to the end. What are these things?

Think. Listen. Speak. Read. Write. Draw. Experiment. Observe. Ask. Question. Argue. Understand.

Projects get stuck when people stop doing these things. Implementation fights become unresolvable when these more abstract tools are traded for less sharp ones. Ideological commitments derail rationality in the absence of the above mentioned practices. 

Entering the world of professional software development without formal training or practical experience (as an Economist who likes to mangle data to answer questions) has its advantages. One of those advantages is that you can get a fresh and relatively untainted view of the way that software development tools - programming languages - are stereotyped.

In software engineering, it is difficult to choose the right tool for the job, to make the trade-offs in the best way you can. But it is even more difficult doing this and making sure you're both cool and hardcore. I mean, who wants to be uncool and not hardcore right? I hope these descriptions of programming language stereotypes will help guide you to make the cool and hardcore choice when you want to solve your problem :)

Some people may get angry when they read this (even though I hope they don't) but at least everyone should be equally offended no matter what language they worship.

Java - Designed to help lots of idiots make a huge bloated verbose overly abstracted monster, wonderful to hate on, yet industry standard

C - Classic, hardcore, but why don't you just use C++?

C++ - Even harder core; C++ers have a bitter distaste for Java, a _real_ language

Assembly - Mostly for Russians, people tend to say they've done things with it rather than want to do more with it (gets you street cred)

Objective C - For iOS people who love fonts, shiny things and compiled languages

Python - Toy language for people who can't write a real language that compiles to machine code; eat bytecode motherfucker!

Ruby - The other Python, eat more bytecode!

Fortran - For academic scientists; respected by most, tried by few today

Scheme - For smart people while they are at university

Common Lisp - Only for the most intelligent people on Earth; all other languages live in the shadow on CL; struggling with obscurity though

Clojure - The new generation of clever functional people use this instead of Common Lisp, partly because they are smarter than Object Oriented (OO) people who use _only_ Java, partly because they can at least keep on using the Java Virtual Machine (JVM) that still lives in the underworld of the Enterprise they are working in

Scala - also for new functional people who still love the JVM but have become smarter than their OO colleagues

R - R WAT??? Statisticians use programs WAT? Who cares about the open source implementation of the Grammar of Graphics anyway? Just use GNUPlot

Javascript - '===' and web developers are not real developers; real developers do server-side shit; and don't even try and begin with this node.js nonsense;

C# - Windows, um yeah, nevermind

F# - For functional Windows people

Haskell - For people who were functional before Clojure and Scala became the business

Erlang - Gets respect from almost everyone but noone dares use it in production because people are not clever enough to move away from Java to such a smart person language

Go - The new cool C++, almost like Javascript and Node.js except for real programmers

APL - Almost noone knows or remembers it and who would want to program using arrays anyway? What would we do with all our for loops?

A+, J, K, Q - All variants of APL that almost only quants on Wall Street know - forget about getting street cred for knowing it in your Java home, you're better off learning Brainfuck

Julia - Gets you no cred in the workplace but lots of cred on twitter, github, the blogosphere and the conference schedules, basically a hardcore mixture of R and Python with MATLAB-like syntax

I was walking down the road in Oslo and I saw some wires routed into a truck next to the road. At first I walked past, but quickly stopped, turned around and approached. As I came closer I could see the ends of the wires had been split open and hundreds of tiny tube-like wires were protruding.

Most of them routed into an apparatus of sorts and connected to a measurement tool. The guy inside the van was happily listening to music as I appeared in his view: "Hi, is this fiber optic cable?", I asked. "Yes it is", he said smiling. "Cool, so those are the actual fibres that carry the IP packets around on the Internet?"."Yep". (I forgot to ask if he was a NSA operative).

I had seen a tiny part of the Internet this morning. Excellent! I was reminded of a book I recently read by Andrew Blum - Tubes, A Journey to the Center of the Internet. Now is a good a time as any to write about the book...

The author who is a technical journalist by profession, shapes the book around his quest to learn about the Internet - its physical infrastructure, how it works, the history of the network of networks, the geographical distribution of the networks, the technology. In weaving these potentially dry and technical topics (which are interesting in themselves) into his personal journey he is able to incorporate people and places into the book in a natural way.

The book explains many of the core physical and logical components of the network of networks that underlie the Internet. It does well to incorporate the right amount of techinical detail about TCP/IP protocols, DNS, Internet exchanges, copper wire, fibre optics and stories of eccentric academics who were the effective pioneers of the Internet.

The only critique I have is rather obscure: the word "singular" is used way to often. Otherwise it is a great read which I can recommend to anyone who feels curious about the workings of the network of networks that we all use every day.

1. The Problem

Whenever someone makes a scientific claim, or presents an argument supported by evidence, my first demand is to know how the scientific result or evidence was produced. I want to know the source of the data, the possble measurement errors at the point of data creation, the kind of model that was used, the assumptions and the decisions that shaped the analysis, the extent of statistical confidence, the relative sizes of different influencial factors. Which variables were thrown out and why?

Anyone who has ever read more than a few scientific/academic papers, no matter in what field, knows that you pretty much have to take the integrity of the results on faith. Sure, there's peer review and the potentially negative reputation effects of being caught with nonsense results, but the reality is a bit more complicated than that.

When dealing with areas of knowledge that are less known to the wider public, and maybe even relatively unknown among your peer group of specialists, it is very difficult to get effective oversight of your work, even if you are as pure as can be. The reason for this is that people are simply not there when you're in "the laboratory", making all those tiny decisions and simplifying assumptions. You might even forget about all of them yourself even if you try to be diligent! On top of that, people who give oversight are generally quite busy. The point is, it is not easy to create reproducible research. And forgive me for stating the painfully obvious: this is a problem.

What you need, effectively, is access to the laboratory where the results were produced. Ideally, you want to travel back in time and accompany the researcher, or group of people, as they develop the results, and witness the decision-making around the assumptions and simplifications. If you could have that, then you would be in a much better position to understand the limitations of knowledge at hand.

2. Open Source Tools - Possible Solutions

Almost all science (social science included) entails quantitative work. This means working in some computing environment where analytical arguments are implemented with empirical and/or simulated data - measure, estimate, test, optimise, test... For the majority of people, their laboratories are really on their computers, located in some specific computing environment. So sharing every important step in the evolution of the research (changes in data and code) and sharing that environment (the program that does the math) would amount to sharing the laboratory. And you can do all of this with open source tools - further lowering the boundaries for reproducibility by avoiding commercial software license issues.

2.1 Version Control: share the evolution of data, code and decisions

Version control is a tool that allows you to do many things: keep track of how files and data change, down to the very last bit, without ever changing the file name. You will have this history as a database, and you can revert to any state whenever you want. You can work on different versions of the project at the same time without having to choose one over the other - you can explore different branches. You can share this database with your collaborators and all of you can contribute and work on the shared database at the same time.

One example of an open source version control tool is Git. It transforms a set of directories and files into a repository. Repositories can contain multiple versions of the same files and data at the same time (branches) and they can be changed and shared by more than one person at a time. A web-based tool that makes sharing such repositories easy is Github.

By doing scientific work with version controlled and shared repositories, anyone can see and be part of the evolution of the project data and files - the analysis and it's history is open for people to see.

2.2 Virtual Machines: share your computing laboratory

Personal computers, even when they are work machines, are highly customised to individual needs. This is even more true in the case of research. There's always a very specific set of add-on packages that enable the analysis. Sharing the final code that produces your set of results is normally no guarantee that another person with the same analysis program will be able to run your program out-of-the-box. Further, it is no guarantee that the person who is interested in your results actually has the program that you used.

One solution to this problem is making use of virtual machines such as those provided by Vagrant. Vagrant allows you to set up custom virtual Linux Operating Systems and share these "boxes" with others. In the science example here, I can, for example, constantly replicate my analysis in my Virtual Machine (in addition to some other environment of my choice) and when the time for publication arrives, simply share the Virtual Machine along with the version controlled repository. I can make sure that my whole analysis runs without effort on any computer.

An example of such ready-to-use data anlysis virtual machines, shared via a version controlled repository can be found here.

2.3 Open source and generic data analysis tools

There are certianly cases where proprietary software is appropriate for the analysis at hand. Often the choice of tool is not really even made by the individual, but rather it is set by the context. Institutions typically have existing licenses with software vendors, and are trained accordingly. So using alternative open source tools are not without cost. But my argument here is that the cost of adopting open source tools, where possible, will have more than enough returns - both personally and socially. It will further enable reproducible science. I list a few useful analysis tools in passing: Python (and most notably the Pandas library), R (notable packages and useful language reference), Octave, Weka, Julia, d3.

3. And now?

I first read the quote by Karl Popper that I now struggle to find in Nassim Nicholas Taleb's book - The Black Swan. A great book about the basis of knowledge, modelling economies and markets, complexity and much more. So because I can't find the quote within a reasonable amount of time (my copy of the book is in another country) I will have to paraphrase. Popper was talking about the determinants of what he saw as good philosophy. It went something like this...

The problem with much of the philosophy of the time was that it was being produced for its own sake. And one can only produce good philosophy when you are forced into it by trying to solve practical problems.

Ever since I read that quote I've been taken by its power. One can certainly make the case that sometimes good philosophy (whatever that is) can be produced by another motivation structure, but there is something extremely compelling about this way of looking at knowledge creation.

The same general view emerges in Karen Armstrong's book - The Great Transformation, The Beginning of Our Religious Traditions. The common ethical rule that emerged from all the Axial religions, known today as the Golden Rule was the result of trying to solve the practical problem of how to live. Not the search for some metaphysical truth.

Once I found these perspectives so clearly formulated in the writings of others (like Taleb and Armstrong) it resonated with my own frustrations with philosophy and religion (sometimes more than others). When the solution of practical problems become the yardstick for the usefulness of a way of life, or a body of knowledge you suddenly have a clear way to evaluate what you are doing.

And my claim, and experience, is that the same holds for programming. These days I spend most of my working hours programming "big data" related applications and analyses. And sometimes I get this awfully familiar feeling that I am not programming to solve practical problems. Is this technology in search a problem? Where is the bottom line or am I just hammering out these lines of code because someone wanted it? Someone wanting it or some technology requiring it is just not enough. I need to know what the practical problem is that drives the creation of the abstractions.

Lofoten is a pretty beautiful area to say the least.

It's not the kind of place where you want to encourage heavy industry that will have environmental impacts without thinking twice. Especially in a rich country where you don't really need the money (I know it's hard to imagine but it actually kind-of exists).

The current government of Norway has said that the Lofoten area will not be open for exploration and drilling during the current political office. So, officially, there is no immediate concern for people who fear that drilling might happen. But the current political term is soon ending. The fact is also that seismic data about the ocean floor has been gathered and sold (selectively, exclusively and at a very high price - 43 million NOK) to Oil Prospectors. This is a clear indication that there is interest in Oil in Lofoten. 

But the situation is more complicated. It is not inevitable that any of this happen - because it is public land it will have to be public policy that changes the current ban.

And so we come to the intersection of open data and the quality of democracy.

As I was walking down to Parliament this Saturday morning, to join the small and peaceful demonstration of Lofoten lovers I was wondering about the evaluation of this whole mess. How would one properly form an opinion about it?

I thought that somewhere along the line there had to be measurement, and data and modelling and assumptions and simulations and distributions...and that having access to this decision making data would be valuable to anyone who wanted to dispute whether or not it was a good idea. I've developed too many models myself to trust any empirical results on "authority".

I heard some people say that "they" say the debate should be focused on facts and not on emotions. But if you deprive people of the information on which facts are built then they do not have any othere recourse but their emotions.

Why not make the data public? Perhaps the state generated the data and want to sell it to break even on the project. But then sell it for a while and then make it free after a while. Then the Oilers of the world can fork out their cash and pour themselves over the data in the mean time. And after a while the public can have their turn to scrutinize the hidden assumptions. This will foster a much more balanced debate about the issue.

Recently, at a seminar event in Bergen, I was talking about a report I wrote about ethical guidelines for government bond investments. When the time for questions came around there was a particularly fervent member of the audience who ended up asking (in effect) what alternative socio-economic options there are to the global network of capitalism. Tough question if ever there was one.

It's quite easy to laugh off such questions as being crack-pot nonsense, and most of the time they probably are. But on this occasion, in part because I was sitting in front of an audience of strangers, I decided to give it some thought. How on earth could I give a coherent response? What could I possibly suggest as an alternative for global capitalism? I don't even necessarily want an alternative, I thought to myself...

So I decided to set the bar a bit lower: instead of trying to answer the audience member's quesiton I decided I would respond with a concrete example of economic and social activity that operates according to different incentives than those arranged by the usual capitalist market economy. What did I say?

I said: let's consider open source software. The flagship example is the Linux operating system. Today the Linux OS is used to provide Internet services all over the world, services that range across the spectrum of human activity. Google, for example, uses thousands if not millions of Linux servers to provide their search and email services - and these are things that improve the lives of people every day. And this is just a superficial example of the far-reaching impact of open source software on society.

My point, and I suspect it was slightly lost on the audience because it came a bit out of left field, was that open source projects are created and coordinated without the existence of the pofit motive. People organise and cooperate on different terms and on the basis of a different reward system, and yet something useful is produced. And often, the quality of open source projects simply outshine their proprietary counterparts.

So on the basis of this - if only this - everyone at university (even earlier) should use and learn about open source software. But there are other more utilitarian reasons for doing so too. And this is where the universities come into the picture.

Students of Economics (my own background) typically learn to do quantitative work with proprietary tools like: Mathematica, Maple, Matlab, Stata, EViews, RATS, Oxmetrics and Excel. The mainstream exception is that many students learn to use LATEX. Less frequently, and this is great, they are introduced to working with R - but this is more popular in the Statistics departments. Every now and again you find a paper published using Python but it is still by far the minority of students that are tought that way. It is very different in Computer Science departments and that is to be expected.

Anyways, if students learn to use open source tools for their work they will get to know the dynamics of open source communities and get exposed to real-world collaboration. Their computing knowledge will become stronger, their horizons will be broadened. And if all of this is not convincing enough then consider the fact that many world class companies use only open source software for data related work. For practical data science, we use no proprietary tools at work.

Quite recently I completed a 20 year forecast of the South African Economy. The South African Department of Energy has committed to an ambitious and very necessary long term plan to increase the energy capacity in the economy. As part of this planning they are have to choose how much of the new capacity will be allocated to renewable energy versus fossil fuel energy. This choice is likely to have implications for the price of electricity: given the state of technology more renewables would likely translate to a higher electricity price (holding everything else constant). How will different electricity prices impact growth in the economy? What impact will this have on socio-economic outcomes like employment, poverty and inequality? These were the questions that my forecast were going to shed light on. Incredibly interesting questions indeed.

Quite recently also, The Norwegian Coalition for Debt Cancellation launched a report I wrote about ethical guidelines for the Norwegian Government Pension Fund's government bond investments. The report develops ethical investment criteria based on transparency requirements in bond issuing countries and applies it to the context of the Norwegian Oil Fund. This was work done earlier in the year. Also really interesting.

It just so happened that most of the work on the energy project occurred at the same time of the launch of the report. As a result of this it struck me how different the knowledge representation and communication mechanisms were between the two. I was doing the energy project with relatively strict requirements for deliverables: Microsoft Word documents, Excel spreadsheets for the microeconomic model, Powerpoint slides. I had been transported back into a knowledge creation process that I do not use any longer. It was striking how the software shaped both the creation and representation of knowledge: I had to make use of Excel spreadsheets to convey what I thought was important in the data, in combination with my writing in the document. And it was expected of me to translate my key findings into Powerpoint slides.

And it is on the point of making slides that the energy job was most different from the ethical guidelines job: for the report launch I could use Prezi! Woohoo! What a great tool to create slideshows for presentations. So user firendly and completely platform agnostic. Having a canvas where I could place ideas on and using zoom and slide as visual cues to represent the relationships between the ideas, while talking about it was so effective. The audience simply loved it. I got such good feedback from people. In fact, many told me explicitly how happy they were that it was "not another Powerpoint". Have a look for here.

One cannot underestimate the importance of such details when representing and communicating knowledge. Every detail matters.

There was another difference I noted in the energy project, but this time the difference was with my current data science work. For the forecast I used a small systems-of-equations econometric model in the style of growth accounting models. While I could have done this in R or even Stata I thought (being under some time pressure) I would rather choose the beast I know from the past: Eviews. Back in the day when I was an Eviews regular, and macroeconomist by day, I did not code at all. I have to admit I was a blackboard writer and button clicker. Anyways, so I developed the model without version control and without code. And wow, what a difference in the knowledge creation process!

Having version control that tracks the development of your analysis and having code that spells out in minute detail the logic of the anlysis really helps to think clearly about the problem. Especially if you have to work on the problem here and there in between other things.

For R users, and for data graphics people, Hadley Wickham's plotting library - ggplot2 - needs no introduction. Knowing this, before talking about my experience of the book, I'll give one just the same.

ggplot2 is the R programming language and/or statistical programming environment's best library for constructing data graphics - in the context of interactive and exploratory data analysis. It is built on top of the grid graphics library and provides a grammatical and programmatic interface to grid graphics methods. Since it is explicitly based on Wilkson's Grammar of Graphics its expressive nature is better suited to the way that graphics are constructed from statistical data. It makes it easy to see what the data looks like.

Reading a book about ggplot2 is certainly not the standard way of learning how to work with it (at least in my perception). I mean, the latest documentation is available on github and the book, published by Springer, is almost a year old by now. Some of the methods discussed in the chapters have been deprecated since. So why bother with the book at all? Why not just go straight to the ggplot2 API documentation on the web? I can tell you my answer.

The book walks the reader through all the core ggplot2 methods showing how they can be used to control all plot elements. In addition to showing this with detailed code examples there are many dicussions of best practice: which colours to use; grid lines or no grid lines; free scales or fixed scales on multiple plots... The book helps you to develop a useful terminology for discussing data graphics. And this is something where the book has the upper hand on the API documentation: engaged discussion of details.

In addition to showing great examples of statistical data graphics the book also introduces many R libraries that are a must-have for the data wrangler: reshape and plyr - both great tools for data analysis in R.

Some may say the days of desktop computing are numbered - in the sense that results are now more often displayed in browsers than not. Others may say that R and ggplot2 are not made for the web at all. And there is truth in both of these statements for sure. That's why the R people are working on making the R computing environment more web friendly (See: RStudio Server, RShiny soon to be released with a stand-alone server, Slidify for HTML5 presentations, Rpubs for publishing Rmarkdown). Be that as it may, reading the ggplot2 book and using and knowing the ggplot2 library teaches you about data visualization. It will help you write better code - no matter what programming language you use.

This is the most engaging book I read this year. I enjoyed each sentence from start to finish. I unequivocally recommend this to everyone who has to deal with quantitative information - whether you are a user of producer of data graphics this will sharpen your eyes to details you've never considered before. Anyways, now that I've got that out of the way, what is it all about?

Well, it's about the visual display of quantitative information. How to display such quantitative information in a way that conveys the meaning of the data. How to make systematic choices to improve on existing graphics and how to develop new ones with a keen eye to the right details.

Without trying to reproduce the book itself I will try to summarize what, to me, is the core message: maximize data ink!

Data ink is a mark that displays information about the data. I have posted examples of data graphics discussed in this book (a slopegraph and a binary sparkline). Each of these convey maximum data information with minimum ink. It is really best to work through the multitude of examples in the book instead of spending your time thinking about graphics in the abstract.

Every time you make a mark on the data canvas (not matter where your viewport is) think about the ink - is it data ink or not? Does the mark contribute to the display of infromation or is it purely decorative fluff? With simple questions like these one can get very far in designing well functioning data graphics.

The great thing about working in a new field is that no single person or institution can even begin to claim a full or final definition of what it is. And in recognition of that fact I will not try to give an exhaustive treatment of what Data Science is. I want to explain what my experience of the practice of Data Science is. My hope is that it will contribute to a more realistic picture of the profession as it exists today. Also, it will give me a much needed chance ro refelct calmly about my daily work - which can seem, from time to time, quite chaotic.

Before delving into the details some context will help: I work at Comoyo - Telenor's digital innovation hub. We build entertainment and communication services. Telenor is Norway's telco giant and like other Telco giants has realized that the future of communication and entertainment will be enabled by Internet Protocols (in short).

Thankfully, "they" also realized that to profit from this inevitable transition they will have to own the intellectual property that generates the entertainment and communication infrastructure. To make it less abstract they realized that they could not continue buying licensed software from external vendors. Instead they have to become originators of new software. And this is where Comoyo comes in with its herd of engineers. The analytics team - 8 of us - has been tasked with making data useful for the organization, and that's how I fit into the story.

What do we do at an organizational level?

We (try to) help executives, product managers, sales, marketing and developers to get feedback from customers. In each case the feedback is used as evidence to support a decision. Should developers continue with a new feature? Is a product gaining traction among users? Decision support in short.

What do we do operationally?

Aha! And so we come to the interesting and the messy part. Firstly, and this is crucial to understanding what the practice of Data Science is like, we are a 'we'. We work within a planning horizon of 2 weeks and every day we start by having a status meeting. I don't think any of us work for a few hours without asking each other for help. Providing the decision support outlined above requires extreme collaboration. For someone looking to get into DS it is wise, therefore, to be prepared for this. Also, this implies that being proficient with tools that enable collaboration have incredible returns. A concrete example: being capable with version control will make your life easier and your team work faster.

Technologically speaking: using cloud infrastructure for everything (know your Linux, know your unix terminal and know your bash!) we ship and parse logs generated by back-end and front-end services to a Hadoop cluster (know your HDFS! know your JVM!). After some data aggregation (know your SQL and MongoDB!) we visualize various results in a web display (know your HTML, CSS and Javascript!). Finding patterns in the data (know your XML and JSON!) is best done with a programming language that facilitates interactive data exploration and visualization (know your Python and your R!). Knowing one tool will get you nowhere. Being willing and able to ask questions and learn something new every day will.

Honestly, I think the most important thing for anyone to enjoy and succeed in practicing DS is to really want it. If you want it you will do what is necessary, whatever that is. I just want interesting data to be where it should be in the format that it should be so I can make pretty pictures of it. Is that too much to ask for?

Suppose we're interested in South Africa's GDP growth performance since 1981. Now also imagine that we just want a quick glance at a data graphic and know whether recent years have been better than the past, in addition to the overall performance. 

Let's say that any year that growth was above its long term trend is a success and the opposite is a failure. How are these successes and failures distributed across time?

If the bar is stacked upwards it indicates a success. You can quickly and easily see that South African GDP growth has been above trend more often in recent years. There is no visual clutter at all.

This is a graphic for the busy person who already understands the statistic being plotted, for the person who has to make quick decisions.