#dev-journal

I spent the last few days setting up automated e2e tests with node for Angular. I hadn't done any real back end JavaScript, so I was learning how to put together grunt tasks with a node server. The problem was simple enough: start a proxy authentication server that registers with a central authority and point the tests to use the temporary proxy server.

I quickly realized my ease of JavaScript use was heavily reliant on Chrome Dev Tools. I had very little intuition of how node behaved and how to structure grunt tasks. I configured a pre-made proxy module (yay npm), and tested to make sure the proxy server functioned as expected. It took me a few tries to get right, as it turned into a guessing game of what would be easier to do vs. what sounded easy. Apparently https->https: is harder than http->https. I had a dozen of these little lessons of how to setup and start the proxy, and then correctly register with the internal authority.

I didn't take the examples verbatim, instead I tried to reason what I thought I needed and make adjustments based on circumstance. This was a mistake as I didn't understand what I really needed, nor did I understand what was necessary in the example. I should have copied the examples out of the documentation, gotten those to work, and then slowly evolved the code to where it needed to be. Instead I ended up reconstructing the bulk of a few different examples through trial and error. The code ended up being about 100 lines, which is embarrassing that it took me the better part of 2 days to get working. Which brings me to my major lesson.

With strict environments, exploring via failure can be efficient. There maybe only one right way to do it, and that will be the only one of a few options that will compile or not throw an error. With forgiving environments, not throwing an error is not a sign of confidence. It's similar to looking at the last compiler error instead of the first one. You end up fixing what looks like a real bug, only to determine that it's irrelevant because something above it misbehaves and so the bug fix was for a completely different problem that may or may not have existed.

This requires more refined up front expectations for environments with soft or hidden errors, the cmd line node debugger isn't as good as Chrome but it gave me a decent picture of the state of the program. Without planning I didn't know when my expectations were wrong vs when the code was wrong. I was adjusting my expectations based on what I was seeing the code doing, which was slowly leading me astray.

I'll end with my worst mistake to illustrate this lesson: When attempting to add an external task to the grunt file, the grunt.loadTasks signature takes a path to the task. I incorrectly read this as path to the task file, when in fact the correct meaning was path to the task directory. Passing the task file didn't produce any errors, unlike passing it an incorrect path (which did produce errors). This left me stumped for awhile thinking my task contained the problem, when in fact I was just hooking it up incorrectly. Luckily, asking for a quick sanity check from my office-mate solved this, just by asking about first principles I immediate saw my mistake. The worst part was when I corrected the error, I didn't make the connection that grunt would then attempt to load all .js files in that directory into the node instance. It didn't throw any errors or warnings when it loaded files without grunt tasks. So then I spent far too much time trying to figure out why my proxy would start with the incorrect arguments when called from the external grunt task!

During the long build cycles, I again tried to go through the Learn You A Haskell For Great Good, but I found that I just wasn't absorbing it without the ability to test it out. I'll need to keep my eye open for a project that would fit well with a functional language.

While fixing bugs, I often found it tempting to do general refactoring of the class to help make it more testable. Most of the classes had some tests, but when I tried to make a test to prevent a regression of the problem I found, it was very difficult. The parts of the classes that were tested were the trivial parts, and they didn't even run fast (contributing to the painfully long build cycle that started this diversion)!

I then happened upon a youtube presentation that caught my interest by the author of Angular.js on the psychology of testing. He brought up a great point, everyone says they like tests, and no one would admit that they don't know how to write testable code, but yet new code continues to get written with incomplete unit tests. 

My experience with TDD has been when writing a utility package or library, something sufficiently isolated and well understood that enumerating the expected inputs is pretty simple. When interfacing with legacy code, unit tests quickly become more complex as the boundaries blur. The most common argument I'll have with myself is "what exactly am I testing here". Because sometimes I end up writing tests that touch and indirectly test all the dependencies of the object, as well as the object (I've had many discussions with my team about what boundaries to draw on tests when working with Angular.js controllers). Angular's use of dependency injection never struck me as the key to it's testability, as I'd never been exposed to DI before Angular (not being Java or Factory pattern user). Separating the program's construction from the program's business logic sounds like a good idea, but I haven't seen or attempted it in practice. I'll have to keep this is mind when putting together new subsystems in the future!

I could count the number of pages that showed I spent too much time reading and mapping out code when I made much bigger steps in the debugger. I could see that my tests were effective because the projects where I wrote tests first had fewer issues later when I made changes. The length and detail of my notes was a metric I used to measure my technical progress.

But for my most recent project, I found that metric wasn't informative anymore. I took very few notes during most of my tasks (user stories), and a disproportionate amount of my notes were on one or two really nasty infrastructure bugs that I needed my team to help fix. I found I wasn't taking notes for myself anymore, I was taking notes to share what I was doing with my team. This was great for stand-ups and other collaboration opportunities; I always had an updated list of the specific technical problem I was on and what other related problems were in progress or just solved.

Upon review, I found them to be mundane and uninformative. Noting that I was 'testing and integrating fields i,j,k from the database to the front-end' was important for my team if they had issues accessing or verifying the correctness of field k, but now that information is irrelevant. I don't have any information about what I was thinking in a technical sense outside of my commit history.

I'm now going to start keeping my domain notes separate from a technical journal. I didn't like the experience of looking back at 6 month project and thinking "Did I actually learn anything?". I knew I was getting things done, my commit history and those fancy graphs of story points proves when and where that happened. I now need to keep track of what articles I'm reading and what tools I'm learning to use. Before, I didn't realize that over the course of the project I gained a whole ton of T-SQL experience until a team member asked how in the heck I knew how to do a feature that needed it.

I don't know if I'll post my insights here, as I feel they might be trivial things like learned more tmux shortcuts or used an sql output clause. Or even less useful notes about internal tool quirks and infrastructure experiences.