Brendan Graetz

As promised in the previous post, let us take a look at how to extract data from a wordpress blog, and transform it for docpad.

Get Your Node On

mkdir blog-extract cd blog-extract npm init #accept all the defaults, it isnot very important npm install --save request mkdirp moment tumblr.js touch index.js

Edit index.js, and add the following:

var fs = require('fs'); var url = require('url'); var path = require('path'); var mkdirp = require('mkdirp'); var moment = require('moment'); var request = require('request'); var tumblr = require('tumblr.js');

Now we have a shiny new NodeJs project ready to go, with batteries (dependencies) included.

Wordpress Posts API

Wordpress exposes a JSON API that allows you to extract your posts. There is almost no set up required, as no form or authentication is required.

In order to get our posts, we can follow these instructions.

Extract and transform

With the API documentation in hand, we can now write some code to automate that - we certainly do not want to be issuing multiple wget or curl calls, and then copying the results from them into new files. I would do that for maybe a couple of posts, but I am dealing with about 80 posts here, and that is certainly going to be too time consuming of an endeavour!

var pos, step, total; var wordpressSite = 'yourblogname.wordpress.com'; // replace with your own pos = 0; step = 20; total = 0; do { /* * Here we do the queries, and be sure to set total so that it loops more than once * The looping is necessary, because you cannot download all posts at once * and we must paginate the requests */ } while (pos < total);

That is the basic run loop. Within the run loop, we perform the actual query:

var reqUrl = 'https://public-api.wordpress.com/rest/v1/sites/'+wordpressSite+'/posts/?number='+postsAtATime+'&offset='+postIdx; request(reqUrl, function(err, resp, body) { if (err || resp.statusCode !== 200) { console.log(err); return; } body = JSON.parse(body); if (body.total_posts > total) { //set total count, should only happen the first time total = body.total_posts; } //parse each of the posts in the response body.posts.forEach(function(post) { //transform the post into the format required by docpad //and write to file }); });

We can take a look at what the API response for each blog post looks like in these instructions.

The format that we need to translate to consists of two important parts:

Directory and file name

Metadata

The third part is the post's contents, but that can be copied verbatim without any transformation.

For a default docpad blog configuration, this would usually be: src/documents/posts/slug-for-this-post.html

We get check this by looking at docpad.coffee, and inspecting docpadConfig.collections.posts:

`@getCollection('documents').findAllLive({relativeDirPath: 'posts'}, [date: -1])`

We are however, not going to put our extracted files in the posts folder, and put them in a wordpressposts folder instead. Instead we will create a separate folder for all the wordpress posts, and configure docpad to look there as well. This configuration will be covered at the end, so if you want to test things out right away, skip to the bottom of the post

I am using the plugin, docpad-plugin-dateurls, so the URL paths of each of the posts is will match the default wordpress URL paths. Here, we want the directory and file name to follow this pattern: src/documents/wordpressposts/YYYY-MM-DD-slug-for-post.html

var postUrl = url.parse(post.URL); var pathname = postUrl.pathname; if (pathname.charAt(pathname.length - 1) === '/') { pathname = pathname.slice(0, -1); } pathname = pathname.slice(1).replace( /\//g , '-'); var filename = path.normalize('src/documents/wordpressposts/'+pathname+'.html');

For the metadata, we use moment to format the date and time

var title = post.title && post.title.replace(/"/g, '\\"'); var date = moment(post.date).format('YYYY-MM-DD hh:mm'); var tags = Object.keys(post.tags).join(', '); var contents = '---\n'+ 'layout: post\n'+ 'comments: true\n'+ 'title: '+title+'\n'+ 'date: '+date+'\n'+ 'original-url: '+post.URL+'\n'+ 'dateurls-override: '+postUrl.pathname+'\n'+ 'tags: '+tags+'\n'+ '---\n\n'+post.content;

Finally, write the output to file:

var dirname = path.dirname(filename); mkdirp(dirname); fs.writeFile(path.normalize(filename), contents, function(err) { if (err) { console.log('Error', filename, err); return; } console.log('Written', filename); });

Tumblr Posts API

Tumblr is a little more involved than Wordpress, as in order to query any of their API, you will need to have a tumblr account (which you probably already have since you are extracting your posts from it), and [register a tumblr app] to obtain an API keys. Copy your "OAuth Consumer Key", and you are good to go.

Once that is done, we simply need to follow this section in the documents. The upside ofthis slightly higher complexity is that tumblr provides a NodeJs client library that makes it easier to call the tumblr API, and avoid having to deal with making raw HTTP requests, like we did for the Wordpress API.

Extract and transform

var tumblrSite = 'bguiz.tumblr.com'; // replace with your own var client = tumblr.createClient({ consumer_key: 'sfsdfsdfsdfjkjksjdfhkjkjhkjshdfkjhkjhskdjfhkjhkjhd' //replace with your own }); pos = 0; step = 20; total = 0; do { /* * Perform the paginated requests */ } while (pos < total);

Performing the requests:

client.posts(tumblrSite, { offset: pos, limit: step, }, function(err, data) { if (err || ! data) { console.log(err, data); return; } if (data.total_posts > total) { //set total count, should only happen the first time total = data.total_posts; } data.posts.forEach(function(post) { //transform the post into the format required by docpad //and write to file }); });

Here, we want the directory and file name to follow this pattern: src/documents/tumblrposts/YYYY-MM-DD-slug-for-post.html

var ts = moment(post.timestamp*1000); var postUrl = url.parse(post.post_url); var dateStr = ts.format('YYYY-MM-DD hh:mm'); var filename = 'src/documents/tumblrposts/'+ts.format('YYYY-MM-DD')+ '-'+postUrl.pathname.split('/').slice(-1)[0]+'.html';

For the metadata, we want to set the dateurls-override property. Note that this feature is not yet available on in docpad-plugin-dateurls, and you will need my patch for this to work. To get this, modify package.json in your root folder, replacing the version number of the plugin with an explicit git URI, like so:

"docpad-plugin-dateurls": "git+ssh://[email protected]:bguiz/docpad-plugin-dateurls.git#exclude-option",

This tells npm to install a NodeJs package, not from the default npm repository, but instead by cloning a git repository. Unfortunately, this also means docpad will not be able to run the plugin yet, as npm installing a git url does not run prepublish. To work around this, for now, you need to do the following:

npm install docpad run # fails "Error: Cannot find module 'node_modules/docpad-plugin-dateurls/out/dateurls.plugin.js'" cd node_modules/docpad-plugin-dateurls cake compile ls out #you should see dateurls.plugin.js cd ../.. docpad run # success!

For tumblr posts, the default URL path follows the format /post/12345678/slug-for-this-post, and if we migrate posts from the old blog to the new blog, any links, especially extrenal ones, to the site will be broken. That will make for a really annoying experience for those visiting your sites, so it is best to preserve URLs where possible; hence the need to override the default URLs.

var title = post.title && post.title.replace(/"/g, '\\"'); var tags = post.tags.join(', '); var contents = '---\n'+ 'layout: post\n'+ 'comments: true\n'+ 'title: '+title+'\n'+ 'date: '+dateStr+'\n'+ 'original-url: '+post.post_url+'\n'+ 'dateurls-override: '+postUrl.pathname+'\n'+ 'tags: '+tags+'\n'+ '---\n\n'+post.body;

Finally, write the output to file:

var dirname = path.dirname(filename); mkdirp(dirname); fs.writeFile(path.normalize(filename), contents, function(err) { if (err) { console.log('Error', filename, err); return; } console.log('Written', filename); });

Docpad Configuration Changes

We edit docpad.coffee, in the root directory of the docpad project. Modify docpadConfig.collections.posts to look like this instead.

@getCollection('documents').findAllLive({relativeDirPath: {'$in' : ['docpadposts', 'tumlrposts', 'wordpressposts']}}, [date: -1])

All the wordpress posts should be in src/documents/wordpressposts, tumblr posts in src/documents/tumblrposts. When writing any new docpad posts save them in src/documents/docpadposts.

If you have any docpadConfig.environments configured, be sure to modify each of their collections.posts accordingly too.

That is all there is to do for now. Execute docpad run, and visit the newly extracted blog in a browser!

Where to from here?

One task in blog extraction, that we have not covered here, is that of any static assets, such as images, that may have been hosted on your previous blogs. Most notably, images. If you have hosted these on CDNs, they will continue to work. Otherwise, you will need to extract them too.

Another extraction task that we have not covered are links between posts. Since we have preserved the path for each post's URL here, this should not pose a problem.

I currently write my blog using tumblr, and previously I blogged using wordpress. While both of these are great platforms, they share common pitfalls, when it comes to giving you control over your writing.

I wanted to be able to have a copy of all the assets that comprise my blog, in its entirety, on my hard disk, and be able to modify and publish them as I pleased. I also wanted to be able to include fancier things in my pages - like embed a Github gist, or create my own d3 visualisation, or, well why not take it to an extreme, create an AngularJs app running within one of my posts; and I wanted to be able to do all of these things without having to log into some website hosted in a far away country, and wait for all those bytes to fly across several oceans and back each time.

Flexibility and control - that is key.

Enter Static Site Generators

For a blog, the contents are almost static. The server only needs to send a different response for a page, when that page has been modified by the author. The exception to this are comments, but with the advent of disqus, that is no longer even a consideration.

A content management system, including both tumblr and wordpress, builds each page upon demand, which can be an expensive operation, as it involves database queries, assemlby of templates, et cetera. Quite often, when a CMS driven site receives a lot of concurrent visitors, its response times start to lag noticeably. To work around this, it has become common practice to cache the results of each dynamically generated page, using tools like memcached.

Static site generation is all about taking caching to the next level. The author of the site knows exactly when the previous cache needs to be invalidated - when they write a new post or update an existing one. Why not, at that point of time, generate the cache contents, and upload them directly to the server? Well, that is exactly what static site generators do; the static files are the cache

What about collaboration?

One of the big advantages of a CMS is that it enables collaboration. If everyone just logs into the same website, be it wordpress.org or tumblr.com, and made their edits on the site, then there is only one copy of the site, and therefore it is easy to manage collaboration on the contents of the site.

Indeed that is a very direct and simple solution that addresses collaboration. We do, however, have a more sophisticated solution, that is already readily available: distributed version control systems. Tools such as git and mercurial have solved the distributed collaboration problem in a rather elegant way. All collaborators get to keep a copy of the site that they are contributing to on their own computers, and thus get the benefits that come along with that. When they are done writing a post, they simply have to push their latest contributions to the master copy. There are built in mechanisms to resolve any conflicts, for example, if two collaborators edit the same file.

Docpad

After reviewing the top few in this humungous list, I have decided that Docpad suits my needs the best, and I should be able to hit the ground running. I will give it a go, and the best part is, if I do not like it, my data is not stuck on some server somewhere - it will all be on my computer, and easily moved to a different static site generator.

In a website which uses session-based authentication, when a file needs to be downloaded, and that file should only accessible by the currently logged in user, making that work client side in a web page is extremely easy. That is because the session credentials are typically stored inside cookies, and the browser automatically adds the cookies to every HTTP request's headers for that domain.

When you create an anchor tag, and set its URL to point to the route that responds with the file to be downloaded; and this anchor tag is clicked, that file will get downloaded, as the authentication requirement for that route is satisfied by the cookie that gets automatically added to the HTTP request header by the the browser.

However, it is not quite so simple if the website uses token-based authentication. This is because browsers do not have any mechanism where it can be told to add the token to each HTTP request's headers across the board.

Let us say that you do the same thing as before: create an anchor tag, and set its the URL to point to the route that responds with the file to be downloaded. The only difference is that this time, that route requires token in the header, and there are no cookies involved. Now when you click on this anchor tag, the authentication requirement is not met, and the file does not get downloaded.

My instinctive reaction to this was to find out a way to add the token to the header of the HTTP GET request that gets sent upon clicking the anchor link. It turns out, however, that there is no way to do this; there is simply no way to intercept that request and modify it before it gets sent.

So I asked this question on Stackoverflow.

The only way to add a header to any HTTP request is using AJAX - by creating a XMLHttpRequest. However, the catch is that you simply get the data in a JavaScript variable in the callback function when the AJAX response arrives. It does not trigger a file download, like clicking an anchor tag would.

How to set a header for a HTTP GET request, and trigger file download? http://t.co/u7PnurXDjM

— Brendan Graetz (@bguiz) July 1, 2014

How do we get around this? Turns out that there are a couple of rather creative solutions to the problem.

Set a header on the HTTP request and have that trigger a file download with the response http://t.co/UIXr9CXXcO #javascript #html5 #solved

— Brendan Graetz (@bguiz) July 2, 2014

When the anchor tag is clicked, intercept the event, and initiate an AJAX request, being sure to add the appropriate token in the request header:

var id = 123; var req = ic.ajax.raw({ type: 'GET', url: '/api/dowloads/'+id, beforeSend: function (request) { request.setRequestHeader('token', 'token for '+id); }, processData: false });

When the response is returned, we use a temporary anchor tag when handling it:

req.then( function resolve(result) { var str = result.response; var anchor = $('.vcard-hyperlink'); /* transform the response into a file */ }.bind(this), function reject(err) { console.log(err); } );

Depending on the size of the response, and whether the browser is modern enough to support HTML5 File APIs, we either use base64 encoding or temporary files.

Using HTML5 temporary files:

var maxSizeForBase64 = 1048576; //1024 * 1024 var windowUrl = window.URL || window.webkitURL; if (str.length > maxSizeForBase64 && typeof windowUrl.createObjectURL === 'function') { var blob = new Blob([result.response], { type: 'text/bin' }); var url = windowUrl.createObjectURL(blob); anchor.prop('href', url); anchor.prop('download', id+'.bin'); anchor.get(0).click(); windowUrl.revokeObjectURL(url); }

Using base64 encoding:

else { //use base64 encoding when less than set limit or file API is not available anchor.attr({ href: 'data:text/plain;base64,'+FormatUtils.utf8toBase64(result.response), download: id+'.bin', }); anchor.get(0).click(); }

In both cases we set the anchor tag to a data URI or file URI, and then trigger a click event on it.

The caveat for this however, is that both of these approaches are going to be rather inefficient when downloading and processing large files. More so for the base64 encoding method than the HTML5 File API method.

One way of solving this problem is to modify the server such that the route that requires the token in the HTTP header does not respond with file contents, but instead with the URL of a another route, which does not require anything in the header at all, but expires very quickly. It is this route which actually returns the file contents.

Recently, I released broccoli-sprite. I was just a week into using BroccoliJs for the first time, and writing a plugin for a build system that I had barely used was understandably tricky.

While writing it, I googled quite a bit for how to write a BroccoliJs plugin, but there really has not been much written about it. I would like to make it easier for others doing the same thing, so here is a quick overview of the process of creating a BroccoliJs plugin.

Basics

What is BroccoliJs? Think [GruntJs], but different. Different how? Well in a number of ways. Its creator, Jo Liss, [explains its philosophy in her post on its release].

tl;dr= Plugins can chain their output to one another, and the built-in watch only rebuilds what has changed rather than the whole lot.

There is one more thing to it: If you are building an app using ember-cli, you will need to use BroccoliJs.

Sold! Now Time to Write a BroccoliJs Plugin

The first thing to take a look at is the plugin API specification. Looks very straight forward: There are just two things that you need to implement: tree.read()and tree.cleanup(). The former, however, does not really do much that is useful, on its own at least.

Getting started

All BroccoliJs plugins are NodeJs modules that should be installed inside a project

cd my-project/ npm install --save-dev broccoli-my-plugin

… and thus the first step is to create an npm package:

mkdir broccoli-my-plugin #replace `my-plugin` with the name you would like cd broccoli-my-plugin/ #if you plan to use version control (which is a good idea), do it now, e.g. #git init && git flow init npm init #this creates `package.json` #be sure to specifiy one of the keywords as `broccoli-plugin`

Now you will need to edit index.js, which exports your plugin:

var BroccoliMyPlugin = function BroccoliMyPlugin() {}; modules.exports = BroccoliMyPlugin;

Extending an Existing BroccoliJs Plugin

BroccoliJs has several plugins, that are designed to be extended. The one that we will look at here is broccoli-writer.

Install it:

npm install broccoli-writer

Edit index.js:

var brocWriter = require('broccoli-writer'); var BroccoliMyPlugin = function BroccoliMyPlugin() { if (!(this instanceof BroccoliMyPlugin)) { return new BroccoliMyPlugin(); } }; BroccoliMyPlugin.prototype = Object.create(brocWriter.prototype); BroccoliMyPlugin.prototype.constructor = BroccoliMyPlugin; BroccoliMyPlugin.prototype.description = 'my-plugin'; modules.exports = BroccoliMyPlugin;

Here we have simply extended the function exported by broccoli-writer using prototypical inheritance. At the moment it does not do anything at all, and we will add that next.

Adding functionality

Firstly, we should make the plugin able to accept some input parameters. All BroccoliJs plugins must accept an input tree as its first argument. Any subsequent parameters are completely up to you as the plugin developer. A common pattern, however, seems to be to accept just one parameter, and options hash, which is what we will do here.

var brocWriter = require('broccoli-writer'); var BroccoliMyPlugin = function BroccoliMyPlugin(inTree, options) { if (!(this instanceof BroccoliMyPlugin)) { return new BroccoliMyPlugin(inTree, options); } this.inTree = inTree; this.options = options || {}; }; BroccoliMyPlugin.prototype = Object.create(brocWriter.prototype); BroccoliMyPlugin.prototype.constructor = BroccoliMyPlugin; BroccoliMyPlugin.prototype.description = 'my-plugin'; modules.exports = BroccoliMyPlugin;

We add the inTree and options parameters to the constructor function, and then save them in the instance. If you wish to specify default options, or other instance variables, this is where you would parse and set them.

Next we can implement the main functionality, the part where we specify the thing that this plugin does. Since this plugins extends the broccoli-writer plugin, we do this by specifying a writefunction:

BroccoliMyPlugin.prototype.write = function(readTree, destDir) { var self = this; return readTree(this.inTree).then(function (srcDir) { /* use srcDir and information from self.options to figure out which files to read from */ /* use destDir and information from self.options to figure outwhich files to write to */ /* synchronously read input files, do some processing, and write output files */ }); };

readTree is passed in as the first variable to the write function, and this is a function that returns a promise that you should return. Callthen()`on the promise, and do the processing in the callback function. Here you do whatever it is the plugin needs to do; but you have to do it synchronously - no callbacks allowed.

Asynchronous Plugins

Most of the time however, we want to do things asynchronously - after all, that is the NodeJs way! See Mixu's article on control flow in NodeJs for an excellent introduction to asynchronous code in NodeJs. We need to get a little more advance than this however, and use promises instead of callbacks. Not to worry though, promises are actually much more straight forward to use than callbacks! In fact, we have already used the one returned by the readTree function previously.

We shall use promises implemented in the RSVP library, as that appears to be the most popular choice amongst Broccoli plugins; although you are free to use any other promise library.

Install RSVP:

npm install --save rsvp

Include RSVP:

var rsvp= require('rsvp');

Modify the readTree callback to create a promise an return it.

return readTree(this.inTree).then(function (srcDir) { /* use srcDir and information from self.options to figure out which files to read from */ /* use destDir and information from self.options to figure outwhich files to write to */ var promise = new rsvp.Promise(function(resolvePromise, rejectPromise) { /* asynchronously read input files, do some processing, and write output files, for example, here we have `someAsyncFunc` that does this` */ someAsyncFunc(function(err, asyncData) { if (err) { rejectPromise(err); } else { resolvePromise(asyncData); } }); }); return promise; });

Here, since we return a promise, BroccoliJs knows to wait until it is either resolved or rejected. For the more astute, you will notice that here we actually have a promise within a promise, as readTree itself returns a promise. We could possibly refactor this to chain the promises instead of nesting them, but I shall leave that as an exercise for the reader!

Fin

Now we have a functional BroccoliJs plugin, and it is ready to be published:

npm publish

… and now anyone can npm install it!

Going further

Besides broccoli-writer, there is also broccoli-filter, and broccoli-caching-writer, which I have not covered here.

Depending on what your plugin does, you might want to extend these instead. One great way to learn more about writing BroccoliJs plugins is to search for existing ones, and examine the source code for each one. Most of them are fairly simple, only containing a single index.js file, which means that you will likely find what you are looking for rather quickly. In fact, that is precisely what I did to get up to speed, when writing brocoli-sprite.

After much frustration attempting to get broccoli-compass to work, including asking a question on Stackoverflow, and getting no responses despite offering a bounty on it, I decided to bite the bullet and write my own BroccoliJs plugin.

It was a little tough, because I am completely new to BroccoliJs, having used GruntJs up until now. In order to build an app using ember-cli, however, switching to BroccoliJs was a necessity.

So I present broccoli-sprite, hot off the presses.

To install it:

npm install broccoli-sprite

To use it:

var broccoliSprite = require('broccoli-sprite'); var spritesTree = broccoliSprite('public', { src: [ 'public/images/sprites/*.png' ], spritePath: 'assets/sprites.png', stylesheetPath: 'assets/sprites.css', stylesheet: 'css', stylesheetOptions: { prefix: 'sprite-', }, });

Stefan Penner converted Ember App Kit to ember-cli about a month ago, and its feature list is pretty amazing, and I thought that it was about time to give it a whir.

When developing on beta, bleeding edge, software libraries; and when beginning to use a completely new tool chain - in this case I was switching from GruntJs to BroccoliJs; however, ones tends to run into quite a few walls!

Here are all the questions I have asked on Stackoverflow, and issues/ requests/ pull requests I have created on Github, in the past couple of days:

Stackoverflow

How to add broccoli-compass to ember-cli v0.0.28?

How to get test to access ember-cli server when run in CI mode?

Ember Data deleteRecord() followed by rollback() - how to make object reappear in list?

Creating CSS sprites using SASS without Compass?

Setting up integration tests in an ember-cli app - how to access module() and visit()?

Github

Testing Ember with Mocha - visit() causes tests to abort

Allow SASS sourcemaps

Support for CSS source maps and providing options to preprocessors

Add instructions for setting up integration tests

explore CI/tests running against compiled + minified production builds

ember test works in Linux, but doesn't on Windows

Improve documentation for usage

Include instructions on how to use with Broccoli

So I was tasked with documenting how to set up a development environment in Windows.

Here is my favourite part:

I was looking for a way to combine images together programmatically, and spent some time dabbling around in gimpfu python land without getting very far at all.

Who would have thought just placing one image next to another could get so hairy?

What if I said that it was possible to create the above image from three images using a single command line? Read on!

I had one of those "Aha!" moments of discovery when I chanced upon this articles: Imagemagick montage usage.

montage foo.png bar.png -geometry 200X200+2+1 foobar.png

No script involved, just a single command line; and that resizes both foo.png and bar.png to 200 by 200 pixels, arranges them into a two by one grid, adds some default spacing between the two, and saves the output to foobar.png.

If you want the background to be transparent, and to add a dropshadow, like I did, it is as simple as:

montage foo.png bar.png -shadow -background None -geometry 200X200+2+1 foobar.png

Modernizr is an an excellent library used for client-side feature detection.

The full set of features detected can be obtained using its development version, however, including this entire file is not ideal in production - why make users download all the extra code for feature detection tests that will not ever be used?

Thankfully, Modernizr has a nifty download feature, which lets you fill in a form to select which tests you want, and then it generates a customised file for you to download.

How about a CLI?

+ = ?

That is great, however, what if you want to use Modernizr to generate custom files on the command line?

I came across this closed ticket raised against Modernizr, and that helped somewhat, as I found a way to generate custom files on the command line, but the command I had to issue was extremely convoluted.

Seeking an easier way, I came up with this, which was a huge improvement in usability.

Now it is even easier:

npm install -g modernizr-cli modernizr custom.json

(Where custom.json should conform to this format)

Ever struggle to remember what the plots for common math functions look like - well, I have got just the GIF for you!

… not interested in the math, just want to see the guy dance?

… and the obligatory epilepsy-inducing edition:

Created with <3 in gimp

Recently, I needed to make a single-page web app that would be served by a Java backend. That would be as simple as copying the static files into the relevant folder, right? Well, as it turns out, not quite, as each project used a different build system.

Single-page web apps are typically built using the Grunt and Bower toolchain, which are NodeJs scripts. Java backends are typically built using either the Ant or Maven toolchains.

Pick one, or make a bridge?

So, should we replace the build tool of with the build tool of the other, or should we just make one of them talk to the other? Addy Osmani has an article, Making Maven Grunt, whichdescribes this situation quite succinctly:

"More times than not, it's the back-end team who will define the build process and end up having to wrangle your front-end code into there."

"Whilst there are many plugin options for JavaScript processing in the Maven world, these plugins can be a real chore to create, maintain and update. In addition, they often have outdated dependencies – meaning you need to monkey-patch them to work and developers – especially those not coming from a Java background – dislike it."

Thus it appears that in order to get the best of both worlds, the best option would be to create the bridge that allows one build system to talk to the other. In this case, to make maven invoke grunt.

The options

Turns out that this is a problem that several have had, and solved, before.

yeoman-maven-plugin

maven-antrun-plugin

exec-maven-plugin

If you your web app is generated by yeoman, then the yeoman-maven-plugin provides some sensible configurations that work out of the box. However, if you need something a little more flexible, the maven-antrun-plugin, or the exec-maven-plugin is your best bet.

However, I did not want to throw Ant into the mix, as I was already using two other build systems, and their syntaxes - and thus reluctant to throw a third syntax into the mix; thus exec-maven-plugin it was!

Why don't my plugins run?

I asked a question on Stackoverflow: "Maven exec-maven-plugin and maven-resources-plugin not running, not sure why"

Excuse the newbie question on this one - this was my first time using maven after all!

When including a maven plugin, you define a list of <executions>. I thought that defining a <phase> for each <execution> was sufficient to trigger them. I learnt that it simply was not. It is necessary to define at least one <goal> for the plugin to get invoked.

Why can't my plugin find executables?

A follow up question, also on Stackoverflow: exec-maven-plugin says cannot run specified program, even though it is on the PATH

Now this one was not a newbie question at all - it turned out to be a Windows-specific bug.

Basically, even though entering grunt on the command line worked, as it was on the path, when the maven plugin ran, it failed to find it, much to my puzzlement.

After much tinkering, and inspecting the source code of exec-maven-plugin, I found what I think is a bug in the plugin:

CommandLine toRet; if ( OS.isFamilyWindows() && exec.toLowerCase( Locale.getDefault() ).endsWith( ".bat" ) ) { toRet = new CommandLine( "cmd" ); toRet.addArgument( "/c" ); toRet.addArgument( exec ); } else { toRet = new CommandLine( exec ); }

It assumes that only commands that invoke .bat files need to be prepended with cmd /c.

I checked the npm path, where the links to all the global installed NodeJs libraries are, i.e. the ones installed using npm install -g, and found that it had, for each, a Unix shell script, with no extension, and a Windows batch script, with a .cmd extension.

Fixing the problem

Normally I would have fixed the problem by patching the bug in exec-maven-plugin. However, it was using a svn instead of git, and using xircles instead of github. I was not new to Subversion, but had not used it before in the context of a open source contributions. Prior to github the way I contributed to open source projects was:

svn diff > bug-that-this-patch-should-fix.diff

Attach bug-that-this-patch-should-fix.diff to the issue tracker, forum, or email the maintainer of the project.

I have since been spoilt by git's cheap branching, and the ease with which pull requests can be managed in github's interface.

I did give it a go, nonetheless, but I the web interface took more than a minute to load (kid not!), and that was all it took for me to start looking for another way to solve the problem.

A quick and easy way was to simply rename the .cmd files to .bat files, since they were batch script files anyway, and everything magically began working once more.

See my answer for a more detailed explanation.

Rant over

That has been half rant, half description of my learning experience. I must say, I wish that a rather simple task like this were a little easier.

I came across this interesting write up on a star rating system with SASS.

I happened to be creating something with a star rating system myself, so I decided to incorporate this. Unfortunately, I found it to be a bit clunky, requiring classes to be set in two places - once on the <ul> and once on the <li> - in the case of a half star.

While this was possible to do in an AngularJs directive or an EmberJs Component, the code for it proved to be a little unwieldy for my liking. I wanted to simply set one class, on the <ul>, and not have to do anything else.

So I came up with some improvements of my own. And in gist form.

EDIT: Just noticed that my comment has been marked as spam - please fix!

It begins with this clip, aired on television, in the early 70s

It is one of the skits from Flying Circus. Since then, it has made its rounds on Wikipedia & IRC. Now in real life, in a supermarket, too.

I have given talks on qryq a few times now, and, quite often I have been asked, about how to actually begin using qryq.

qryq is a general purpose library, and it can be used in a number of different ways. The most common, or default, use for it is to expose a single API endpoint on a server, where any number of APIs can be invoked in a query queue specified by the client. Thus I have created a template server that does just this, and have (creatively) decided to name it qryq-server.

This project is most useful as a scaffold or template, where you simply make a copy of it, and in most cases merely need to add more methods to the api object.

Check out qryq-server on github

Last week, I had the opportunity to attend two conferences back to back: Open Source Developer's Conference 2013 in Auckland, and Web Directions South 2013 in Sydney. Being in a profession where I am sitting in front of a desk all day, going on the road for a week mixed things up quite a bit. Plus, preparing to talk at one added to the stress, and to the lack of sleep.

What a great experience though, I have been put through an accelerated learning curve on a diverse range of topics. They ranged from the very technical, such as dissecting the GPU's role in rendering CSS using layers; to the more fun, such as how to generate a choose-you-own-adventure game in pastebin; to the unexpected such as a crash course on 3D modelling using Blendr.

Open Source Developers Conference 2013

This was my first time in New Zealand. I had a great time exploring Waiheke island on Sunday - a couple of days before my presentation - while running through the presentation in my head. It was pretty much a picture postcard view every couple of minutes along the coastline. At some point in this rather nice walk, inspiration struck on qryq. That culminated in these code snippets, and was the basis for re-jigging about half my presentation.

Web Directions South 2013

It was a great experience attending what felt like the biggest tech conference in Australia. It was also great to see a few familiar faces presenting: Mark Dalgleish, Ryan Seddon, and Glen Maddern. The talks here were grouped into two streams, code and design - which made it easier to select which ones to attend.

Attending next year?

OSDC focussed a lot more on programming; and its scope was very broad, covering many different technologies, and languages. Web Directions South focussed a lot more on design; and its scope was more specific, covering only technologies in the web stack. Worth mentioning that Web Directions has a sister conference, Web Directions Code which is more focussed on programming, but still focussed on technologies on the web stack.

On attending conferences

This being the first time I have attended large tech conferences, I have learnt much not just about the subject matter of the talks given, but also much about the other aspects of them. When attending them, one wears different hats - traveller, attendee, and speaker - and here are a few things I took away about maximising the time spent at them.

As a traveller

Travelling to a conference is not a lot different to travelling for other reasons. The main thing to remember is that being on conference is like going to work, and thus not very relaxing. Pro-tip: Do not book a hectic flight schedule!

As an attendee

Attending a conference is mostly about sitting down and listening to someone speak. Making the most of this experience is a balancing act. Listen to the speaker; watch the screen projection; take notes; follow the twitter feed; reflecting about how what you have just taken in can be applied to your own work. Juggling all of that can be tricky.

I found that the easiest thing to do is to leave out the twitter feed, except for right at the beginning, and right at the end of the talk. Save reflections and developing ideas for application till after the talk, and just jot the ideas down quickly when you have them. Most importantly, take notes; and when doing so, avoid writing down anything that is on the slides, and instead what the speaker is saying, as usually you can find the slides afterwards anyway.

One thing about conferences is that they often run multiple concurrent tracks. OSDC had three, and WDS had two. While it is great that you get to pick your favourite ones to attend, sometimes there are two that you would like to attend that happen to run at the same time, and a tough choice needs to be made. What I learnt here was that it pays to do your homework - research the topic presented, and what the presenter has written about it prior.

While attending the presentations are what you will spend most of your time doing, it is important not to neglect the other parts. In between the presentations, there will be time to mingle, and that should be used well. Spend it talking to the presenters and the other attendees; usually they are looking to do the same themselves!

Oh yes. Make sure to remember to bring an extra long power cord!

As a speaker

Speaking is hard, mostly because of the need to prepare. Once at the conference, prepare to completely rework your slide deck, because new ideas will occur; as will last minute feel-the-need-for-improvement frenzies. In fact, I redid half of my presentation the day before I spoke.

On my first day in Auckland, I was a little lost walking around the city, and asked for directions from a stranger, who was happy to walk with me as he was going in the same general direction. Turned out he was a budding web developer, and I told him about OSDC - he had not heard of it - and he bought a single day ticket, and came to see my talk.

Something that I thought I did quite well (if I do not say so myself) was to use the breaks in between presentations, and other opportunities, to talk to other attendees, and mention what I was going to talk about to the other attendees and speakers. That gathered quite a bit of interest, and resulted in a full house during my talk, and even had some expressions of interest in using and porting qryq.

Another thing, which I have learnt from one of the other speakers was to bring marketing materials. He handed out one page print-outs of the software he was speaking about. I thought that that was a brilliant idea - so much better than just a name card - keep that in mind for the next time.

Watch this space