Memory Leak

After spending hours today wrestling with a shell script that should have been pretty simple (running a sequence of cURL requests, using some of the response headers from earlier requests in later ones); I was able to rewrite the whole thing in Python in about 10 minutes.  That included the time it took to learn Python along the way, as this was my first ever *.py script.

Aside from being much faster to write, the end result was:

infinitely more readable than arcane bash syntax (case…esac.  What the?)

easier to turn on/off debugging messages (using the logging module and specifying a logging level; rather than uncommenting/commenting ad-hoc echo's sprinkled throughout the script)

In the so-called "Ruby vs Python War", I've traditionally come down on the Ruby side; for no other reason than I'd never previously used Python, and I liked Ruby's syntax (and Rack/Rails etc.).

I've written my fair share of Ruby/Rack/Rails apps (both personally and professionally), and I'd never really seen a need to learn or use Python. I'm also a Git/GitHub user, which seems more Ruby focussed; where as Mercurial/BitBucket seems more in the Python camp. (These are obviously sweeping generalisations...not hard facts :).

Recently, however, I attended the linux.conf.au 2012 conference in Ballarat, Victoria.

Two observations I made at the conference were:

Nearly every talk (it seemed) given over the 4-day conference included at least one reference to Python.  This included big name projects (eg. Samba).

Not a single reference to Ruby was heard.

From this, one can deduce that in the Linux community, Python is not only preferred over Ruby, but has become the defacto scripting language for any script of a reasonable size.

Conversely (at least anecdotally), Ruby seems to be much more prevalent in the Mac OS X community where it seems all, or at least most of, the prominent Ruby hackers develop on a Mac (again, sweeping generalisation...I'm sure there are plenty of Pythonistas that use Macs too). The fact that both Ruby and Python are installed by default in OS X, where as only Python is bundled in most Linux distros also hints at this.

As an aside (nothing to do with Ruby vs Python), all this talk of Linux and Mac reminds me of a Tweet I read once (sorry, no attribution...as I can't recall who said it; and I'm paraphrasing here anyway):

Linux, Mac and Windows all have their uses.

Macs are a great for developing web applications.

Linux is great for hosting web applications.

Windows is great for testing IE.

Personally I use Mac OS X, Linux (Ubuntu 11.10) and Windows all on a daily basis. On my desk in front of me right now I have all three running: I'm writing this on a MacBook Pro, next to that is a Linux box (running KDE) which is also acting as a host for a Window VM.

After today's Python experience, regardless of what platform I'm using at the time, I'm certainly not going to limit myself to just Ruby any more.

UPDATE  22-Jan-2013:

As Marcelo Cantos notes over on www.html5rocks.com, WebSQL performance in this experiment can be significantly improved through the use of an additional index (on the Episode table), and a slight modification to the SQL query (replacing the HAVING clause with a nested subquery).

I would strongly recommend reading Marcelo's comments and bearing this in mind when weighing up the relative performance; as it changes the findings discussed below.

TL;DR

At first glance, the IndexedDB API appears to require a lot of extra code to accomplish tasks that could be done in a single query with WebSQL; as familiar concepts such as joins, aggregates etc., that we take for granted in the relational world, need to be handled manually. Such multiple steps also suggest that IndexedDB must perform worse than the equivalent WebSQL query. Surprisingly, IndexedDB performance stacks up well against WebSQL and is, in certain circumstances, actually faster.

Background

Since November 2010, when the W3C officially deprecated WebSQL, my intention has been to try out it's successor, IndexedDB. In particular, I wanted to see how IndexedDB performs in comparison to a real-world WebSQL-based application.

Data Model

The data model for this exercise is a simple, three-tier, parent-child hierarchy based on television programs.

  A Program consists of zero, one or more Series, which in turn consist of zero, one or more Episodes. In relational terms, these are both one-to-many relationships from parent to child.

Sample Code

The source code for this exercise can be found on GitHub.

Note:

IndexedDB (at least in Chrome's implementation) does not work when the page is accessed using the file:// protocol. To run this sample application, you will need to access it though a HTTP server.

A simple way to do this, if you have python installed, is to run

python -m SimpleHTTPServer 8000

from the source directory, then browse to http://localhost:8000/indexedDbTest.html

 Let's quickly go over the basic code structure, before delving into the database stuff...

The main page is indexedDBTest.html, which references indexedDBTest.css for styling the results, and indexedDBTest.js file for running the sample app.

indexedDBTest.js consists simply of a jQuery ready function that instantiates the two data stores (more on these shortly), and listens for the "Go!" button to be clicked to run the tests.

A pair of booleans at the top of the script (initWebSqlDb & initIndexedDb) control the initialisation of the two data stores. For the initial setup, you will need to set these both to true, to create & populate the WebSQL database and IndexedDB object stores with sample data when the page loads. After this is done once, these can be switched back to false so that you're not rebuilding the world on each page refresh.

Each data store is populated with an identical data set: 100 x programs, each containing 6 x series, each containing 11 x episodes; giving a total of 600 x series records, and 6600 x episode records. Not a huge amount of data by normal DB standards; but enough to compare the relative performance of each store. The findings of this exercise may not hold true for a larger data set, so bear that in mind. I've kept it small on the basis that most apps that store data locally in the client typically don't store an enormous amount.

Data stores

The two remaining files are indexedDb.js and webSQL.js, which are wrappers for their respective data stores, and both have the following interface:

init() - initialises the data store (if required) and populates it with the test data

start() - used in conjunction with stop(), records the start time of an operation

stop() - calculates the duration since start() was called; and renders the operation results/duration.

countPrograms() - counts the number of Programs in the database (ie. 100)

countSeries() - counts the number of Series in the database (ie. 600)

countEpisodes() - counts the number of Episodes in the database (ie. 6600)

schedule() - a 'real world' query requiring joins, aggregates etc.  (more on this later)

Counting objects

One of my initial concerns with IndexedDB was how it would handle aggregates (counts, sums, averages etc.). Most of the IndexedDB tutorials/examples I found online seemed very simplistic, showing basic tasks such as putting an object into a store, getting an object out of a store, and using cursors to iterate over a set of objects; but there were no good examples of aggregating objects. Also, at that time, Chrome 17 had not yet implemented the IDBObjectStore.count() and IDBIndex.count() methods as per the W3C spec, so the only way to get a count of objects from an object store or index was through a cursor iteration.

To my 'set-based' SQL mind, the initial reaction to this was: computing aggregate values by iterating over a set of records/rows/objects is surely going to be slow.  Sure enough, counting the 6600 episodes (which took < 10ms in WebSQL with a simple SELECT COUNT(*) FROM Episodes) took > 900ms with an IndexedDB cursor.  Nearly 100 times slower.  WebSQL: 1, IndexedDB: 0.

Thankfully, I didn't have to wait long for Chrome 18 to be released through the Dev channel, which has native count() implementations. After replacing the cursors with these, the time dropped from > 900ms to ~150ms. WebSQL's ~8ms time still easily beats it, but clearly the native count() methods are a massive improvement over cursor iteration.

From a coding perspective, let's compare the countPrograms() functions for both WebSQL and IndexedDB:

As you can see, there's not much difference. I'd even argue that the IndexedDB version looks more succinct. So far, even with IndexedDB's slightly worse count performance, I'm willing to call it a tie.

Joins

My other concern with IndexedDB was regarding relationships/joins. In the relational-world, when in comes to modelling database entities the rules are usually pretty clear cut (thanks to E. F. Codd's theory of normalisation). Conversely, the schema-less nature of NoSQL databases means that there is often more than one way to model your data. It's up to you, as a developer, to determine which model best suits your application.

For this exercise, there are three different approaches to modelling the data:

Separate object stores for each of the three entities (corresponding to the three tables in the WebSQL database), i.e. a "Programs" object store, a "Series" object store, and an "Episodes" object store. This is a many simple objects design.

A single object store for storing everything; and a "type" attribute on each object to distinguish between them. Programs, series & episodes would still be stored as separate objects, but all in the same object store.

A single object store that stores only Program objects.  Within each program object would be an array holding it's set Series objects, and within those nested Series objects would be an array containing it's Episode objects. This nested object structure takes full advantage of IndexedDB's ability to store complex objects; something that a relational database such as WebSQL cannot do. This is a few complex objects design, which works well for one-to-many relationships (but no so well for many-to-many relationships).

Each has it's pros and cons.  The first approach is probably most familiar to relational database designers, and has the benefit of providing very granular access to each entity's records, as well as the ability to use those native count() methods I mentioned earlier. The downside is that retrieving objects from multiple object stores (ie. the equivalent of SQL's "JOIN" keyword) has to be done by manually merging the results of multiple requests.

The latter approach sacrifices granular access to child objects in exchange for simplified joins (i.e. get a program object, and you automatically get all of it's child series and episode objects along with it).

For simplicity, I chose the first approach (separate object stores).

I wanted to compare the performance of a reasonably complex SQL query consisting of multiple joins, aggregates with a group by/having clause, an order by clause, and an in-line case expression.  An example of this from my tvmanager app is the initial view for the app called the "Schedule" view. The Schedule view lists any series that you either have, or expect to have, recorded to your DVR. The criteria for inclusion in the Schedule view is:

any series that is marked as nowShowing OR

any series that has at least one episode with a status of Recorded OR

any series that has at least one episode with a status of Expected

For each series in the Schedule view, we also include additional information from the parent program (ie. it's name), as well as some summary aggregate data from it's child episodes (total # watched, # recorded, #expected etc.), and finally we order the results by the day of week that they are shown, then by program name.

In WebSQL, we achieve all of this with a single query:

Note: there are no indexes of any of the tables in the WebSQL database. I did run some benchmarks after adding indexes on Episodes(EpisodeID, Status), Series(SeriesID) and Series(NowShowing); but there was little improvement, as the query essentially has to do a full table scan of the Series table.  But it's worth noting anyway.

The code for the schedule() function for WebSQL looks like this:

An equivalent operation in IndexedDB is much more complex, requiring 3 + 3*N separate database requests (where N is the number of series that meet our above criteria) to retrieve all of the information.   Let's look at each of these in more detail...

The first task is to identify the series objects that meet our criteria, which you'll recall has three elements.

Firstly, we need to get the list of series objects that are regularly airing (i.e. have a non-null nowShowing property), which is pretty easy as we have an index for that ("series_nowShowing"). So we open a cursor using a key range with a lower bound of 0, iterate over the cursor and populate an associative array.  We use an associative array here rather than a normal array, because it's likely that there will be duplicates when we get to our next steps, and we want to filter those out.

Secondly, we want the list of series objects that have at least one recorded episode. Again, we can do this using an index on the episodes store ("episodes_status"), and specifying a single key value of "Recorded".

Finally, we want the list of series objects that have at least one expected episode.  This is identical to the above, just with a different key value.

For performance reasons (and, let's be honest, because no browsers yet support IndexedDB's synchronous API), we can fire off each of these three requests asynchronously, and wait for a signal from each to indicate that it has completed (this is the processCandidates.call(this) at the end of each request).

Now that we have our candidate series, we need to loop through these and collect the remaining information required for display (e.g. program name, episode counts etc.). We start by getting the series object itself.

Note: You might notice that I'm still using openCursor to get a single series object. I'm not sure if it's a bug with Chrome, or I was doing something wrong, but I simply couldn't get get() to work (I kept getting a DataError exception, which generally means that the key you're attempting to get doesn't exist, even though I knew the key existed). Using a cursor seemed to solve the problem, but probably incurs some extra overhead beyond a simple get.

Once we've got the required values from the series object (including the programId it belongs to), we get the related program object (again, using openCursor() because of the issue with get())

While we're firing off a request to get the program, we can simultaneously do the request for the related episodes. We use a cursor to iterate over all of the episode objects for the series, and manually calculating the total/watched/recorded/expected counts.

As per earlier, at the end of each of our parallel requests we set a flag to indicate the request is complete, and call updatePopulatedCount(). Only when both requests for a given series are done will we update our progress counter.

Finally, once we have all the data we need, we sort the results in the same order as we did for the WebSQL version; before rendering the HTML table in the same way as we did for WebSQL.

The Results

As demonstrated above, the code for IndexedDB can be a lot more complicated, as we're having to do a lot of the work ourselves in JavaScript that is otherwise provided natively by SQL.

That said though, WebSQL takes on average between ~750-850ms to complete the query and render the results; and IndexedDB takes on average ~300-350ms to render the exact same results. Thats over twice as fast.

At this point I should add the disclaimer that

My timing method (diff between two JavaScript Dates) is not particularly scientific, and is at the mercy of the browser's main event loop. Given that we're measuring down to the millisecond, it's probably not 100% accurate.

The code could possibly be optimised more (either on the WebSQL or IndexedDB sides). I didn't spend a great deal of time squeezing every last ounce of performance out of it.

Amidst a flurry of announcements from Twitter last week was the release of the new TweetDeck for Mac.

As a user of the old Adobe AIR based TweetDeck, and a big fan of the official iOS Twitter apps (which were also upgraded); I made a bee-line to the Mac App Store to get my hands on the new version.

After living with it for almost a week, my verdict is in:  the "1.0" version number is truly deserved, and man, I can't wait for a 1.1 release to fix the many problems.

Before I list my grievances, I should make one disclaimer: I'm the "must read every tweet" type of Twitter user.

I know, I know...Twitter timelines are intended to be treated as a stream (or fire hose, depending on how many people you follow); rather than as an queue like you treat email inbox.  General consensus seems to be that you're supposed to skim through recent tweets, accepting the fact that there are going to be things that you missed while you weren't actively looking at your Twitter client of choice.  Indeed, twitter.com only recently replaced the "Older tweets" link at the bottom of the page with an infinite scroll approach, where older tweets are automatically loaded as your scrolling nears the end of the page.

FWIW, I do treat my Facebook news feed as a stream where I don't feel that I need to read every status update.  Perhaps that's because my Facebook "friends" are actually more accurately described as acquaintances (extended family, people I went to school with, work colleagues etc.); where as the (relatively few) people I follow on Twitter are writing about things I'm genuinely interested in.  It's probably no surprise that I care less that "Simon" (who I probably last spoke to at school 20 years ago) "...is recovering from a big night"; and more that @joshuatopolsky from @verge has a new post on some new gadget.

So with that said, here's what's wrong with the new TweetDeck for Mac:

No longer tracks which Tweets you've read

In the old AIR version, a small unread indicator dot would show on the left edge of any unread tweets:

Returning to the app, you could quickly see where you were last up to, and continue reading.  Additionally, you could read tweets out of order if that's your thing. Now, for obsessive-compulsive types like me, you need to remember which Tweet you last read, which is a pain.

Historically, Twitter has been reluctant to build a native mechanism into the service itself for tracking read/unread status (presumably because in the "stream" vs "queue" debate described above, they're clearly on the "stream" side).  Logically I can see why they wouldn't want to do this too:  relationships in Twitter are currently user-to-user (user A follows user B); where as tracking read status would require a user-to-tweet relationship (user A has/hasn't read tweet 1234).  The storage requirements alone for tracking unread status for every user/tweet would be astronomical.  Think of celebs that have a million followers:  every single tweet they post would cause a million read/unread status records being created.

But why, for the love of god, couldn't this state be managed locally in the TweetDeck app? (ie. a local database that tracks, for every tweet cached locally in the application, which have or haven't been read?).  Hell, if that's too much to ask (which it's not...it was in the old version after all);  even just a marker at the position of the last tweet that I read would be be better than nothing (with a "scroll to last read" option).

Other apps such as Twitterific for iOS uses the 3rd-party Tweet Marker service to track (and sync between clients) unread status, so arguably TweetDeck could also hook into something like that (although given Twitter's propensity for buying out competing apps lately, it seems that they're looking to bring the entire Twitter experience in-house).

No unread counts in the dock icon (or in the app itself)

This is obvious, given the issue above.  But I'm listing it here because it was already an issue in the old AIR version, even when it did track read/unread status.

So, if they were to ever bring back the unread markers to the application, it would also be nice to be able to glance at the dock icon and see if there's a reason to bring the app to the foreground or not.

Again, if a count is too much to ask (which it's not); even a simple red dot indicating that there's at least one unread tweet would do.

Also, being able to mark a tweet as read directly from the popup Growl notification would be a useful feature, rather than having to switch to the app to indicate that you've read it.

No more Foursquare feeds

One of the compelling features of the old TweetDeck was that it wasn't just Twitter.  From a single hub, you could monitor all (or most) of your social feeds, including Facebook, Foursquare, Buzz etc.

Now that Buzz is no more, I can understand why it's no longer an option (and given that Google+ is probably considered a significant alternative to Twitter, I can understand why an app developed by Twitter wouldn't include it).   The fact that they didn't also remove Facebook is odd (welcome, but odd).   Does Twitter not consider Facebook a competitor, or is Facebook so entrenched in people's lives that the thought of removing support for it in TweetDeck was simply a bridge too far?

What is even more curious is the removal of Foursquare feeds.  Twitter doesn't have a "check-in" service to match Foursquare (unlike Facebook & Google+).   Yes, some clients do allow you to geotag your tweets with your current location; but that's not the same thing as Foursquare, Gowalla et al.   So why remove the ability to include a Foursquare feed in TweetDeck then?

Column widths

This is not strictly an issue with the new version of TweetDeck, as it was the same in the old AIR version too.

On a 1920x1200 resolution display, you can add five columns per page.  Adding a sixth column goes to the next page.  Yet, to the right of the 5th column on each page is a black area approximately the width of about half a column.

Is it just me, or is this blatant waste of screen real estate inexcusable.  Which designer chose these column width defaults, and did they even look at the app @ 1920x1200?

Ideally, columns should be resizable by the user; so that I can choose how much or how little information to cram into a single screen.

But if resizable columns is too much to ask (which..you guessed it: is not), then couldn't they simply optimize the columns to better fit?  Either make them all slightly smaller and get the sixth column onto the same page;  or slightly bigger and take up all of the available space.   Having an inch and a half of useless black space on the right side doesn't make any sense at all.

Timeline gaps when resuming from sleep

I have a 17" MacBook Pro at work.  It's just that bit too big to shove into my bag and carry to and from work each day, so at the end of the day I close the lid and pop it into my desk drawer.  On returning the next morning, thanks to the Mac's excellent battery and instant-resume; I flip open the lid and start working immediately.

In TweetDeck, I have the "Twitter Streaming" option enabled, which means that new tweets come in almost instantly rather than on some other polling schedule.

I don't know if it's because of this feature (certainly in the old AIR version, I had Twitter Streaming enabled, and it was never a problem there);  but when I return in the morning, the last tweet in my timeline is whatever was there the night before; and the next tweets that arrive are as of right now.  So there's a whole chunk of time missing.

I can right-click (Control-click) anywhere in the timeline and choose "Reload", which does fill in the gap with the missing tweets.  But I shouldn't have to do this.   It also doesn't help that the "Reload" option is only found in the popup context menu;  there's no global "Refresh" button anywhere else in the app (again, unlike the old AIR version).

Links open in embedded browser after resume from sleep

Another problem that occurs after resuming from sleep is that links in tweets no longer open in my default browser (Chrome).

Instead, they open in a "browser" window that seems to be part of TweetDeck.

What's worse, there are no navigation controls in that window to return to the main TweetDeck UI after reading the page.  At first, I thought it must have been a popup window, so I closed it. But that just closed the (only) open TweetDeck window, and clicking the dock icon simply returned me to the webpage window that I was previously viewing.

Finally, I figured out that the right-click (Control-click) context menu has a back option.

Really?  Would it kill you to have a "Back" or "Return to timeline" link/button somewhere less obscure?

But what really concerns me is why this 'embedded' browser window is only used after resuming from sleep.  Indeed, if I quit TweetDeck and then relaunch it; links in tweets once again start opening in Chrome.

Picture viewer

Speaking of links in posts, pictures posted to Facebook status updates appear as thumbnails in the feed.  In the old AIR version, clicking these pictures opened a larger version in a popup viewer (complete with the ability to cycle through other pictures from the same FB photo album using the arrow keys).

Now, clicking those thumbnails does absolutely nothing.  Thanks for that.

Also, links to picture services such as twitpic, yfrog, instagram etc. would previously open in a similar viewer.  Now, they open in your browser.

That said, there is a way to view these images within TweetDeck by clicking on the tweet itself (not the link).  This expands the selected tweet to show additional info (eg. if it's part of a larger conversation thread, it shows other replies);  and includes a preview of the aforementioned image.

Arguably you could say that this is more consistent (ie. hyperlinks now always open in your browser); but I actually prefer the old way.   I always want to read linked articles in my browser; but for pictures, I don't need to see it my browser....just show me the pic.

To achieve this now, I need to consciously think:  if it's a picture link, click the tweet; if it's something to read, click the link.

That's just my personal preference though.

Wrap up

Verifying that your PC backups can actually be restored is just as important as having created the backups in the first place.  Any I.T. pro will tell you this.

Since building a Windows Home Server (v1) a couple of years ago, it has been reliably backing up all my home PCs every night, and thankfully I haven't (yet) experienced a catastrophic failure of a client machine that has required a full restore.

This also means, I'm ashamed to admit, that I've never actually tried out the restore process for WHS.  I guess I just assumed that for a product as good as WHS is (and it really is), that the restore process would be as quick & painless as setting up the backups.

This weekend, the PC that my two young kids use to play games etc. (all the stuff that I forbid them from installing on my well-maintained machine) ran out of disk space.  Unsurprisingly, the measly 10Gb Seagate drive (salvaged from an original Xbox...don't ask) reached it's limit.

Like most tech geeks, I've got more old hard drives laying around than I care to count; so finding a replacement drive bigger than 10Gb was an easy task.  This would also be the perfect opportunity to try out the WHS restore process.

To begin, I ran one last manual backup to the WHS to ensure any last minute changes were included.  I then swapped the old drive for the new one, modified the BIOS to boot from CD, and popped the WHS "Home Computer Restore CD" (RTM version) into the drive.   Time elapsed: about 15 minutes.

Here's where I hit my first snag.  Now, in my defence, I was probably a bit over-tired and booting from CD is not something I do on a daily basis, but when the message "Boot from CD/DVD:" appeared on the screen, I mentally confused it with a similar message we all know and love: "Press any key to boot from CD/DVD".  After jabbing the "y" key, I wondered why my keystroke hadn't registered and all I got was an "NTLDR is missing" message.  (In hindsight, I now recall that the message "Boot from CD/DVD:" is shortly followed by the "Press any key..." once it detects the presence of bootable disk in the drive.  Which begs the question: why the colon at the end of "Boot from CD/DVD:", suggesting that my input is required?).

My first though was that my USB keyboard wasn't working (which was odd given that I had enabled USB Keyboard support in the BIOS, not to mention the fact that the DEL key had previously got me into the BIOS, indicating that the keyboard was working).  So I spent half an hour hunting around for a PS2 keyboard, only to find that it too didn't recognise my "y" keystroke.  So clearly it wasn't a keyboard issue then.

I then decided to try a different disk to see if it was a problem with the restore CD itself.  I popped in the WHS server disk and, lo and behold, I got the "Press any key to boot from CD/DVD" message that I was previously missing  (at which point the penny dropped, I called myself a few choice names, and regretted wasting nearly an hour on a keyboard issue that wasn't there).

The WHS server CD continued to boot successfully, so I knew at that point the problem must be the Restore CD.  I downloaded the latest WHS Restore CD image from Microsoft, and burnt a fresh copy to replace the RTM CD.  Unfortunately, that too wasn't recognised as a bootable disk, and I was again staring at "NTLDR is missing".

On a hunch, I plugged in an external USB CD drive and tried booting from it rather than the internal IDE drive in the PC.  To my surprise, the restore CD was able to boot from the external drive. Weird.

Anyway, with all that now behind me, I waited for the Restore CD to load its files into memory.  The restore process detected my ethernet connection and storage devices, and then tried to locate my WHS, which it couldn't.  I tried to manually locate it (by replacing "SERVER" with the name of my WHS box), but still no luck.  A network problem, pehaps?

After confirming that the network cable was correctly patched, and on the same subnet as the server; I figured it could be a driver issue (even though it had recognised my NIC).   So, I plugged a USB thumb drive into another client PC, opened the most recent WHS backup of the machine to be restored, and copied the contents of the "Windows Home Server Drivers For Restore" folder.  I then repeated the restore process again, this time allowing it to scan the USB key for drivers (which it did, and found a bunch of them).

Unfortunately still no luck finding my WHS.   In the end, I discovered that rather than trying to do the restore on the actual the client PC,  I could simply plug the new hard disk into another working client PC (either via eSATA, USB, or even an old school IDE removable caddy) and run C:\Program Files\Windows Home Server\ClientRestoreWizard.exe.  Using this method, I successfully restored the image in about 7 mins.

What I don't understand is why MS don't advertise this as a preferred, or at least equal alternative method for restoring a client PC?  I mean, why force users to go through the hassle of a) making sure they have an up to date restore CD (and if not, downloading & burning one), b) reconfiguring their BIOS to boot from CD (assuming their PC is able to boot from CD) and c) manually copying required drivers to a USB device?

By definition, WHS owners already have a home network, so in most cases there should be another working WHS client machine on the network. In addition, one could argue that temporarily connecting a hard drive to another PC (with hot-swappable eSATA and/or USB capability now a standard offering) is no more difficult than having users go into their BIOS to boot from CD, deal with drivers on a USB stick, or burn an ISO image.

Don't get me wrong: I can see there's a need to be able to do a restore directly on the target machine from CD, so I'm not suggesting that they do away with this.   What I'm suggesting is that the alternative method be officially supported (if it's still not) and given more prominence.

If you Google "whs restore process", the first page of results is all about how to use the WHS Restore CD, how to create a WHS restore CD etc.    No where in the top results does it mention anything about using the ClientRestoreWizard.exe on another PC.

When I first heard about GTD, the thing that really resonated with me was the quote:

We exert more mental energy trying to remember things to be done, than actually doing them  

"YES!", I thought.

For as long as I can remember, I've lived my life with the help of a set of "lists".  If it's not written on a list somewhere, the chances of it getting done is next to none; and I won't be able to focus properly on any other tasks while it's still rattling around in my head.

20 years ago, it was a spiral shorthand notebook & pen that I carried around with me; containing my lists of things to do, places to go, stuff to buy etc.

Throughout the day, I would strike items off the list as they were completed; and every couple of days (depending on how messy things were getting) I would transcribe any outstanding items to a fresh new page.  Clearly, such a manual process could only last for so long.

The notepad & pen was soon replaced with a fancy new Casio digital organiser.  Bear in mind, we're still in the very early 90s at this point, before devices like the Apple Newton were readily available (or affordable), many years before PDA's like the Palm Pilot, HP Journada, etc. were first seen, and over a decade before smart phones such as the Palm Treo, and later iPhone.

So a Casio digital organiser was about as high-tech as it got back then (and mine, having an expansion slot that could accept add-on modules such as dictionaries, financial calculators etc.  was the highest-tech of them all.  Many an hour was wasted playing 'Organizer Golf'.....).

As time rolled on, the Casio was replaced with a Palm V,  then a Treo 650, iPhone 3GS, iPhone 4 etc.  To this day, my life seems only held together by the notes, todo's, calendar entries and checklists stored on whatever device is in my pocket.  Which is why GTD makes a lot of sense to me: empty your brain of thoughts, organise those thoughts into actionable & non-actionable items, assign each of them a context in which they can be accomplished (@home, @work, @computer etc.), and review every day.

What's strange then, is why it has taken me so long to formalise my notes, todo's calendar entries & checklists into a proper GTD system.  Part of the problem has been identifying a suitable application that works for me.  There's no shortage of GTD-capable apps out there, for pretty much any and every device or platform; but every one of them has an Achilles heel (at least for me).

Maybe its because of my requirements.  At home I have a Windows XP machine as my main 'personal' PC (it's very old and soon to be replaced by a shiny new iMac, but until then a Windows client is still a mandatory requirement).  At work I have a MacBook Pro and a Ubuntu workstation; and on the go I have an iPhone and iPad.  So I was looking for a application that works on all these platforms and can sync effortless between them.

On MacOS/IOS, CulturedCode's Things is certainly beautiful to look at; but with no Windows/Linux option, not to mention the price (AU$70 for OSX + AU$13 for IOS), it was out.

(I don't know why I baulk at paying $83 though....my time is worth way more than $83, so it should be 'money is no object' for anything that improves my efficiency like a good GTD app.  Maybe I'm just so used to free open source software these days that any price is a sticker shock?)

Anyway, long story short, already being an Evernote user on the iPhone/iPad/Mac (albeit only to for trivial notes); I decided to look at using it for my GTD workflow.  There's plenty of blog posts of other people's GTD systems in Evernote, so it's not hard to find an example from someone who closely matches your preferred way of working.

Why Evernote?

One real benefit of Evernote is it's platform support & syncing.  Every platform mentioned above that I use is covered, with the exception of Linux (but for that, there's a free open source implementation: Nevernote).  Syncing between them is effortless, and I particularly like the feature on mobile platforms to only sync when in Wi-fi range (if you're concerned about 3G data charges).

For my GTD system, I use a single notebook.  There are other systems that use multiple notebooks (eg. one per project, one for Someday/Maybe items etc.);  but I find that I can do everything I need with a single notebook.

The crux of using Evernote for GTD is a combination of tags and saved searches.

Tags

You'll need tags for each of your projects, contexts, and priorities (eg. Today, Next Action, Waiting, Someday etc.).  I like to use a naming convention that helps distinguish between these different types of tags, by using a special first character:

Context tags always start with @ (eg. @Home, @Work, @Computer)

Priority tags always start with ! (eg. !Today, !Next Action, !Someday)

Project tags always start with # (eg. #Gardening, #Car, #New Website)

Misc. list tags always start with ^ (eg. ^Reference, ^Kitchen Ideas, ^Gift ideas)

Let's go through each of these....

Contexts (@)

In GTD, a context is used to indicate the resources required to complete the task.  Often this is a location (eg. 'mow the lawn' obviously can only be done when you're @Home); but context can also reflect a more general resource (eg. all your phone call tasks might be assigned a context of @Phone, meaning they can be actioned anywhere that you have access to a telephone).

Only you will know what are suitable contexts for you.  For me, I generally only need three: @Home, @Work, @Computer

You can assign more than one context to a task, but with my limited set of contexts, I find that if a task has more than one context it's often better to split it up into separate tasks.  So for me, most tasks have a single context.

Priorities (!)

I only use two different priorities: !Next Action and !Someday.  Some people also like to have a !Today priority, for things that you want to focus on today, but aren't necessarily the next action for a project.  I tend to make anything that I want to get done today a Next Action, so I don't need an explicit !Today tag.

As part of the organising/processing/reviewing stages in GTD, you need to determine which tasks are actionable, and what is the next action for any given project to progress.  For these items, tag them with !Next Action.

Anything that you don't expect to actively action in the near future, tag as !Someday.

What you should be left with are any tasks are still currently relevant (ie. are not a !Someday task), but aren't immediately actionable because they're waiting on something else to happen first (eg. another !Next Action task, or some other dependency)

Projects (#)

Projects are a way of grouping related tasks together.  Naturally, my projects will be different to your projects;  but they all start with a #.  In pure GTD terms, anything that requires more than one action should be a project and not a standalone task.

In Evernote, you can have nested tags, so often I will use those to sub-divide a project.  For example, you might have a #Home Maintenance project, and within that you might have #Gardening, #Repairs, #Pool etc. sub-projects.

One thing to note, though, is that tagging a note with one of these sub-project tags (eg. #Pool) doesn't automatically tag it with the parent tag (#Home Maintenance).  So, if you want to view all of your #Home Maintenance tasks, you either need to have previously tagged them all as #Home Maintenance yourself; or you'd need to craft a search for anything having any of the sub-tags.  It would be nice if Evernote had an option to automatically add any parent tags when you use one of the nested ones.

Misc. lists (^)

Sometimes you'll have items that aren't actionable at all (not even Someday); but you want to keep for reference purposes.  For me, these are things like gift ideas for my wife, or some random ideas I have for when we eventually renovate our kitchen.

Basically, anything that's not part of a project, is not actionable, and does not have a specific context, can go in as a miscellaneous item.

Saved Searches

Any GTD workflow starts by getting ideas & thoughts out of your head and into your GTD system (the "Collection" step).

Evernote excels at this part, because of the multitude of ways you can get stuff into it; such as emailing things to your Evernote email address (tip: setup a easy to remember address, such as <you>[email protected] or <you>[email protected], that forwards to the email address Evernote assigned to your account), clipping web pages and or URLs, quick entry from you phone, photos & voice memos on your phone etc.

If you have more than one notebook in Evernote, you should configure the one you use for GTD as the default, so that anything new automatically goes there.

Now, regardless of how items get into Evernote, you need a way of identifying those new items, so that they can be processed, organised & reviewed in later steps.

My preference is to create a Saved Search that will show anything that is untagged.  To do this, in the search box enter "-tag:*" (without the quotes) and save it as "Inbox" (note the "-" which negates the search...ie. "tag:*" means anything that has any tag, and "-tag:*" means anything without any tags).

Some people prefer to have an explicit "Inbox" tag that they tag new items with.  I don't, because that's one extra step that I need to remember to do when dumping thoughts in, and if I forget to tag something with Inbox it can get lost in the system.  Also, when processing items the Inbox tag needs to be manually removed, which again is an extra step.

With the Saved Search, anything untagged is automatically in your Inbox;  and tagging an item with either a context, priority or project moves it out of your inbox automatically.  The only downsides are that Saved Searches don't show a note count like tags do (that would be awesome, Evernote....hint, hint);  and it's harder to add multiple tags to an item from your inbox (because it drops off your list as soon as the first tag is added, so you have to then go and find it elsewhere to add the other tags).

Next, you want to create a few more Saved Searches that make it easy to quickly see what tasks you can immediately work on, depending on your context.

For me (because I have only 3 contexts),  this means:

Next Action @ Computer (notebook:GTD tag:"!Next Action" tag:@Computer)

Next Action @ Home (notebook:GTD tag:"!Next Action" tag:@Home)

Next Action @ Work (notebook:GTD tag:"!Next Action" tag:@Work)

This way, if I'm at a computer I can click on "Next Action @ Computer" to quickly see any tasks that I can start right now.

Additionally, I have another 3 saved searches:

Waiting @ Computer (notebook:GTD -tag:"!Next Action" -tag:!Someday tag:@Computer)

Waiting @ Home (notebook:GTD -tag:"!Next Action" -tag:!Someday tag:@Home)

Waiting @ Work (notebook:GTD -tag:"!Next Action" -tag:!Someday tag:@Work)

These searches enable me to quickly see any tasks that are neither actionable (-tag:"!Next Action") nor tagged as Someday (-tag:!Someday); in other words: any tasks that are waiting for something else before they can be done.

In the daily organise/review, tagging an item as !Next Action will add that item to the appropriate "Next Action @ ...." list AND remove the item from the corresponding "Waiting @ ...." list.

Finally, create a "No Context" Saved Search, which includes any items that haven't been tagged with a context, and aren't misc. lists (eg. notebook:GTD -tag:@* -tag:^*).  In my GTD system, a rule that I impose is that everything that isn't a misc. list must have a context.  The reason for this is that it simplifies my organise/review steps each day, as I only need to review the items for each of my three contexts (rather than having to review every project individually).

The GTD workflow

So lets walk though the steps....

Each day, set aside a specific time to go through your Inbox.  Your goal is to completely empty your Inbox of all items, either by doing the item right there and then (if it will take less than 2 minutes), or by tagging it with a context (and optionally, a priority and/or project that it relates to), which removes it from your Inbox search.

When your Inbox is completely empty, check your "No Context" saved search for any items that may have slipped through without a context.

Next, review your 3 x "Next Action @ ..." lists.  Delete any items that have either been completed, or are no longer required.  For any items are are no longer a Next Action, remove the !Next Action tag.

Finally, review your 3 x "Waiting @ ..." lists.  Delete any items that are no longer required.  Any items that are no longer waiting and can now be promoted to a Next Action should be tagged as such. Anything that has lost it's urgency or immediacy can be tagged with !Someday to get it out of focus.

Tickler file

Strict GTD-ers may be wondering about the lack of a tickler file.  David Allen's concept of tickler file is for items that are to be put off to some future date, and consists of 31 day folders and 12 month folders.

Some Evernote GTD-ers have setup a tickler file using tags, which seems like a good solution.

For me, I'm not quite ready to give up my Calendar just yet; and Evernote doesn't (yet?) have any way of integrating with popular calendars (Google, iCal, Outlook).  It would be great if you could enter a date into an Evernote item and have it automatically create an entry in your preferred calendar.