#test-unit

Thoughts on testing—part 1

Friday, September 17, 2010

It doesn’t take a genius to realise that testing is a very personal matter. From Test::Unit purists to RSpec evangelists, everyone has their own take on testing Ruby code. And whilst we believe that not only there are no silver bullets in most of the problems in software development but personal preferences make it unlikely that there will ever be any, we’d like to share our thoughts about testing in this series of posts. We hope it will be as enjoyable for you to read as it was for us to write.

Consider using Test::Unit before delving into third-party frameworks.

There’s nothing wrong with the plain ol’ Test::Unit. In fact, choosing it over fancier solutions such as RSpec, has many advantages.

Firstly, you’ll have hard time finding a Rails developer is unfamiliar with Test::Unit. Using it in your projects guarantees minimum hassle when swapping developers between projects.

Whilst RSpec-like syntax offers some sugar candy it can be quite misleading sometimes. Consider the following two examples:

class PostTest < ActiveSupport::TestCase test "adds 20 points to post author" do @post.save! assert 20, @post.author.points # should be assert_equal end should "add 20 points to post author" do @post.save! @post.author.points == 20 # no assertion, should be "@post.author.points.should == 20" end end

The first test will always pass, regardless of the value of @post.author.points since 20 evaluates to true. Apparently mistakingly writing assert instead of assert_equal is quite a common fallacy of Test::Unit, often cited by RSpec enthusiast. It is not uncommon however to encounter the second example in an RSpec or a shoulda + matchy test suite, where a similar mistake is made by forgetting to add .should before the comparison.

Nested contexts make writing tests easy, but reading them tough as hell.

It isn’t unusual to find between 10 and 15 contexts in a single test case. Each with its own setup, some nested three or four leves deep, yet sometimes containing only a single test. We found those test cases to be extremely unmaintainable, especially when you’re trying to figure out which setups apply to a particular test. Keep your tests as flat as possible (or don’t use contexts at all) and avoid deeply-nested setups at all cost. Don’t worry too much when you repeat some code in your tests — DRY is great for your application code, but tests need to be readable first and foremost.

Don’t get all touchy-feely with your database. Save records only when you

have to.

Avoid saving records in setups — saving your records in unnecessary in the majority of your test cases, especially when you’re dealing with validations or properties which can be tested without ensuring database persistence. Save your records only when you have do, and do so from within that particular test. Try not to add things to your setup which are only used in one or two tests — you’re better off having some repetition than initialising your data set unnecessarily way too many times.

To illustrate this further — the following should be considered a bad practice:

class PostTest < ActiveSupport::TestCase def setup @post = Post.make # "generic" post @old_post = Post.make(:created_at => 1.year.ago) @reported_post = Post.make.tap(&:report) end # ... end

And refactored as follows:

class PostTest < ActiveSupport::TestCase def setup @post = Post.make_unsaved end test "reporting posts" do @post.report! # saves and reports the post when needed assert @post.reported? end # ... end

Set up in startup.

Parts of your test case don’t need to be re-initialised for every single test case. You can avoid unnecessary database queries by instantiating them in self.startup instead. In fact, you can apply this principle to all data which is not the subject of the individual test case, e.g. when testing Posts, try to avoid creating new users for every single test scenario — do it once during startup.

class PostTest < ActiveSupport::TestCase def self.startup @@author = User.make end def setup @post = Post.make_unsaved(:author => @@author) end test "saves successfully" do assert @post.save end test "can be reported" do @post.save assert @post.report end # ... end

Sharing is caring — do it with modules.

Often times you’ll end up with common behaviours you’d want to test across multiple models. You can achieve modularity in no effort by using Ruby modules:

module RemovableTests def test_sets_removed_at_attribute @model.remove assert @model.removed_at end end class PostTest < ActiveSupport::TestCase include RemovableTests def setup @post = @model = Post.make_unsaved end # ... end

No magic, pure awesomeness. This simple trick will help you DRY-up your tests and share common test cases between different models.

Summing up — the good, the bad and the ugly.

Stay tuned for more thoughts on testing, but until then, here’s a short summary of what we believe is worth taking away from this article:

- Keep things simple — Test::Unit is good enough most of the time. Avoid choosing RSpec just because it’s in fashion at the moment.

- Keep things lightweight — only save records to the database if you absolutely have to. Doing it before every test case is bad for performance.

It only has an "assert_equal_set" at the moment, but I thought I better introduce SetAssertions. If nothing else, it shows you how to make a clean, conventional, Gem for custom assertions.

Usage

gem install set_assertions

or add it to your Gemfile

In your test_helper.rb or at the top of your test case file, just:

require 'set_assertions'

Then you can use the new assertions (well, one assertion, for now) in your tests.

Just a word on process.

I used Bundler to create the gem and publish it. It really is simple and elegant.

I also modified the Rakefile to support test-unit testing:

require "bundler/gem_tasks" require 'rake/testtask' Rake::TestTask.new do |t| t.libs << 'test' end desc "Run tests" task :default => :test