Matthew O'Riordan

This is where I record my rants, comment, quotes and thoughts on things. I welcome your input, please fire away.

Find out more about me at: http://mattheworiordan.com

April 17, 2011 at 11:48pm

Home

Full stack integration testing with Rails 3, Cucumber, RSpec, QUnit and Capybara

I find it enlightening to be part of such an active Ruby and RoR community where Gems, code samples and responsive community support is available, however I also find it hard sometimes for that very same reason to figure out what stack to use for my new projects because there are just too many options.

One of the problems I’ve had for a while now is figuring out what stack to use for my testing, both for my back end code (unit & integration) as well as my front end code (unit & integration).  So this blog post is about my journey in April 2011 to find the best full stack integration components for a Rails project and document it for others to use.  I hope others find this useful.

Background

I use RSpec and Cucumber for all my testing, but my application’s front end is becoming increasingly complex and needs testing as well.  I would ideally like a combination of both unit testing and integration testing on the Javascript and front end elements.  And I want my tests to be as fast as possible because the longer it takes the less likely it is I’ll run them.

The objective of this exercise

  • Document the entire process collecting all useful resources together in one place so others can learn from this
  • Set up a basic Rails 3 project and push it to Github which has enough back end & front end code to write some meaningful tests.  This must include some asynchronous client-server requests.  And a Javascript framework such as jQuery should be used to make the stack and environment realistic.
  • Write RSpec & Cucumber tests for traditional unit testing and integration testing.
  • Explore options to unit test and integration test the front end code, documenting the pros & cons for each
  • Find the best stack for me and put that up on Github for others to use

My journey to find the ideal full stack integration suite for Rails 3 (as of now)

I installed a clean version of Rails 3.0.3 on Ruby 1.9.2.  See commit.

I like to use RSpec for all unit tests, so I installed RSpec and removed all the old Test::Unit files in /test/.  See commit.  After you add the gems to your Gemfile, remember run `bundle` and also to run the RSpec generator `script/rails generate rspec:install`.  For those of you that use scaffolding (I avoid it), I also modified config/application.rb to instruct Rails 3’s generators to always use RSpec rather than Test::Unit. 

My favourite tool for integration testing is Cucumber.  RSpec can be used for integration testing too, and I know there is a current view from DHH himself that both RSpec and Cucumber are unnecessary, however I like Cucumber and the readable user stories are great.  So I went ahead and installed Cucumber that by default now uses Capybara (it used to default to Webrat).  I followed the Cucumber Rails installation instructions, however to ensure Capybara is used for tests as opposed to Webrat I ran the following generator command `rails generate cucumber:install —capybara`.  See commit.

In order to get the project ready for front end development, I cleared out the index.html file, installed a copy of jQuery and RoR JQuery UJS, removed all javascript files and added the UJS Javascript file), and added in a stripped down jQuery UI code base to allow for some more complex user interactions.  See commit.

I then went ahead and installed SASS because writing traditional CSS just sucks.  I normally use HAML as well instead of ERB, but for the purposes of this example I thought it best to keep things as understandable as possible and it’s quite possible HAML is less understood (and once again, DHH does not like it). See commit.

Building the Calculator app and tests

Following good TDD, I then went ahead and wrote the tests for a new Calculator model class that I am about to build.  The calculator is intentionally very simple and only supports addition, subtraction, multiplication and division.  I kept the tests simple and adequate for the purposes of this project.  I then ran RSpec `rspec spec/models/*.rb` and got a load of errors as the Calculator class does not yet exist.  

In order to satisfy the tests, I built a simple Calculator class that lives in the /app/models/ directory.  I ran RSpec until everything was green, and voila, we now have a test suite for the model which passes.  See commit.

I then built a very simple HTML based calculator and a controller called Calculator to manage all calculator operations.  See commit.  I pushed it up online using Heroku and posted it at http://full-stack-testing.heroku.com/calculator

Once finished, I added some Cucumber integration tests which will run using Capybara’s default driver rack_test as they do not need Javascript support.  Ideally I should have written the features first, but for the purposes of this example I was very much discovering as I built, and had no spec to work from.  Fortunately in most traditional situations I have a spec I can work from and can therefore follow TDD and build out the Cucumber features first.

The first test was extremely simple as follows:

I ran `rake cucumber` and ran into the following issues:

Firstly I received the error:

schema.rb doesn’t exist yet. Run “rake db:migrate” to create it then try again. If you do not intend to use a database, you should instead alter /Users/matthew/Projects/full_stack_integration_testing/config/application.rb to limit the frameworks that will be loaded

To fix that I simply ran rake db:migrate which created an empty schema.rb file.

I then re-ran `rake db:migrate` and ran `rake cucumber` again.  Unfortunately this time I received a rather less intuitive error:

undefined local variable or method `node’ for #<Capybara::Driver::Node tag=”a” path=”/html/body/a”> (NameError)

After a little searching it seems this is a known error.  I found this strange considering I thought I was on the latest stable versions of all gems, and my feature is extremely simple.  However, after looking at a related closed issue in Capybara I see that this issue was corrected, so I clearly don’t have the latest gems.  I had to modify my Gemfile to explicitly request cucumber-rails with a version greater than or equal to 0.4.0 to replace the 0.3.2 version that we was used by default.

I then re-ran `rake cucumber` and unfortunately hit another error as described at http://blog.firsthand.ca/2011/02/cucumber-rails-undefined-local-variable.html

I had to re-run `rails g cucumber:install` to regenerate the cucumber files, and then I was ready to run rake cucumber again.  See commit. 

So I finally ran `rake cucumber` and fortunately my very simple test passed as follows:

So I went on to writing some more expansive Cucumber scenarios to provide better integration testing of the calculator.  And after building only one additional step, I managed to create a Cucumber feature which covered the bulk of the functionality offered including exceptions for the Calculator.  See commit.

Test performance and Spork

So a recurring problem I have is that whilst Cucumber tests report completing quite quickly (in around 2.5s in this case), it took considerably longer than that in reality as the rails environment start up time is very slow.  Whilst I appreciate testing is an absolute necessity, if it’s going to be painfully then that’s going inevitably deter me from testing.  So I have heard about spork on a few of the podcasts I follow, and understand that spork is effectively a library which runs your test suite by forking a new copy of the Rails environment each time you run your tests.  So I went ahead and followed the Spork installation instructions.

Specifically I added the following lines to my Gemfile

I then ran `spork —bootstrap` which modified the file spec/spec_helper.rb and added further installation instructions into that file.  I looked through the instructions and read that out of the box spork works well with rails.  So I left all settings as they were.

I then started up the spork server by simply running `spork` and got the following:

I then ran my first set of RSpec tests with `rspec —drb spec/**/*` and it was damn quick.  Everything was completed in under a second, amazing.

So I then read that to run the cucumber tests using spork all I needed was to run `script/cucumber —drb` so I went ahead and did that but unfortunately was not quite so lucky.  I got the response:

  WARNING: No DRb server is running. Running features locally:

So I did a bit of searching and found an article on using spork with DRB on the cucumber Wiki.  It turns out I need to run the generator again and include spork support.  As I need capybara support too (see above), I ran the following:

rails generate cucumber:install —capybara —spork

It asked to overwrite my paths file, so I let it do that and just reinserted the custom path I had added earlier.

I then ran `script/cucumber —drb’ and was told again:

WARNING: No DRb server is running. Running features locally:

I did a bit of searching and discovered that this was because the spork DRB server was actually running for use with RSpec and it needs to be run for use with Cucumber.  So I stopped the spork server and executed the spork server for cucumber as follows `spork cuc`

I then run `script/cucumber —drb’ and as expected the test ran immediately, considerably quicker than before.

As I want to run both Cucumber tests and RSpec tests as fast as possible, I wanted to find out how I can run spork for both cucumber an RSpec simultaneously.  It turns out this is quite simple, all I need to do is run the following command:
`bundle exec spork cucumber & bundle exec spork rspec &`
which gets spork for cucumber and rspec running as background tasks, and on different ports namely 8989 for RSpec and 8990 for Cucumber.  Note you can access these tasks using `jobs`, or `fg [job number]`.  You can also kill these tasks using `kill %[job number]`

Spork for both environments is something I’m going to use often, so I set up a simple script script/spork which runs spork in the background for both RSpec and Cucumber.  To make this an executable I ran the command chmod 777 script/spork in the root.  I ran `script/spork` and both servers were started in the background.

In order to measure performance improvement of the tests using spork, I set up a basic benchmarking script residing in scripts.  The code is extremely simple and runs the rspec tests and cucumber tests without DRB (that’s what spork uses), and then with DRB to see the difference.  I noticed however that cucumber.yml was configured to run —drb as an option automatically, so I removed this and added -c for colour output.  I then ran script/benchmark_tests and got the following results:

  • RSpec without spork took: 9.81s
  • RSpec with spork (DRB) took: 1.11s
    Spork saved 8.70s when running RSpec tests
  • Cucumber without spork took: 13.69s
  • Cucumber with spork (DRB) took: 4.31s
    Spork saved 9.38s when running Cucumber tests

So it looks like spork is doing a great job in saving on the start up time of the tests for both Cucumber and RSpec.  Admittedly as my tests increase in size this start up time saving will become less relevant, but any speed improvement is welcome especially when running RSpec unit tests which are typically quite fast.  See commit.

Automated tests and Watchr

And this is all great, but I don’t really like jumping into the console every time I need to run a test.  What I want is to run tests automatically whenever I change the relevant files in my project.  I’ve heard of two apps which achieve this, autotest and watchr.  Watchr seems to be gaining more traction of late, and my previous experience with autotest has not been brilliant and have always had problems getting it running reliably.  So without much further research, I decided to use Watchr quite simply because influential Rubyists have been talking about it recently so I trust their judgement.  So firstly I added watchr gem to my Gemfile and ran `bundle`

I then found a Gist which seems to roughly achieve what I want in that it supports DRB, Cucumber and RSpec.  I saved the Gist source code to script/watchr.rb and I tried running `watchr script/watchr.rb` but unfortunately was told that Growl was not installed.  I added growl to the Gemfile, ran `bundle` and then reran `watchr script/watchr.rb` and it started up as expected.  However, when changing an RSpec file in spec/models the tests did not run as expected, so I made a few changes to watchr.rb.  Please take a look at my Gist to see the list of changes.

So once I made my changes, everything seemed to be working except that the growl notifications did not have an image displaying, and the output from RSpec was not colour coded.  So firstly to address the Growl issue I downloaded some PNG status icons and saved them into /script/.watchr_images.  In order to resolve the ANSI colors being lost for RSpec issue, I tried a number of alternatives using all methods described in the article “5 ways to run commands from ruby“ but unfortunately none of those solutions worked.  So I posted a question on Stackoverflow  and someone helped me find the solution.  In order to have colour show up in RSpec you need to tell it you are outputting to tty so that colour is supported, so I modified the $spec_cmd in watchr.rb to the following:

Now any changes to features will trigger Cucumber, and any changes to models or unit tests will trigger RSpec and confine to the effected files where possible.  See commit.

Front end unit and integration testing

So onto the more difficult parts, front end unit and integration testing.  So I have gone ahead and added some trivial front end code to the calculator which presents a set of buttons for the calculator so users don’t have to use the text field when entering the number used in operations.  I wanted to ensure that my tests had some AJAX functionality too as this is generally needed in my web applications.  So in summary, the code I added provides us with the following to be tested:

  • Simple Javascript class methods that can be unit tested i.e. no DOM is required
  • DOM manipulation on load to add additional features to the calculator
  • AJAX communication with a server and dynamic updating of the DOM including handling of errors
  • Prompt dialog box which requires the user to interact with to continue when performing an operation on zero
  • Alert dialog box which prevents a user from dividing by zero
  • JQuery UI Dialog box which again needs user to interact with to continue
  • Drag & drop functionality (you can drag values from the calculator to a pretty pointless memory bank)
  • Use of persistent cookies to persist the values stored in the memory bank

I committed all those changes and you can see the calculator live at http://full-stack-testing.heroku.com/calculator.  Please note that the calculator is not particularly intuitive so you need to enter the value first and then press the operation you wish to use! This was done so that most operations can be performed using simple HTML forms.


My search for the best front end testing libraries

The first library I found was the Harmony gem.  Harmony uses Johnson to access the Mozilla Spidermonkey Javascript engine developed for Firefox.  Harmony then uses env.js to simulate a browser DOM.  Now if we’re just going to be doing unit tests at first, then arguably we don’t need a DOM, however considering most of the JavaScript relies on jQuery and in turn the DOM, I think it’s wise to ensure there is support for a DOM and thus we won’t need to mock or stub too much code to get the unit tests running.  I also came across Holygrail which is a Harmony plugin specifically for Ruby on Rails.  I added the following gems to my Gemfile:

I then run `bundle` but unfortunately I ran into issues installing Johnson.  Looking through the errors (and there were loads), they were all related to the C build of this so not something I wanted to try and dig into and solve.  I then took a look at the last commit dates for Johnson and it was worrying, March 2010, so over a year since anything was contributed.  I did the same for Harmony and the last update was July 2010, so again, closing in on a year since the last update.  I also found some posts indicating that Johnson does not support Ruby 1.9.2 which is the version of Ruby I am using.  Encouragingly env.js was updated days ago, so that project is still very active.  So I abandoned the Johnson / Harmony option and looked for a testing framework that has a more active community and was still being developed.

I found RubyRacer which is constantly being updated and arguably uses the best available Javascript engine as it uses Googles V8.  Unfortunately I could not find any examples where The Ruby Racer had been used for RSpec or Cucumber testing other than a few discussions on the matter, some experiments and some basic examples:

So after some further research, I found that RSpec itself supports Capybara as well as Webrat.  See the RSpec and Capybara Railscast and the following commit on RSpec Rails.  There are also some examples on the Capybara Github home page.  So I set out to get Capybara running with Rails Rspec to see if we could get it to run an RSpec test.  The problem is that we need to decide which driver to use.  The common options are:

  • Selenium (launches and automates browser, brilliant but slow)
  • Envjs (after trying to install this gem and capaybara-envjs this there are numerous problems getting this working with Ruby 1.9.2, probably because it relies on Johnson which is not Ruby 1.9.2 compatible, see http://groups.google.com/group/ruby-capybara/browse_thread/thread/1556c7a2b0fe0996)
  • Celerity (headless browser in Java, requires JRuby)
  • Culerity (headless browser that can be run from normal MRI ruby i.e. not JRuby, but does not appear to support Ruby 1.9.2 at present)
  • Akephalos which according to Thoughtbot back in Nov 2010 was the best option out there.
  • New (as of 26 Jul 11): Capybara Webkit is the new kid on the block, and supports a full high performance Webkit headless browser.  Read at the bottom of the post for an update on how to get this working after following all the steps in this post.

So I installed Capybara and tried to run an RSpec test placing it in the requests folder (which automatically invokes Capybara).  Unfortunately I wrote some tests which assume Javascript is running, but I could not get Capybara and Akephalos / Selenium to run with RSpec.  I am told that Capybara is being used, but it refuses to use a Javascript driver for some reason even with my RSpec tests metadata set as follows:

So eventually I decided to focus for now on Cucumber tests to ensure that Capybara is set up correctly to use Selenium and Akephalos first.  I wrote some basic tests with tags for @javascript and @selenium to test Javascript functionality.  In my cucumber env.rb file I added the following so that Akephalos is the default Javascript driver and Selenium can be used where necessary:

And I ensured that the following gems were added to the Gemfile and bundle was executed:

Cucumber ran as expected, so the following two simple example tests passed and Firefox was instantiated for the test tagged with @selenium:

So now that I knew that Capybara is working correctly, I wanted to get RSpec working with Capybara.  So instead of using RSpec metadata according to the instructions on the Capybara page under the section “Using Capybara with RSpec”, I manually set the driver in the RSpec tests as follows:

And this worked.  So this led me to believe that for some reason RSpec is simply ignoring the metadata.  I did a bit of searching and found a ticket about Driver swapping with RSpec 2 which pointed to a useful Gist.  So I used the code form the Gist as a base and added the following to my spec_helper.rb file:

With this configuration in place I was able to run the following two simple tests, the first using the default Javascript driver Akephalos, and the second using the driver Selenium as it is explicitly required.

However, when I ran the tests from under watchr using —drb Akephalos failed with an error messages:

Failure/Error: within(“#calculator”) do
  RangeError:
     0x0000008184a6a8 is recycled object

I found an open ticket created around a month ago describing this issue.  It seems that as both Cucumber and RSpec are failing Akephalos tests with spork, we’re going to unfortunately need to disable Spork for now.  Whilst Spork helps speed up the start up time for tests, I have read it actually marginally slows down the tests themselves, and based on how long my test are taking to run now once we start up a headless browser and an actual browser, I don’t think the start up time is that significant anymore.  So until the spork and Akephalos issue is resolved, (see http://spacevatican.org/2011/7/3/sporking-with-akephalos for a workaround) I’m going to opt for using watchr without spork.


Front end unit testing from Rails

Whilst for this calculator app there is not a specific Javascript unit testing requirement, for the purposes of this article I wanted to work how I will be unit testing my front end code in future, and what’s the best way for myself.  On one hand, I can simply insert Javascript code into my RSpec tests using code such as:

however that doesn’t feel quite right to me.  Firstly, if the Javascript test code got complex I’d have lots of Javascript code embedded in my Ruby code which is just messy and hard to debug.  Also, I feel like I’d like a solution where the tests could also be run in my browser so that whilst developing my front end code I could manually initiate tests without having to invoke RSpec each time.  I know there are lots of Javascript testing frameworks out there, but the one I hear of most often (and is used by the JQuery team) is QUnit.  I also came across this QUnit plugin for Rails, however it’s very out of date (last update Dec 2009) so I expect no longer relevant.  I then looked at JsTestDriver which allows you to have multiple browsers running as slaves and simultaneously run tests against all those browsers.  Whilst this sounds useful, I don’t want to overcomplicate what I’m trying to achieve at this stage, so I decided to stick with QUnit for this article.

So first thing I did was download and add qunit.js and qunit.css to my project.  I then wanted to keep my QUnit tests in the specs folder along with all my other tests files, but this was not possible out of the box as those files are not accessible from the browser.  So I set up a simple route and controller for my Javascript tests which serves up the Javascript tests using a special QUnit HTML template I set up.  The controller is simply:

Note that I store all my javascript test in /spec/javascript/ and suffix them all with _spec.js.erb.  And I have also forbidden the running of these tests unless in test or development environments.

I then added a route:

And added a simple HTML template javascript_test.js.erb into layouts for use with QUnit tests.

I then modified my calculator.js class to be a bit more testable for the purposes of Unit tests (i.e. removed dependency on the DOM for various methods including the AJAX functions), and set up my first set of tests in a new file I named calculator_spec.js.erb.  Here is an excerpt from the spec file which shows a normal Unit test as well as one which supports asynchronous AJAX requests:

So I now have QUnit running nicely in my browser when I visit http://localhost:3000/javascript_test/calculator on my dev machine.

However what I really want is for these QUnit tests to be run automatically as part of my RSpec tests.  My rationale for this is that all my unit tests are run from RSpec whilst all integration tests are run from Cucumber.  So I don’t want to now have to manually run QUnit tests as well, I want these tests to be run automatically whenever Watchr picks up changes or RSpec is invoked manually.

So I wrote a new RSpec test which simply iterates through the tests in the /spec/javascript folder, invokes the Akephalos headless browser, and renders the HTML and Javascript for the page.  It then sleeps until the QUnit tests have completed (with a 60 second limit per suite of tests), and once the tests have completed it checks that each QUnit test passed using Nokogiri which parses the HTML.  As QUnit generates an HTML based microformat it’s easy to iterate through each passed/failed test.  As you can see in my commit, test 5 “Wait up to 5 seconds but fail with timeout after 3.5 seconds” fails as expected, whereas all the other tests pass.  To ensure the Javascript tests run automatically, I modified watchr to include the Javascript tests too.  See corrected commit here.


More complex user interactions for the Cucumber integration tests

The last thing I want to do now is have some more complex integration test examples working so that we can simulate a more extensive set of user interactions.  Our calculator already supports some more complex interactions as follows:

  • Drag and drop - you can drag the calculator current value into the history area
  • Use of cookies - a user’s history of drag and drop should be retained across sessions
  • Handle a confirm dialog box which requires user input to continue - I have added a confirm when performing an operation on a zero value.
  • Handle an alert dialog box which should appear when dividing by zero.

Confirm dialog boxes are supported natively by Ruby Selenium Webdriver, however this method only works with Firefox and would also then fail for any Akephalos tests.  My feeling is that unless Capybara has a generic way of dealing with alert / confirm dialog boxes, then it would be better to stub out this functionality using Javascript.  I found a an article on this, “How to test a confirm dialog with Cucumber” which led me to the this Gist which I forked to support confirm & alert dialog tests with cucumber.  

So I got confirmation and alert dialog boxes working, along with drag and drop testing in this commit.  It was worth noting that Akephalos & HTMLUnit failed the drag and drop test in that no error was raised when calling drag_to, however I found the drag & drop event was never fired for some reason.  So in order to get all my features passing I have used Selenium for the drag & drop tests, and Akephalos for all other tests.

Finally, I realised that although my application has no database, most applications would so I needed to configure RSpec2 to not use transactional fixtures when running Javascript tests.  See commit.

Updates to this post as of 26 July 2011 - Capybara Webkit

Since I wrote this article, the great folk over at Thoughtbot have indeed come up with a solution called Capybara-Webkit that seems to be superior to both Akephalos and Selenium in many regards.  They have integrated QtWebKit, which appears to be a library to enable cross platform development using the popular WebKit which powers Chrome, Safari and most mobile browsers.  The beauty of the solution is that it seems to be really fast, it supports advanced features like drag & drop which were not possible with Akephalos, yet this can all be achieved without opening up a browser which was required previously through the use of Selenium.  I’ve updated the Git repository with the changes so you can see how little you need to do to get Capybara-Webkit to replace both web drivers.

That’s it

I’ve got a test suite I’m pretty happy with now that reliably tests both back end and front end code with both unit and integration tests.  It works well with the latest version of Ruby 1.9.2 and Rails 3.0.  Thanks certainly goes to Capybara, RSpec2, Cucumber, Selenium, Akephalos (and now Capybara-Webkit), Watchr and Spork.  The only thing I’m not entirely convinced with at the moment is the use of Watchr to automatically invoke Cucumber as the Cucumber tests can be particularly slow, but I’m sure each person will have their preference.

You can see the full source code at https://github.com/mattheworiordan/Full-stack-testing/

The live (and pretty pointless) demo project is up at http://full-stack-testing.heroku.com/

Useful links I came across whilst researching this article

What next?

I would like to get this all working with a continuous build server which not only runs the full suite of tests whenever commits are pushed, but also runs the front end tests across multiple browsers including multiple operating systems.  I think I’ll be using Selenium and possibly JsTestDriver with QUnitTestRunnerPlugin, but that’s for the next article.  I’ll be working on that as soon as I can, and I’ll post a new blog post when it’s ready.

Follow me on @mattheworiordan 

If you have any questions, suggestions, or corrections, please do get in touch Matthew O’Riordan


Notes

  1. unimatrixzxero reblogged this from mattheworiordan
  2. mattheworiordan posted this

← Next post
Previous post →
Home  Archive  Mobile  RSS    
Themed by langer, powered by Tumblr.