Saturday, October 16, 2010

Displaying screenshots on acceptance test failures

When running GUI tests it can be difficult to determine what was being shown on the GUI at the point of failure, especially if the tests are running minimised or on a CI server.

We developed a screenshot extension for Concordion to include screenshots in the output whenever failures occur. We've been finding this extension incredibly useful for diagnosing intermittent failures in our automated acceptance tests.

This requires the new Concordion 1.4.1, which introduces support for extensions. Amongst other things, extensions can add new listeners to commands and make changes to the Concordion output.

The following example is running an end-to-end test using WebDriver (Selenium 2) to test a Google search through the Firefox browser.

While the results show that the test is failing since Netherlands is displayed in the Google results, it would be helpful to see the actual results page. The screenshot extension allows us to hover over the failure to see an image of the browser page at the time the failure occurred.



Clicking on the image opens it for further inspection:

In this example, we've configured the extension to use WebDriver's TakesScreenshot interface, so we see an image of just the browser page, irrespective of whether it is currently visible on the screen.

The extension can also be used to take screenshots on success, and includes a command to explicitly take screenshots.

The screenshot extension project is on Github. You'll need to set the concordion.extensions system property to use it - see the README for details.

The source code for this example is in the demo project.

Acknowledgements:
This extension was partly inspired by Mark Derricutt's ScreenshotCommand, and by Adam Setch's post to the Concordion list.

UPDATESOct 25 2010. Added acknowledgements
Jan 03 2011. Project moved to Google Code and docs to Concordion site.
Oct 26 2014. Updated project and demo location to Github projects