Improving a Concordion specification

Katrina Clokie will be running a workshop on three styles of automation at Let's Test. The 3 styles use different test runners and have different patterns of using WebDriver. With one of the styles using Concordion to drive the tests, I'd like to suggest a few areas for improvement with the Concordion specification while retaining the original feel. (Katrina has acknowledged that the Concordion suite is using a traditional approach).

Markdown

Concordion 2.0 introduced Markdown support, in addition to HTML and Excel. Markdown makes it easy to create rich documents using plain text and is easier to read and write than HTML. We can replace the HTML specification:



with:

When previewed, for example on Github, or in an IDE, the Concordion commands are shown as links with the command displayed when you hover over the link:

Use of assertTrue

I view the assertTrue command in Concordion as a "spec smell" ("a surface indication that usually corresponds to a deeper problem in the system" [1]).

At the very least, assertTrue should be used sparingly since it only displays true or false on failure, which necessitates additional debugging whenever an issue occurs.

Acceptance Criteria and Examples

As described in Seeing the wood and some trees, including both the acceptance criteria and some examples helps us discover gaps and gain a deeper understanding of the requirements.

The IRD PAYE specification above is documenting the acceptance criterion as an example which, in my view, is too high-level for an example. By omitting the concrete details of the example, we fail to ask the "what-if" questions which limits our discovery of additional interesting examples. 

Writing the specification as follows, pulls the test up to be an acceptance criteria and shows the concrete example. 

This also removes the assertTrue spec smell. Note that we still only show relevant details, eg. user type and tax year are not relevant so are handled in the fixture.

Seeing this example, I start questioning "what-if" about the comma in the input amount field? Do we need it? If used, should it be restricted to separate groups of three digits? Or might our Indian customers want to express in Crores and Lakhs, for example "1,00,00,000"? 

• Would "10,24.00" be a valid amount? (Yes currently)
• What about "1024.0,0"? (No)
• Or "1,2,,0,4.00"? (Yes currently)

These are all valid questions that result in an additional acceptance criteria about comma placement. We tend to only need to ask these questions once per system, and can then reuse any decision across the system. (This is also what tends to make those first few spec workshops / 3 amigo sessions / example mapping sessions slow. Once we have those points straightened out and part of our team's shared language, they don't need to be discussed in detail each time.)

If we were to automate these examples, we would focus the example to only contain the context for the behaviour we want to show:

Conclusion

The refactored specification is smaller, easier to read and write with fewer errors. Any failures of the test now clearly display the actual and expected values on failure. 

By cleaning up the specification, we have discovered other acceptance criteria that may be worth exploring, such as the placement of the comma in the input amount.

Other improvements could still be made to the specification, such as rewording the Given-When-Then to make it less clunky.

The updated code is available on Github.

[1] Wikipedia - Code smell

YOW! Retrospective

The YOW! 2011 conference in Melbourne was fantastic. My brain is totally buzzing from the great presentations, discussions and energy from the conference. Before the buzz settles, here's my personal retrospective.

YOW will be making the presentations and videos available online in due course.

1 - What I Learnt

Key themes

1. Continuous Delivery of Business Value
2. JS everywhere, REST, push technologies, HTML5
3. Having fun with technology

Continuous Delivery/Business Value sessions

Continuous Design - Mary Poppendieck

Design Thinking:

1. Assemble Diverse Team
2. Frame the problem
3. Ideation - prototype
4. Experimentation
5. Iterate back to 3, or Reformat the problem/pivot back to 2

Continuous Design puts coding into this loop

The Learning Cycle. Learn -> Model -> Build -> Measure -> Learn ...

The fastest learner wins.

A small, experienced, dedicated, self empowered team can introduce 4x more product - George Stalk, competing against time.

Need good people, whole team (marketing, design, dev, test, ops, support)

Measure product metrics not project metrics

Amazon.com, working backwards:

1. Write a press release
2. Write FAQs
3. Describe customer experience
4. (Write User Manual)

Innovation Accounting

• Start with hypothesis - a business growth model
• Establish a baseline - the Minimum Viable Product
• Target every initiative at a Growth Metric
• Split Test A/B
• Definition of Done includes feature value validated

Cohort Metrics

• Measure a cohort of new users every week
• Track behaviour of each cohort against product events and changes

Mary then covered some continuous delivery principles - which made Jez Humble and Martin Fowler adjust their subsequent presentation somewhat :-)

• Low Dependency Architecture
• Feature Toggles
• Trunk-based development
• Branch by abstraction
• Canary releasing

Continuous Delivery - Jez Humble, Martin Fowler

This covered many of the concepts from the excellent Continuous Delivery book. I won't repeat this content - go read the book. Some useful soundbites from the talk:

• On acquisition, Flickr were initially told by Yahoo to stop continuous deployment. The Flickr team then gained stats that they had less downtime than any of the other Yahoo services. Flickr deploy approx 10x/day to production. Downtime was 4 x 6mins per year.
• Mingle has 20 hours of acceptance tests. Running in parallel across 70 boxes, these complete in 45 minutes.
• On any acceptance test failure you should find out how it got through, and beef up your commit tests to catch the failure earlier.
• The job of the deployment pipeline is to kill code that isn't suitable for release
• Devops - development is measured on throughput, operations on stability. Need to establish a culture of collaboration, automation, measurement, sharing.
• Dark launching - deploy back end service first before switching it on
• Benefits in decoupling deployment and release (eg. using feature toggles). Facebook has every major feature to be released in the next 6 months already deployed to production, and waiting to be released.

Introducing Continuous Delivery

• Focus on behaviours, not tools
• People are the key
• Get everyone together at the beginning
• Keep meeting
• Make it easy for everyone to see what's happening (eg. big visible display)
• Kaizen. Continuous Delivery should be part of continuous improvement process. It's going to take months. Do it step-by-step.

ITIL and Continuous Delivery

Jez provided an excellent answer to a question about using Continuous Delivery with ITIL. See the last 5 minutes of the video (when it comes out).

• The deployment pipeline is a superior way to manage risk (over paperwork)
• Under ITIL v3, continuous delivery can be approved as a standard change with known low risk.
• 2 other strategies for integrating with ITIL v3
• Need to work closely with auditor
• Auditors love deployment pipelines, everything you could possibly audit is available and version controlled.

Adventures in Infrastructure as code - Julian Simpson

Julian demonstrated code using Puppet and Chef, that installs Ubuntu, nginx, jetty and deploys a war file. In short, Puppet is configuration-based and appeals more to sysadmins who like to see all the moving parts. Chef is convention-based and tends to appeal to developers more.

He discussed testing using cucumber-puppet and cucumber-nagios, and mentioned cfengine, vagrant and the veewee plugin.

The Limited Red Society - Joshua Kerievsky

Parallel to the Limited WIP Society, the goal is to minimise the development time spent in the red (failing tests), or in the pink (code that won't compile).

By visualising the development flow, developers get feedback for improvement. Analogies with sports visualisations. Joshua recommended "The art of winning an unfair game" book, on which the Moneyball movie was based.

Measure and practice the Shortest Longest Red Etude. A good retreat is better than a bad stand.

Refactoring strategies missed in Joshua's book:

• Parallel Change - write refactored code alongside current code and gradually switch over (like building a 2nd bridge)
• Narrowed Change - reduce the number of change points (eg. using Extract Method), before making the change. This is preferred over parallel change.

We had covered these in depth on Joshua's excellent Refactoring Workshop on Wednesday. The eLearning package that Industrial Logic have put together is exceptional, with IDE plugins to allow us to visualise our refactoring sessions and provide feedback on our work.

The development visualisation is available in Industrial Logic's Sessions tool.

Technology Sessions

Domain-driven design for RESTful Systems - Jim Webber

HTTP is an Application Protocol. You have to narrow it down to your Domain Application Protocol.

PUT is idempotent. Always use it rather than POST for financial transactions.

Jim presented some of the Restbucks examples from his book.

His talk was very engaging. It was somewhat alarming to find that only 1 of the audience had played a PS/1 or later as their first computer game, while over 50% of the audience had first played games that were loaded from cassette tape.

The Live Web - Drag n' drop in the Cloud - Dan Ingalls

A component architecture for HTML5. Builds a structured scene graph in JavaScript. Integrates HTML, SVG and Canvas graphics.

It's like Squeak (and Scratch, BYOB, Snap) but runs behind firewall

Kit approach - components, connections, encapsulation

Parts bin - cloud based repository of active content

Live connect, dataflow binding

The IDE runs using Lively Web, and saves using WebDAV (to node.js, to svn)

Not sure how I'd use this for commerical projects yet, but it's a lot of fun and I'll be trying it out with my son.

Better testing with less work - John Hughes

I've previously looked at (and discounted) QuickCheck in Java, but the Erlang version does so much more that I'll be looking at it again. There is a lot of difference between the different language implementations.

QuickCheck allows us to write general properties that a system should meet. The properties should hold true across a range of values (similar to JUnit Theories). It then generates random inputs to test those properties. On failure it will retry with simpler test cases until it finds the minimal test case that fails. This simplifies debugging.

A particular sweet spot for QuickCheck is finding race conditions. It's easy to convert sequential to parallel tests. John used QuickCheck to easily find some race conditions in a database implementation that had been eluding the developers for months.

NodeJS and the JavaScript-everywhere strategy - Matthew Eernisse

Matthew discussed how they're using Node at Yammer. Three of their customer-facing services use Node - the Upload service, the collaborative document editing service and an undisclosed new service.

Internally, Yammer use Jake and FooUnit for build and test. It's good to see Yammer developers open sourcing some of their work on Github.

Matthew described the Geddy web framework and showed how they use the yammer.util.promise framework for async code.

Webbit - a lightweight WebSocket web server for Java - Aslak Hellesoy

Webbit is a single-threaded, non-blocking, Java-based, high throughput web server (no servlet API). It supports 3 models:

• HTTP request-response
• WebSocket (long-lived 2 way connections)
• EventSource (long-lived push server->client connections)

EventSource is a W3C standard and is implemented in most decent browsers. It's more lightweight than WebSockets.

At DRW, Aslak's team ported a trading app to Webbit, Chrome, WebSockets.

Aslak coded a HTTP client app in less than 100 lines (including Java, JS, HTML, CSS). See https://gist.github.com/1421652.

Blocking or long operations must be offloaded to another thread.

Webbit has no hot-deploy capability out-of-the-box, and does not support HTTPS. (Aslak recommended stud as a HTTPS terminating proxy).

For testing, Webbit provides StubRequest, StubResponse and StubConnection classes. For end-to-end tests, the server can be started and stopped in 1.5ms.

Webbit tends to be embedded in an app (rather than the app being embedded in the web server). It has a small footprint, using about 100kb RAM and almost no CPU when idle. It has REST (JAX/RS) and RPC add-ons.

Above the Clouds - Introducing Akka - Peter Vlugter

It was a surprise to find out that NZ has a developer working for Typesafe as part of the core Akka development team.

Peter introduced the core features - Actors, STM (Software Transactional Memory) and Dataflow (a way of composing futures, inspired by the Oz framework). They are looking at supporting Agents and Transactors (2 phase commit STM).

Akka supports Scala and Java APIs.

Peter gave a good, detailed introduction to Scala Actors.

Akka integrations include Camel, Spring, ZeroMQ, AMQP, Play framework, REST. It has a testing toolkit (and Peter is keen to explore a QuickCheck for actors), It is open-source, except for the commercial Atmos tool (trace, monitor, manage, provision).

General

The keynotes were entertaining and insightful. Hearing Cameron Prudy describe potential JVM enhancements gave me some warm and fuzzies about Java's future as a platform.

Mike Lee, Damian Conway, Mary Poppendieck, Simon Peyton-Jones and Jim Webber were outstanding speakers.

Thanks to the great company - especially John Hurst, Martin Paulo, Paul and Luke (the AEMOs) and Balaji, and to everyone else that made this great.

And above all, thanks to Dave Thomas, the YOW committee, the volunteers and speakers who made YOW happen.

WOW - that turned into an epic. Illustrating just how much I got out of YOW!

2 - What I will do next

Books to read

The Lean Startup - Eric Ries

REST in Practice - Jim Webber

Technologies to play with

Lively Web

QuickCheck

Node.js

Webbit

Presentations to watch online (that I missed)

The Post-Java Virtual Machine - Ola Bini

JRuby for the win - Ola Bini

Three 'Tall' Tales - Kevin O'Neill (mobile dev and testing)

3 - Impediments

Above all, the speakers brought out the joy of programming (especially Damian Conway!)

My main impediment is time. I really need to re-instate my "20% time" for research, play, innovation and general hacking.

Must be about time for my new year's resolutions..

A message from the past from someone that cared

Last year I saw a tweet that a failing unit test was "a message from the past from someone that cared".

I was reminded of this recently when picking up a project that hadn't been touched for over 2 years. One of the acceptance tests failed with the following error:



The error message was sufficiently detailed to lead me straight to the solution. Thankfully the test writer¹ had cared enough to make sure that the failing test provided sufficient detail to be easily fixed.

The original tweeter was no doubt referring to failing unit tests helping to detect unwanted code regressions. When relying on external configuration or data, tests should also check any assumptions they are making about the data. In this case, the Given clause of the "Given-When-Then" statement is clear about its expectations, and the test fixture is checking that these preconditions are met.

This type of environment issue could even be treated as a different type of test failure, as discussed in Triaging Test Failures.


¹ OK the test writer was me :-) But since I hadn't worked on the application for over 2 years, it may as well have been someone else!

Getting Groovy with Jar Versions

Oh how I love the Groovy!

I just needed to check version numbers of jar files in multiple folders and collate a table like:

While most jar files have the version number in either the Bundle-Version or Implementation-Version attributes of the MANIFEST.MF file, some do not follow this convention, and we need to either look for a file ending in _VERSION, or parse a pom.xml within a META-INF subfolder to get the version number.

Groovy makes it so easy to do this stuff. On top of this, creating the HTML table is a breeze. The icing on the cake is to save the file and open in Firefox in a couple of lines.

This would have taken hundreds of lines in Java. Groovy makes it concise, while allowing access to all Java libraries, resulting in code that is readable (if a little alien at first) to a Java developer.

concordion-extensions is now a Concordion sub-project

I'm pleased to announce that concordion-extensions is now a sub-project of Concordion. This project includes extensions to add screenshots and logging information to Concordion output, and to change the format of the timestamp in the footer. See http://concordion.org/Extensions.html for further details.

An updated release, concordion-extensions v1.0.1, is now also available in Maven central. See http://concordion.org/Download.html for details.

The classes have been repackaged under org.concordion.ext. Any users of concordion-extensions v1.0.0 will need to update their package definitions with this new release.

If anyone has ideas for additional extensions, please discuss them on the Concordion mailing list. add them to http://code.google.com/p/concordion-extensions/wiki/ExtensionIdeas.

Update:
Oct 26, 2014. These extensions are now packaged as individual projects and hosted on Github. See http://concordion.org/Extensions.html for details.

Displaying screenshots on acceptance test failures

When running GUI tests it can be difficult to determine what was being shown on the GUI at the point of failure, especially if the tests are running minimised or on a CI server.

We developed a screenshot extension for Concordion to include screenshots in the output whenever failures occur. We've been finding this extension incredibly useful for diagnosing intermittent failures in our automated acceptance tests.

This requires the new Concordion 1.4.1, which introduces support for extensions. Amongst other things, extensions can add new listeners to commands and make changes to the Concordion output.

The following example is running an end-to-end test using WebDriver (Selenium 2) to test a Google search through the Firefox browser.

While the results show that the test is failing since Netherlands is displayed in the Google results, it would be helpful to see the actual results page. The screenshot extension allows us to hover over the failure to see an image of the browser page at the time the failure occurred.



Clicking on the image opens it for further inspection:

In this example, we've configured the extension to use WebDriver's TakesScreenshot interface, so we see an image of just the browser page, irrespective of whether it is currently visible on the screen.

The extension can also be used to take screenshots on success, and includes a command to explicitly take screenshots.

The screenshot extension project is on Github. You'll need to set the concordion.extensions system property to use it - see the README for details.

The source code for this example is in the demo project.

Acknowledgements:
This extension was partly inspired by Mark Derricutt's ScreenshotCommand, and by Adam Setch's post to the Concordion list.

UPDATESOct 25 2010. Added acknowledgements
Jan 03 2011. Project moved to Google Code and docs to Concordion site.

Oct 26 2014. Updated project and demo location to Github projects

What's happening in my acceptance tests?

Agile Acceptance Testing allows us to describe desired behaviour using examples that describe the business intent. Good acceptance tests are written as plain language specifications, not scripts. Implementation details are coded in a separate test "fixture" class.

One downside of this approach is a loss of transparency of what the tests are actually doing. The "fixtures" are often written by developers, which may need a leap in faith for testers to trust. On a recent project, this trust was dented when the tests didn't do what they were supposed to be doing.

With this in mind, I set out to provide more insight into what our tests are actually doing, without undermining the principles of acceptance testing.

The result is a logging extension for Concordion. This adds a "tooltip" to the Concordion output HTML that shows log output when hovered over:


This tooltip is proving useful not only to testers, but also for developers to gain insight into what is happening in their tests and to find performance improvements. For example, in the above example we were surprised to see the web page being loaded twice, and a number of element lookups being duplicated.

This approach could also be used for user documentation of the steps required to complete an action, potentially with screen shots, or even as an embedded screen cast.

Implementation Details
We've added a new extension mechanism to Concordion 1.4.1 to make it easy to add features such as this.

This extension is available on Github. The extension captures java.util.logging output and has a number of configuration options. You'll need to set the concordion.extensions system property to use it - see the README for details.

For a example using this Concordion extension with WebDriver, see the demo project .

UPDATESOct 4  2010. Changed github link to new concordion-extensions project
Oct 6  2010. Source code moved to trunk of Concordion
Oct 24 2010. Updated to reference Concordion 1.4.1 and concordion-extension-demo project
Jan 03 2011. Project moved to Google Code and docs to Concordion site.

Oct 26 2014. Updated links to new Github projects.

Triaging test failures

One of the goals of Open Spaces conferences is to turn "corridor conversations" into the focal point of the conference. This was aptly demonstrated at CITCON ANZ when Richard Vowles introduced a topic we'd been discussing over kebabs the night before.

Richard has subsequently discussed the topic on the Illegal Argument podcast. This post is an extension of the discussion.

The problem
When running integration and acceptance tests, test failures may be caused by factors other than incorrect code. This is most apparent when performing end-to-end testing through to Enterprise Information Systems. A number of factors can cause the test to fail - system unavailable, test data not in required state.

It would be useful to categorise the failures by cause, for notification and reporting purposes. Developers should be notified of code-related issues, testers might be responsible for data issues, and sys ops for server errors. Over time it would also be useful to visualise how often server and data errors are occurring.

Richard provided the example:

Given a customer has been in arrears for over 90 days...

In order to run this test in an end-to-end environment, the test code has to get a customer in this state. Richard's system uses an AS/400 back-end, and it simply is not possible to automate the setup of a customer in this state. The test code may need to be configured with a specific customer id, or it may be smart enough to search for a customer in the required state.

Over time, the customer data may no longer be available. For example, periodic data refreshes may remove or update the customer details.

Richard's not the only one with this problem - I'm also seeing it on a current project.

The problem of finding adequate test data is exacerbated when the test updates the state:

Given a customer has been in arrears for over 90 days,
when her invoices are paid in full,
then her status is changed to black.

In this case, the destructive change to the customer's state means that the data is no longer suitable for running this test. The test needs to find a different customer in arrears the next time it is run. Since debtors are a finite resource, the test may be unrunnable at some stage.

Why run these fragile end-to-end tests?
With Agile Acceptance Testing (ATDD, BDD etc), the focus is on testing business examples that will prove to the customer that a feature is "done". Running the tests end-to-end provides the greatest assurance to the customer that the functional requirements are being met, and reduces the need for manual regression testing.

Depending on the project, it may be possible to implement these tests "under the covers" of the user interface, or using mocks for back-end functionality. We often use these approaches to drive the design of our code, possibly before the user interface or back-end are available. However, these approaches don't provide the full benefits that we get from end-to-end tests.

"Unrunnable" test result
In the past, I have tackled this issue by making assertions on the Given clause of the tests. If the test pre-conditions are not met, the test results in a failure.

The proposal made at CITCON is to introduce a new "Unrunnable" test result state. This state is neither success nor failure. The discussion led to introducing a new colour for this state, to differentiate it from red (failure) and green (success).

Triaging test results
Extending the idea, it would be useful to be able to triage test failures into user-defined categories. Depending on the nature of the failure (and possibly the severity) the failure would be assigned a category.

The CI server would send failure notifications to a category-specific list. For example, system failures would be notified to sys ops, data issues to testers, and code issues to developers.

Each category would be displayed with a different failure colour, allowing the causes of test failures to be tracked over time.

For some categories, for example server errors, it may not be worthwhile continuing with the test run. The test runner could potentially be configured to abort the test run dependent on the category of the failure.

Comparison with Pending state
Many BDD and ATDD tools already model a separate Pending or Unimplemented state - displayed in yellow (Cucumber), or grey (Concordion). The pending state can be viewed as one of these test failure conditions ("code unavailable").

Example
A test could be annotated as follows:

@Triage(nature="SERVER_ERROR", exception=HostUnavailableException.class)
public class ArrearsFixture() {
@Triage(nature="DATA_UNAVAILABLE",exception=NoDataException.class)
public Customer findCustomerInArrears(Condition condition) {
....
}
}

On Hudson, test failures might show up as:

clearly showing that there is an ongoing issue with server errors, impacting on the team's ability to adequately test the system. Intermittent data-releated issues are also causing some tests to be unrunnable.

I'm not aware of any test tools/frameworks currently offering this capability. Does anyone know of anything similar?

UPDATE:
1. This topic was discussed on the Illegal Argument list. I liked Mark Derricutt's point:

It's my thought that the finer grained reporting you CAN get the better,
whether you make use of it depends on the project and problem space.

After all - Exception is good enough to be thrown for all errors right?
IllegalArgumentException, IOException, FileNotFoundException are all rather
"controversial, mean many things to many people, and cause inconsistency and
confusion" - but we need that differentiation of exceptions to separate out
a chain of responsibility.

We know this in our code, but I think we also need this for our
builds/tests.

Creating Executable Specifications for Swing apps with Concordion and FEST

The FEST Swing Module provides an easy-to-use API for testing Java applications that use the Swing GUI toolkit.

For a recent project, we created executable specifications with Concordion that used FEST as a driver for the desktop Swing-based GUI. The combination was very powerful, resulting in readable specifications of the system behaviour that invoked end-to-end tests through the GUI.

Here are some tips on using FEST effectively with Concordion:

1) Use FEST's built-in checks that all of the GUI updates are performed in the Event Dispatch Thread (EDT):

  @BeforeClass
  public static void setUpEDTChecks() {
      FailOnThreadViolationRepaintManager.install();
  }

2) Install a big red switch. When running the tests locally, they're going to take over your desktop. Use FEST's EmergencyAbortListener to allow you to terminate the GUI using ctrl - shift - A:

  @BeforeClass
  public static void setUpAbortListener() {
      listener = EmergencyAbortListener.registerInToolkit();
  }

  @AfterClass
  public static void tearDownAbortListener() {
      listener.unregister();
  }

3) Create your user interface once for all tests:

   public static UserInterface getInstance() {
       if (instance == null) {
           instance = new UserInterface();
       }
       return instance;
   }

4)) Create your window within the Event Dispatching Thread (EDT)

  private FrameFixture window;

   private UserInterface() {      
       GuiActionRunner.execute(new GuiTask() {
           public void executeInEDT() {
               JFrame frame = ... // code to create main frame
               window = new FrameFixture(frame);
               setLookAndFeel();
           }
       });
       window.show();    

5) Slow the tests down when demonstrating to users, so they can see what's happening:

   public void slowDown() {
      window.robot.settings().delayBetweenEvents(500);
      window.robot.settings().eventPostingDelay(500);
   }

6) When using FEST's assertions, you can allow assertion failures to bubble up to Concordion as AssertionErrors. However, you will get more readable Concordion output if you translate them. For example, with the following Concordion specification:

   <p>Only things of type 'Thing One' should be prioritised....</p>
   <table concordion:execute="#prioritised =
           isPrioritised(#thingType)">
       <tr><th concordion:set="#thingType">Thing Type</th>
           <th concordion:assertEquals="#prioritised">Prioritised</th></tr>
       <tr><td>Thing One</td><td>Yes</td></tr>
       <tr><td>Thing Two</td><td>No</td></tr>
   </table>

, the corresponding fixture can be written as:

   public String isPrioritised(String thingType) {
       doStuffWithThing(thingType);
       return userInterface.isPrioritised(request) ? "Yes" : "No";
   }

, and integrated with Fest using:

   public Boolean isPrioritised(Thing thing) {
       int rowIndex = getRowIndex(thing);
       try {
           table.fontAt(row(rowIndex).column(col)).requireBold();
       } catch (AssertionError e) {
           return false;
       }
       return true;
   }

7) Follow all of the usual hints and tips for creating Concordion specifications. In particular:

• Write Specifications not Scripts.


• Don't mention the GUI. Your specification should be independent of the implementation.


• Evolve a domain-specific language

I'll write a follow-up to describe how we got these tests running in our Continuous Integration server.

Correction about Greenpepper

In my recent APN presentation on Agile Acceptance Testing, I stated that Atlassian had bought Greenpepper (the tool).

Well I was wrong. I showed some of the slides at CITCON ANZ last weekend, and the Atlassian guys corrected me that they had bought Greenhopper (the tool) from Greenpepper (the company), but not Greenpepper (the tool).

What types of application can I test with Concordion?

Following on from my Agile Acceptance Testing presentation at the APN on Monday, there were some questions about what types of application you could test with Concordion.

Concordion itself is a test framework which runs your executable specifications using test fixtures.

The executable specifications are written in HTML and are independent of any programming language. The same executable specifications can be used with Concordion for Java, or Concordion.NET. (and a subset of the syntax can be used with the Ruby and Python ports of Concordion).

The test fixtures are written in a programming language. For the Java version of Concordion, the test fixtures are written as JUnit classes. (However they don't include test methods and don't tend to include assertions, since these are defined in the executable specification.)

There are 2 main ways in which Concordion is used. Either through the User Interface, or "under the covers" to the application or services layer.

When testing "under the covers" of the application, the test fixture calls into the application code directly. In this case, the type of application is largely irrelevant.

When testing through the user interface, you need a Test Driver. Test drivers provide the functionality to find widgets, click buttons, enter text etc. There are a lot of high quality, open source test drivers available. Some of these, such as Selenium and SoapUI also come with their own test frameworks, but can be used purely as test drivers. The test drivers I have used with Concordion are:

Application Type	Test Driver
Web Application	WebDriver and Selenium
Swing GUI Application	Fest
Web Service	SoapUI

Additionally there are a lot of test extension libraries available, such as DbUnit and XmlUnit that can be called from your test fixture.

Concordion is a great framework for running your executable specifications. When combined with the right driver you can test pretty much any type of application.

So what does that make me?

While tarring up an old file system:

tar: .: implausibly old time stamp 1970-01-01 12:00:00

Using SchemaSpy with multiple schema

A happy coincidence occurred last week.

One of my current projects has a database with foreign key relationships spanning across two schemas. I needed to create some database documentation.

And SchemaSpy v4 was in late beta. One of the great new features of v4 is support for multiple schema. John Currier was short of a "real" database to test these changes on.

John gave me great support in working through a few issues and fixing up a couple of bugs, that have made it into the SchemaSpy 4.0.0 release.

The result is a new main page showing all of the selected schema:



Drilling down, the relationship graph now shows related tables in foreign schema:



SchemaSpy does an excellent job of hyperlinking all of the tables, columns and relationships, allowing you to easily navigate around your schema.

To use the multiple schema feature, add a -all parameter to your SchemaSpy command line. With this parameter, SchemaSpy will include all schema except for system tables (which are determined by a schemaSpec pattern in the properties file for the specified database type). To further restrict the schema, add a -schemaSpec command line parameter with a regular expression to match the required schema.

The following ant task instructs SchemaSpy to document the PMH and SHO schema of our DB2 database.

<java jar="lib/schemaSpy_4.0.0.jar" fork="true"/>
<arg line="-t udbt4"/>
<arg line="-host ${db.host}"/>
<arg line="-port ${db.port}"/>
<arg line="-db ${database}"/>
<arg line="-u ${db.userid}"/>
<arg line="-p ${db.password}"/>
<arg line="-all"/>
<arg line="-schemaSpec (PMH)|(SHO)"/>
<arg line="-dp ${DB2_LIB}/db2jcc.jar:${DB2_LIB}/db2jcc_license_cu.jar"/>
<arg line="-o ${reports.data.model.dir}"/>
</java>

Crap4J Hudson update

The last graph I posted didn't really do the Crap4J Hudson plugin justice, especially since Daniel Lindner fixed the bug where %age figures disappeared when Crappyness was < 1%.

Daniel replied to me at the time that "when your code isn't crap, the Crap4J plugin is". Well, now the plugin's fixed, I'd better start cutting the crap in my code :-)

I'll be doing a mini-presentation on Crap4J at the Wellington Java User Group meeting tomorrow.

CITCON Melbourne '08

The Continuous Integration and Testing Conference (CITCON) recently held in Melbourne was full of insights.

The Friday night session taught me just how many people really don't like Brussel Sprouts and Vegemite (in isolation, though it would make an interesting mix).

This was the first Open Space conference for the majority of the attendees (including me), and I was blown away by the energy that people brought to the conference, and the number of great sessions that sprung up (see the whiteboard and the wiki).

Jeremy Wales presented on his experience with Acceptance TDD on a Suncorp project. This led to some interesting discussions on (not) using the DRY principle with acceptance tests, and having testers involved upfront in designing the tests.

Other great sessions included Continuous Performance Testing, Show us your Build!, and Using Groovy in Testing.

I've come away with a few new tools to try (including Concordion for writing active specifications, and JChav for charting JMeter results), a renewed focus on writing my tests in Groovy and plenty to read...

Thanks to all who made the conference happen. The format was fantastic and I'm already looking forward to next year.

Crap4J, Hudson and Windows

There is now a Crap4J plugin for the Hudson continuous integration server, thanks to Daniel Lindner.

For those of you unfamiliar with Crap4J, it is a metric that "combines cyclomatic complexity and code coverage from automated tests to help you identify code that might be particularly difficult to understand, test, or maintain".

The Hudson Crap4J plugin maintains "Crappyness Trends", which is a feature missing from the standard Crap4J reports. It also shows details of any methods that exceed the crap threshold.

In order to use the plugin, you must first download the Crap4J Ant task and integrate it into your Ant build file. If you're running Windows, this is where you're likely to run into a road-block. On Windows, a bug stops the Ant task from figuring out the Crap4J home. The workaround is to set the ANT_OPTS environment variable:

set ANT_OPTS="-DCRAP4J_HOME=c:\java\tools\crap4j-ant"

, where c:\java\tools\crap4j-ant is the location of your Crap4J ant tasks.

Once your Ant build is producing the Crap reports, you're ready to integrate it into Hudson.

After downloading and installing the plugin, you'll need to add the Ant target that is running the Crap4J task. In the example, I've set up a crap4j target.

Then, on Windows, you'll need to add the Ant options to detect the Crap4J home. Click on the "Advanced..." button in the Build section. In the Java Options, enter

-DCRAP4J_HOME=c:\java\tools\crap4j-ant

, where c:\java\tools\crap4j-ant is the location of your Crap4J ant tasks.

Then, in the Post-build Actions section, specify the location of the output from your Crap4J ant task.


The next time Hudson runs your build, look for the toilet roll on the left of the dashboard for your Crap details.

A few comments on the current plugin (currently at v0.2):

1. It would be great to be able to adjust the CRAP threshold. The default threshold of 30 is really too high for new code, and I typically set it to 15.
2. I'd also love to be able to view the complexity, coverage and CRAP scores for all methods, not just the scores for the methods over the CRAP threshold.
3. When the crap method percentage is less than 1, the Crappyness Trend chart does not show any %age figures on the scale.

Static imports for Hamcrest and Theories

The Hamcrest and Theories features of JUnit 4.4 rely on a number of static imports.

Code completion for static imports is tricky. For example, if I want to use the both Matcher, I firstly have to remember that it is in the JUnitMatchers class, type in JUnitMatchers (or JUM), then ctrl-space to get code completion to fill in the import, then type in .both and remove the JUnitMatchers.

As mentioned in the JUnit 4.4 release notes, Eclipse provides a Favorites preference that automatically includes your favourite classes in code assist. For example, after setting up JUnitMatchers in your Favorites, you can type in both, ctrl-space, and Eclipse will import JUnitMatchers.both.

Adding the static types shown here to your Eclipse Favorites will make your JUnit 4.4 journey a lot smoother.



PS. The both Matcher allows you to create assertions such as:

assertThat(e.getMessage(), both(startsWith("Invalid environment")).and(containsString(environmentName));

as opposed to the more common:

assertThat(e.getMessage(), allOf(startsWith("Invalid environment"), containsString(environmentName)));

It's debatable which is cleaner. The top statement reads closer to the English language, but is longer, more complex to construct and can't have additional matchers added in the same way that allOf can.

On the eradication of software defects

I loved Andy Glover's hip comparison of mousetraps to testing tools and mice to defects.

I live in a country that had no mammals, other than a few bats, and no software defects, until man arrived around 1000 years ago. The introduced mammals have had a devastating effect on the native biodiversity. A desire to protect the remaining species has led New Zealand to be world-renowned in the removal of introduced species. I wish I could say the same for software defects.

K.P. Brown and G.H. Sherley, in their paper describing the eradication of possums on Kapiti Island, show the following success rates for different phases of the trapping programme.



The first phase, Commercial Trapping, resulted in a 24% success rate of trapping possums. Once trapping stopped being a commercial proposition, 4 trappers were paid to set up to 1500 traps per night and had a success rate of 0.7%. The final Eradication phase introduced dog teams in addition to the trapping program. At this stage, the trapping success rate was down to 0.007%, and dogs caught the remaining 32 trap-shy possums.

I suspect that similar success rates would be encountered in finding software defects. A good unit testing program, at a cost of 3-4 lines of test code for each line of code under test, might yield a success rate similar to Commercial Trapping. Some of the remaining bugs would be detected in intensive system testing. Many escape undetected into production.

Most development shops don't even make it through the Commercial Trapping phase.

The true cost of the $1 mouse trap is in the time, and cheese, taken to set it.

Agitar changed the equation. At the press of a button, Agitator would set the traps for me hundreds if not thousands of times. This would drive my unit testing past the Commercial Trapping into Intensive Control territory. With a bit more tuning, I could even start dreaming of bug eradication.

My hat goes off to those hardy souls who rid Kapiti Island of possums (and mice, rats, stoats etc.) and to the folks at Agitar who wanted to make it easier for us to eradicate those damn electronic vermin.