What's happening in my acceptance tests?

Agile Acceptance Testing allows us to describe desired behaviour using examples that describe the business intent. Good acceptance tests are written as plain language specifications, not scripts. Implementation details are coded in a separate test "fixture" class.

One downside of this approach is a loss of transparency of what the tests are actually doing. The "fixtures" are often written by developers, which may need a leap in faith for testers to trust. On a recent project, this trust was dented when the tests didn't do what they were supposed to be doing.

With this in mind, I set out to provide more insight into what our tests are actually doing, without undermining the principles of acceptance testing.

The result is a logging extension for Concordion. This adds a "tooltip" to the Concordion output HTML that shows log output when hovered over:


This tooltip is proving useful not only to testers, but also for developers to gain insight into what is happening in their tests and to find performance improvements. For example, in the above example we were surprised to see the web page being loaded twice, and a number of element lookups being duplicated.

This approach could also be used for user documentation of the steps required to complete an action, potentially with screen shots, or even as an embedded screen cast.

Implementation Details
We've added a new extension mechanism to Concordion 1.4.1 to make it easy to add features such as this.

This extension is available on Github. The extension captures java.util.logging output and has a number of configuration options. You'll need to set the concordion.extensions system property to use it - see the README for details.

For a example using this Concordion extension with WebDriver, see the demo project .

UPDATESOct 4  2010. Changed github link to new concordion-extensions project
Oct 6  2010. Source code moved to trunk of Concordion
Oct 24 2010. Updated to reference Concordion 1.4.1 and concordion-extension-demo project
Jan 03 2011. Project moved to Google Code and docs to Concordion site.

Oct 26 2014. Updated links to new Github projects.

Triaging test failures

One of the goals of Open Spaces conferences is to turn "corridor conversations" into the focal point of the conference. This was aptly demonstrated at CITCON ANZ when Richard Vowles introduced a topic we'd been discussing over kebabs the night before.

Richard has subsequently discussed the topic on the Illegal Argument podcast. This post is an extension of the discussion.

The problem
When running integration and acceptance tests, test failures may be caused by factors other than incorrect code. This is most apparent when performing end-to-end testing through to Enterprise Information Systems. A number of factors can cause the test to fail - system unavailable, test data not in required state.

It would be useful to categorise the failures by cause, for notification and reporting purposes. Developers should be notified of code-related issues, testers might be responsible for data issues, and sys ops for server errors. Over time it would also be useful to visualise how often server and data errors are occurring.

Richard provided the example:

Given a customer has been in arrears for over 90 days...

In order to run this test in an end-to-end environment, the test code has to get a customer in this state. Richard's system uses an AS/400 back-end, and it simply is not possible to automate the setup of a customer in this state. The test code may need to be configured with a specific customer id, or it may be smart enough to search for a customer in the required state.

Over time, the customer data may no longer be available. For example, periodic data refreshes may remove or update the customer details.

Richard's not the only one with this problem - I'm also seeing it on a current project.

The problem of finding adequate test data is exacerbated when the test updates the state:

Given a customer has been in arrears for over 90 days,
when her invoices are paid in full,
then her status is changed to black.

In this case, the destructive change to the customer's state means that the data is no longer suitable for running this test. The test needs to find a different customer in arrears the next time it is run. Since debtors are a finite resource, the test may be unrunnable at some stage.

Why run these fragile end-to-end tests?
With Agile Acceptance Testing (ATDD, BDD etc), the focus is on testing business examples that will prove to the customer that a feature is "done". Running the tests end-to-end provides the greatest assurance to the customer that the functional requirements are being met, and reduces the need for manual regression testing.

Depending on the project, it may be possible to implement these tests "under the covers" of the user interface, or using mocks for back-end functionality. We often use these approaches to drive the design of our code, possibly before the user interface or back-end are available. However, these approaches don't provide the full benefits that we get from end-to-end tests.

"Unrunnable" test result
In the past, I have tackled this issue by making assertions on the Given clause of the tests. If the test pre-conditions are not met, the test results in a failure.

The proposal made at CITCON is to introduce a new "Unrunnable" test result state. This state is neither success nor failure. The discussion led to introducing a new colour for this state, to differentiate it from red (failure) and green (success).

Triaging test results
Extending the idea, it would be useful to be able to triage test failures into user-defined categories. Depending on the nature of the failure (and possibly the severity) the failure would be assigned a category.

The CI server would send failure notifications to a category-specific list. For example, system failures would be notified to sys ops, data issues to testers, and code issues to developers.

Each category would be displayed with a different failure colour, allowing the causes of test failures to be tracked over time.

For some categories, for example server errors, it may not be worthwhile continuing with the test run. The test runner could potentially be configured to abort the test run dependent on the category of the failure.

Comparison with Pending state
Many BDD and ATDD tools already model a separate Pending or Unimplemented state - displayed in yellow (Cucumber), or grey (Concordion). The pending state can be viewed as one of these test failure conditions ("code unavailable").

Example
A test could be annotated as follows:

@Triage(nature="SERVER_ERROR", exception=HostUnavailableException.class)
public class ArrearsFixture() {
@Triage(nature="DATA_UNAVAILABLE",exception=NoDataException.class)
public Customer findCustomerInArrears(Condition condition) {
....
}
}

On Hudson, test failures might show up as:

clearly showing that there is an ongoing issue with server errors, impacting on the team's ability to adequately test the system. Intermittent data-releated issues are also causing some tests to be unrunnable.

I'm not aware of any test tools/frameworks currently offering this capability. Does anyone know of anything similar?

UPDATE:
1. This topic was discussed on the Illegal Argument list. I liked Mark Derricutt's point:

It's my thought that the finer grained reporting you CAN get the better,
whether you make use of it depends on the project and problem space.

After all - Exception is good enough to be thrown for all errors right?
IllegalArgumentException, IOException, FileNotFoundException are all rather
"controversial, mean many things to many people, and cause inconsistency and
confusion" - but we need that differentiation of exceptions to separate out
a chain of responsibility.

We know this in our code, but I think we also need this for our
builds/tests.

Creating Executable Specifications for Swing apps with Concordion and FEST

The FEST Swing Module provides an easy-to-use API for testing Java applications that use the Swing GUI toolkit.

For a recent project, we created executable specifications with Concordion that used FEST as a driver for the desktop Swing-based GUI. The combination was very powerful, resulting in readable specifications of the system behaviour that invoked end-to-end tests through the GUI.

Here are some tips on using FEST effectively with Concordion:

1) Use FEST's built-in checks that all of the GUI updates are performed in the Event Dispatch Thread (EDT):

  @BeforeClass
  public static void setUpEDTChecks() {
      FailOnThreadViolationRepaintManager.install();
  }

2) Install a big red switch. When running the tests locally, they're going to take over your desktop. Use FEST's EmergencyAbortListener to allow you to terminate the GUI using ctrl - shift - A:

  @BeforeClass
  public static void setUpAbortListener() {
      listener = EmergencyAbortListener.registerInToolkit();
  }

  @AfterClass
  public static void tearDownAbortListener() {
      listener.unregister();
  }

3) Create your user interface once for all tests:

   public static UserInterface getInstance() {
       if (instance == null) {
           instance = new UserInterface();
       }
       return instance;
   }

4)) Create your window within the Event Dispatching Thread (EDT)

  private FrameFixture window;

   private UserInterface() {      
       GuiActionRunner.execute(new GuiTask() {
           public void executeInEDT() {
               JFrame frame = ... // code to create main frame
               window = new FrameFixture(frame);
               setLookAndFeel();
           }
       });
       window.show();    

5) Slow the tests down when demonstrating to users, so they can see what's happening:

   public void slowDown() {
      window.robot.settings().delayBetweenEvents(500);
      window.robot.settings().eventPostingDelay(500);
   }

6) When using FEST's assertions, you can allow assertion failures to bubble up to Concordion as AssertionErrors. However, you will get more readable Concordion output if you translate them. For example, with the following Concordion specification:

   <p>Only things of type 'Thing One' should be prioritised....</p>
   <table concordion:execute="#prioritised =
           isPrioritised(#thingType)">
       <tr><th concordion:set="#thingType">Thing Type</th>
           <th concordion:assertEquals="#prioritised">Prioritised</th></tr>
       <tr><td>Thing One</td><td>Yes</td></tr>
       <tr><td>Thing Two</td><td>No</td></tr>
   </table>

, the corresponding fixture can be written as:

   public String isPrioritised(String thingType) {
       doStuffWithThing(thingType);
       return userInterface.isPrioritised(request) ? "Yes" : "No";
   }

, and integrated with Fest using:

   public Boolean isPrioritised(Thing thing) {
       int rowIndex = getRowIndex(thing);
       try {
           table.fontAt(row(rowIndex).column(col)).requireBold();
       } catch (AssertionError e) {
           return false;
       }
       return true;
   }

7) Follow all of the usual hints and tips for creating Concordion specifications. In particular:

• Write Specifications not Scripts.


• Don't mention the GUI. Your specification should be independent of the implementation.


• Evolve a domain-specific language

I'll write a follow-up to describe how we got these tests running in our Continuous Integration server.

Correction about Greenpepper

In my recent APN presentation on Agile Acceptance Testing, I stated that Atlassian had bought Greenpepper (the tool).

Well I was wrong. I showed some of the slides at CITCON ANZ last weekend, and the Atlassian guys corrected me that they had bought Greenhopper (the tool) from Greenpepper (the company), but not Greenpepper (the tool).

What types of application can I test with Concordion?

Following on from my Agile Acceptance Testing presentation at the APN on Monday, there were some questions about what types of application you could test with Concordion.

Concordion itself is a test framework which runs your executable specifications using test fixtures.

The executable specifications are written in HTML and are independent of any programming language. The same executable specifications can be used with Concordion for Java, or Concordion.NET. (and a subset of the syntax can be used with the Ruby and Python ports of Concordion).

The test fixtures are written in a programming language. For the Java version of Concordion, the test fixtures are written as JUnit classes. (However they don't include test methods and don't tend to include assertions, since these are defined in the executable specification.)

There are 2 main ways in which Concordion is used. Either through the User Interface, or "under the covers" to the application or services layer.

When testing "under the covers" of the application, the test fixture calls into the application code directly. In this case, the type of application is largely irrelevant.

When testing through the user interface, you need a Test Driver. Test drivers provide the functionality to find widgets, click buttons, enter text etc. There are a lot of high quality, open source test drivers available. Some of these, such as Selenium and SoapUI also come with their own test frameworks, but can be used purely as test drivers. The test drivers I have used with Concordion are:

Application Type	Test Driver
Web Application	WebDriver and Selenium
Swing GUI Application	Fest
Web Service	SoapUI

Additionally there are a lot of test extension libraries available, such as DbUnit and XmlUnit that can be called from your test fixture.

Concordion is a great framework for running your executable specifications. When combined with the right driver you can test pretty much any type of application.

So what does that make me?

While tarring up an old file system:

tar: .: implausibly old time stamp 1970-01-01 12:00:00

Using SchemaSpy with multiple schema

A happy coincidence occurred last week.

One of my current projects has a database with foreign key relationships spanning across two schemas. I needed to create some database documentation.

And SchemaSpy v4 was in late beta. One of the great new features of v4 is support for multiple schema. John Currier was short of a "real" database to test these changes on.

John gave me great support in working through a few issues and fixing up a couple of bugs, that have made it into the SchemaSpy 4.0.0 release.

The result is a new main page showing all of the selected schema:



Drilling down, the relationship graph now shows related tables in foreign schema:



SchemaSpy does an excellent job of hyperlinking all of the tables, columns and relationships, allowing you to easily navigate around your schema.

To use the multiple schema feature, add a -all parameter to your SchemaSpy command line. With this parameter, SchemaSpy will include all schema except for system tables (which are determined by a schemaSpec pattern in the properties file for the specified database type). To further restrict the schema, add a -schemaSpec command line parameter with a regular expression to match the required schema.

The following ant task instructs SchemaSpy to document the PMH and SHO schema of our DB2 database.

<java jar="lib/schemaSpy_4.0.0.jar" fork="true"/>
<arg line="-t udbt4"/>
<arg line="-host ${db.host}"/>
<arg line="-port ${db.port}"/>
<arg line="-db ${database}"/>
<arg line="-u ${db.userid}"/>
<arg line="-p ${db.password}"/>
<arg line="-all"/>
<arg line="-schemaSpec (PMH)|(SHO)"/>
<arg line="-dp ${DB2_LIB}/db2jcc.jar:${DB2_LIB}/db2jcc_license_cu.jar"/>
<arg line="-o ${reports.data.model.dir}"/>
</java>