The adoption of Automated Testing strategies and tools, both in and traditional , has been patchy – some and communities have embraced it, but many organizations still perceive it as a burden that just slows down development. Those that see the writing and execution of tests as an additional, costly and separate task from development have missed seeing some of the main benefits of an expertly manicured .

This 4 part series will explore the issues related to Automated Testing and make the case for their wider adoption by teams.

What makes a good Automated Test?

A good Automated Test will have the following properties:

and reliable

A test that is run with the same set of inputs (data, etc) against the same piece of code should always produce the same result.

This is particularly important when it comes to UI testing. Traditional UI testing is often done manually and as skilled or disciplined as the tester is, it is impossible to guarantee the test is run absolutely identically each time. Being able to automate UI testing to guarantee consistent test runs can avoid many false positives.

independent

The order that tests in a test suite are run should not be relied upon, so each test should not depend on a previous test passing. Tests will be added to and removed from your test suite over the life of your product, so relying on a specific test to run before another will lead to false positives over time in your testing failure reports.

A classic example of non-independent tests is a create test, like inserting a row in a database, followed by a delete test that attempts to delete the row that was just inserted by the previous test. If the first test fails, the second test will fail as well. When the developer comes to inspect why a test suite failed they see two tests failing. If they decide to debug the delete test first they will spend a lot of wasted time looking for a bug that doesn’t exist.

scoped correctly

There are many types of tests that can be written and automated. Each type of software product will naturally lend itself to be tested in certain ways and not in others (e.g. UI tests might not be helpful when testing a data backup product). So you should first be aware of the type of test you are writing before you start (is it a unit test, an integration test etc) and also understand what aspects of the product that type of test should and shouldn’t try to test.

For example a Unit Test is often considered the most specific kind of test that might only test a single method or even part of a method. In a Unit Test it is not advisable to reach out and connect to other systems (a web service, a database) because by definition you will be exercising a lot more code than just a single method. Instead strategies like mocking out external systems can be used to isolate a small piece of code and test it fully without a reliance on other systems being available.

short running

This is simple – the faster a test runs, the more often it can (and will be) run. This applies at the single test level and extrapolates to the test suite level. Faster is better, but fast is a relative term. For a Unit Test, fast might mean a few milliseconds. For a full run of a UI test suite, this might mean a few hours. Whatever kind of test you are writing, make sure it is as fast as possible, and when you can’t make your tests any faster, start buying faster hardware to run them on.

Many build systems, have ways to enable and disable the running of test suites and individual tests, which can be used where test runs are too long to be practical. For example, an individual developer might be interested in quickly running the subset of tests that relate to the code he is currently . However, having all tests enabled for every build should be the default setting, disabling and manipulating which tests are run should be the exception.

run often

At a bare minimum, your tests should be running on your Continuous Integration (CI) server – if your CI server is not running a comprehensive suite of tests, then what you really have is a much less valuable Continuous Build server. Preferably though, each developer should be running tests on their local machines as well during their development time and definitely before committing changes to a code repository.

integrated to the build and release processes

Triggering tests to be run needs to be scripted into your build and release . One of the gates to get through before making a release should always be a successful run of your tests. If your tests can only be run through a separate , like a shell script, forget it.

Integrating test runs into the build process and enabling the tests by default all help to elevate testing and the writing of test code to the same level as writing the product code itself.

first class development artifact

Raise testing up to be a tenet of your development process in your organization or . Make writing testing code as valued an exercise as writing product code. Then automate the evaluation of the testing effort, through tools that measure test coverage and other metrics. One of the QA gates your product should pass through before it is released is that the associated tests meet the quality metrics you require.

When a build on the CI server fails, make a big deal out of it and hold someone responsible. The old red-light/green-light (or red-lavalamp/green-lavalamp if you prefer) idiom actually has value and is more than just an inside joke for the in the office. The more public these kinds of failures are made the more committed the team will be to limiting or (better still) eliminating them.

Conclusion

Many of the benefits that can be reaped from Automated Testing can only be achieved if your tests meet a certain level of quality. This means measuring the quality of your tests and generating metrics about your tests just like you would for your product code. In this article I outlined what those quality characteristics are. In the next article I will detail the benefits you will get from a high quality Automated Testing strategy.