Customer focused test automation

We often limit our test automation projects to internal use only. This restricted use of test automation has contributed the typical sub-standard quality of automated test code. If you think production code is poorly documented you haven’t reviewed very much automated test code. But, since the test code is for internal use only we really don’t need to put as much effort into it as we do production code. Or do we?

Unfortunately, the restricted view of test automation as an internal only tool in the testing process has perhaps limited its potential and its overall value. But, as testing matures into a profession testers are going to have a greater impact not only during the product development lifecycle, but testers will also extend their scope of influence directly to the customers. You may ask how or why test automation be used by customers? Read the following scenarios, and think about the possibilities!

A company contracts for a customized software solution based on a specific set of requirements. At the completion of the project the product is handed off to the company who then conducts several suites of tests such as user acceptance tests and integration tests to test compatibility with the existing environment. Additional ‘acceptance’ testing by the company costs them time and uses their valuable IT resources to re-verify the product’s functional capabilities and other quality attributes. Now imagine the software company also provides well-documented automated test suites along with the product that demonstrates the product does what it is supposed to do, and does it in the customer’s environment. The customers ‘acceptance’ testing time is greatly decreased, their overall cost of adoption is decreased, and (just possibly) customer satisfaction is increased.

Another company purchases several Microsoft products such as Windows Server, Exchange Server, and SQL. But, it is often not as simple as installing these platforms and setting the preferred configuration. So, the company either staffs an IT department that rivals an IT team assembled by a Fortune 500 company, or hires a company similar to Avanade to put the pieces together and make them work with the desired configuration settings. Of course, either solution is going to cost the company a lot of money, time, and potentially use internal resources to get things setup correctly. Now suppose Microsoft creates an enterprise Software Test Kit (STK) that contains thousands of automated tests. The IT team identifies the specific configuration settings for its platforms, identifies applications in the corporate environment, and selects other requirements for adoption. Based on the IT department’s criteria the STK will automatically select a suite of tests the company’s IT department can execute to verify their platform configuration and other acceptance tests. Automated test suites in the STK can even be used for non-functional testing such as performance and stress testing by the company based on their configuration settings.

Are these potential uses of test automation by customers? I believe these are only some of the possibilities that exist. However, to realize this potential our test automation must evolve. The quality of our automated tests must rival that of our production code. In fact, test code must be better than production code because there is no room for errors or false positives. If customers use our automated tests to reduce their overall costs of adoption the tests must be bullet-proof or customer confidence decreases and customer satisfaction plummets.

So, what must change in order to increase the overall significance and potential usefulness of test automation beyond its current value? To begin with, automated test cases must be considered a product with a meaningful shelf-life instead of disposable artifact or something we simply pass on to the maintenance or sustained engineering team. The automated tests must be well-planned and well-documented (preferably with XML). The testers who develop the tests must understand the product and be able to develop the automated tests with a modern programming language. (Sorry, but scripting languages, keyword or data-driven approaches, and record/playback are simply too limited in their ability to scope to test automation of this magnitude.) Testers will need to use static analysis tools and perform code reviews to verify the quality of the test code. We will also need an framework that can select and run the appropriate set of tests to satisfy the customer’s needs, but is extensible enough for the customer to easily create and import additional automated tests if necessary.

Perhaps this is all just wishful thinking, or perhaps it is a vision of things to come in the professional world of software testing.

http://blogs.msdn.com/b/imtesty/archive/2006/05/19/601692.aspx