Thursday, December 28, 2006

Test Automation Tools!

As a Quality Architect, I have been tasked with evaluating test automation tools for QA/QE and development more often than I can remember. There is always a fight between homegrown and commercial tools, i.e. a build vs. buy decision. I have always been an advocate of homegrown solutions, until recently. I have architected several sophisticated frameworks myself. In this blog, I uncover some of the very basic requirements of a test automation framework, which should help you with your evaluation or defining the requirements for your homegrown venture.

The complete requirement of a test automation framework can be captured in one line:
A tool that facilitates automating test scenarios and allows anyone to run them from anywhere and at anytime.
This means that automating tests should be easy and intuitive. Tests, once automated, should be able to run on any supported platform or operating system. And most importantly, anyone (QA, Development, Sustaining Team, or even customers, if required) should be able to run these automated tests in an unattended mode.

Appropriate logging and debugging mechanisms should be available to capture false negatives. The tool should provide a framework to test the core technologies that our SUT is built upon - SOA Web Services, .NET, EJBs, RMI, Web UI, Rich Clients, Command Line, SQL, Scripting, APIs, Raw SOAP, Proprietary XML and document formats, etc.

No tool can generate positive results if it does not take people and processes into account. Apart from the core test automation needs, a framework must also integrate with other existing tools in the ALM domain. For example, a test automation tool must integrate with test management system, which should integrate seamlessly with requirements management, defect tracking and other top-level dashboards. There is no one tool that can serve all our requirements and that is why it is very important to have open integration APIs available for customization. Continuous Integration and Agile testing is the new buzz these days. A framework must mesh well with cruise control, ant, maven and build repositories.

A test engineer or a developer automates a test and checks it into a version control system. Cruise control kicks of the nightly build and executes all the pre-deployment test cases. A provisioning system deploys the latest bits and kicks of the post-deployment automated tests. Test results are automatically pushed over to a central server, where they get mapped to the requirements. An email notification is generated with up-to-date report.

Next morning, the manager checks the email, clicks on the link, logs into the reporting system and gets the latest release readiness matrix with detailed drill-down test coverage and code coverage reports.

The company decides to ship automated tests with its product to its customers. Even in absence of the build workspace and central reporting server, customers are able to execute the automated tests and get the local report!
Above scenario captures majority of the requirements of a test automation framework. Some may think it is too extreme and for others some pieces of this scenario may not be applicable at all. But if you really think about it, this is the kind of infrastructure that is required to build high quality software applications. It is required for continuous integration and agile development & testing.


A test automation framework should (choose the ones applicable to you):
  • be platform/OS independent
  • provide detailed logging and debugging mechanisms
  • support SUT technologies
    • SOA Web Services, .NET, Raw SOAP
    • J2EE, EJBs, RMI, POJO
    • Command Line, Scripting
    • Web UI
    • Rich Client - Swing UI
    • Databases, SQL
    • Raw XML
    • Proprietary document and transport
  • be able to execute test cases in headless and batch mode
  • zero coding requirement - but still available for advanced users
  • version control friendly - no binary files to check-in
  • provide APIs/mechanisms to integrate with other ALM tools
    • integrate with development IDEs
    • integrate with build workspace
    • integrate with Continuous Integration tools
    • integrate with Code coverage tools
    • integrate with test management tools
  • provide detailed reporting (text, xml, html, etc) with APIs to customize and integrate
  • provide data driven capability - a must
  • provide distributed application support
  • be able to execute remote commands
  • ease of use, good documentation, training and support available!
Is there such a framework available in the market? I have worked with over 40 different commercial and open source tools and have not found even a single one that delivers even 25% of these requirements. That is why most of the companies revert to home grown solutions. I was also one of the advocates of building a home grown solution, until recently!


Recently, I came across LISA from iTKO. To my surprise, the tool is very impressive - much better than any other in the industry. LISA seems to deliver over 80% of above requirements - as if the company read my mind and captured all the requirements!! There are minor quirks (like all others), but the tool is built on pure java and XML, runs everywhere, provide open APIs for expansion and integration into anything! Amazing data driven capability and provides mechanism to automate complicated end-to-end scenarios. It is a dream tool for developers and test engineers! It allows you to plugin your own java code and mash-up with other technologies. The developers don't have to maintain a separate workspace for test cases - the test cases can be kicked off from the same build.xml file. XML reports and custom report generator can be used to integrate test results into anything.