Test Automation Tools!

As a Quality Architect, I have been tasked with evaluating test automation tools for QA/QE and development more often than I can remember. There is always a fight between homegrown and commercial tools, i.e. a build vs. buy decision. I have always been an advocate of homegrown solutions, until recently. I have architected several sophisticated frameworks myself. In this blog, I uncover some of the very basic requirements of a test automation framework, which should help you with your evaluation or defining the requirements for your homegrown venture.

The complete requirement of a test automation framework can be captured in one line:

A tool that facilitates automating test scenarios and allows anyone to run them from anywhere and at anytime.

This means that automating tests should be easy and intuitive. Tests, once automated, should be able to run on any supported platform or operating system. And most importantly, anyone (QA, Development, Sustaining Team, or even customers, if required) should be able to run these automated tests in an unattended mode.

Appropriate logging and debugging mechanisms should be available to capture false negatives. The tool should provide a framework to test the core technologies that our SUT is built upon - SOA Web Services, .NET, EJBs, RMI, Web UI, Rich Clients, Command Line, SQL, Scripting, APIs, Raw SOAP, Proprietary XML and document formats, etc.

No tool can generate positive results if it does not take people and processes into account. Apart from the core test automation needs, a framework must also integrate with other existing tools in the ALM domain. For example, a test automation tool must integrate with test management system, which should integrate seamlessly with requirements management, defect tracking and other top-level dashboards. There is no one tool that can serve all our requirements and that is why it is very important to have open integration APIs available for customization. Continuous Integration and Agile testing is the new buzz these days. A framework must mesh well with cruise control, ant, maven and build repositories.

SCENARIO:

A test engineer or a developer automates a test and checks it into a version control system. Cruise control kicks of the nightly build and executes all the pre-deployment test cases. A provisioning system deploys the latest bits and kicks of the post-deployment automated tests. Test results are automatically pushed over to a central server, where they get mapped to the requirements. An email notification is generated with up-to-date report.

Next morning, the manager checks the email, clicks on the link, logs into the reporting system and gets the latest release readiness matrix with detailed drill-down test coverage and code coverage reports.

The company decides to ship automated tests with its product to its customers. Even in absence of the build workspace and central reporting server, customers are able to execute the automated tests and get the local report!

Above scenario captures majority of the requirements of a test automation framework. Some may think it is too extreme and for others some pieces of this scenario may not be applicable at all. But if you really think about it, this is the kind of infrastructure that is required to build high quality software applications. It is required for continuous integration and agile development & testing.

RECAP:

A test automation framework should (choose the ones applicable to you):

• be platform/OS independent
• provide detailed logging and debugging mechanisms
• support SUT technologies
• SOA Web Services, .NET, Raw SOAP
  
• J2EE, EJBs, RMI, POJO
  
• Command Line, Scripting
• Web UI
  
• Rich Client - Swing UI
• Databases, SQL
• Raw XML
• Proprietary document and transport
  
• be able to execute test cases in headless and batch mode
• zero coding requirement - but still available for advanced users
• version control friendly - no binary files to check-in
• provide APIs/mechanisms to integrate with other ALM tools
• integrate with development IDEs
  
• integrate with build workspace
  
• integrate with Continuous Integration tools
• integrate with Code coverage tools
• integrate with test management tools
  
• provide detailed reporting (text, xml, html, etc) with APIs to customize and integrate
  
• provide data driven capability - a must
  
• provide distributed application support
• be able to execute remote commands
  
• ease of use, good documentation, training and support available!
  

Is there such a framework available in the market? I have worked with over 40 different commercial and open source tools and have not found even a single one that delivers even 25% of these requirements. That is why most of the companies revert to home grown solutions. I was also one of the advocates of building a home grown solution, until recently!

BREAKTHROUGH:

Recently, I came across LISA from iTKO. To my surprise, the tool is very impressive - much better than any other in the industry. LISA seems to deliver over 80% of above requirements - as if the company read my mind and captured all the requirements!! There are minor quirks (like all others), but the tool is built on pure java and XML, runs everywhere, provide open APIs for expansion and integration into anything! Amazing data driven capability and provides mechanism to automate complicated end-to-end scenarios. It is a dream tool for developers and test engineers! It allows you to plugin your own java code and mash-up with other technologies. The developers don't have to maintain a separate workspace for test cases - the test cases can be kicked off from the same build.xml file. XML reports and custom report generator can be used to integrate test results into anything.

Missed time to market window! Really?

We know project management is all about the juggling the three balls of time, cost and quality. A project is successfull if it meets the functional and non-functional requirements within predetermined time, cost and quality constraints.

The traditional project management approach (and hence 99% of the tools) focus on completing the defined work within given time constraints and cost limits. However, the recent focus has been shifting more to the quality of the final output!!

Let's look at some examples:

• Google. Didn't google missed the time to market long before it released its search engine?
• Apple iPOD. Had it made a difference if iPOD was delayed another 6 months?
• Toyota Prius in 2000. Missed the TTM by three years! (Audi released its first hybrid in 1997 and Honda released its hybrid in 1999)

More and more companies are realising the fact that quality rocks!! If your product is high quality, it doesn't really matter if you are a year or two late to the market. Every product has its life, but if it is of high quality it tends to live longer - which changes the whole Net Present Value (NPV) calculation, in case you are using it to calculate the validity of your projects and releases.

Project requirements can be divided into functional and non-functional buckets. Functional requirements are the core (and supplementry) features of your product. Non-functional requirements are the systemic qualities, which encapsulates all "illities" - Availability, Scalability, Reliability, Flexibility, Extensibility, Interoperability, Compatibility, Testability, Understandability, Load and Performance, Stability, Resiliency, Manageability, Mantainability, Security, Supportability, Adaptability, Configurability and Usability! Note: Not all illities are applicable to all product offerings.

Your product may have over thousand functionalities, but just pick a handful of core ones (maybe 3 to 5) and all of the non-funtional requirements for your first release. A high quality product markets itself: word-of-mouth is the most effective marketing tool. Once a customer is hooked-in, slowly roll-out new features. That way you'll have the relationship going and you can get a continuous inflow of money - easy from SEC's perspective and no hassle of accounting manipulations either! That's what is driving software as a service (SaaS) market today.

SOA is the SaaS enabler and it is changing the way software is released. SOA brings business agility. However, our project management tools are still old-fashioned. Project managers are still focused on TTM and CTM concepts. They are still chasing deadlines and pennies. Quality awareness is forcing ALM companies to come up with more sophesticated tools that stitches the SOA fabric.

For innovation and quality, you are never late to the market!