The complete requirement of a test automation framework can be captured in one line:
A tool that facilitates automating test scenarios and allows anyone to run them from anywhere and at anytime.This means that automating tests should be easy and intuitive. Tests, once automated, should be able to run on any supported platform or operating system. And most importantly, anyone (QA, Development, Sustaining Team, or even customers, if required) should be able to run these automated tests in an unattended mode.
Appropriate logging and debugging mechanisms should be available to capture false negatives. The tool should provide a framework to test the core technologies that our SUT is built upon - SOA Web Services, .NET, EJBs, RMI, Web UI, Rich Clients, Command Line, SQL, Scripting, APIs, Raw SOAP, Proprietary XML and document formats, etc.
No tool can generate positive results if it does not take people and processes into account. Apart from the core test automation needs, a framework must also integrate with other existing tools in the ALM domain. For example, a test automation tool must integrate with test management system, which should integrate seamlessly with requirements management, defect tracking and other top-level dashboards. There is no one tool that can serve all our requirements and that is why it is very important to have open integration APIs available for customization. Continuous Integration and Agile testing is the new buzz these days. A framework must mesh well with cruise control, ant, maven and build repositories.
A test engineer or a developer automates a test and checks it into a version control system. Cruise control kicks of the nightly build and executes all the pre-deployment test cases. A provisioning system deploys the latest bits and kicks of the post-deployment automated tests. Test results are automatically pushed over to a central server, where they get mapped to the requirements. An email notification is generated with up-to-date report.Above scenario captures majority of the requirements of a test automation framework. Some may think it is too extreme and for others some pieces of this scenario may not be applicable at all. But if you really think about it, this is the kind of infrastructure that is required to build high quality software applications. It is required for continuous integration and agile development & testing.
Next morning, the manager checks the email, clicks on the link, logs into the reporting system and gets the latest release readiness matrix with detailed drill-down test coverage and code coverage reports.
The company decides to ship automated tests with its product to its customers. Even in absence of the build workspace and central reporting server, customers are able to execute the automated tests and get the local report!
A test automation framework should (choose the ones applicable to you):
- be platform/OS independent
- provide detailed logging and debugging mechanisms
- support SUT technologies
- SOA Web Services, .NET, Raw SOAP
- J2EE, EJBs, RMI, POJO
- Command Line, Scripting
- Web UI
- Rich Client - Swing UI
- Databases, SQL
- Raw XML
- Proprietary document and transport
- be able to execute test cases in headless and batch mode
- zero coding requirement - but still available for advanced users
- version control friendly - no binary files to check-in
- provide APIs/mechanisms to integrate with other ALM tools
- integrate with development IDEs
- integrate with build workspace
- integrate with Continuous Integration tools
- integrate with Code coverage tools
- integrate with test management tools
- provide detailed reporting (text, xml, html, etc) with APIs to customize and integrate
- provide data driven capability - a must
- provide distributed application support
- be able to execute remote commands
- ease of use, good documentation, training and support available!
Recently, I came across LISA from iTKO. To my surprise, the tool is very impressive - much better than any other in the industry. LISA seems to deliver over 80% of above requirements - as if the company read my mind and captured all the requirements!! There are minor quirks (like all others), but the tool is built on pure java and XML, runs everywhere, provide open APIs for expansion and integration into anything! Amazing data driven capability and provides mechanism to automate complicated end-to-end scenarios. It is a dream tool for developers and test engineers! It allows you to plugin your own java code and mash-up with other technologies. The developers don't have to maintain a separate workspace for test cases - the test cases can be kicked off from the same build.xml file. XML reports and custom report generator can be used to integrate test results into anything.