A Test Automation Strategy for Continuous Testing
Test automation within a continuous testing strategy must be designed to run efficiently while providing reliable, consistent, reproducible results. The automation strategy must be well-developed; it must include plans for choosing the test cases for automation, developing an automation framework that enables easy test maintenance and creating a schedule of which automated tests to be run at what points along the delivery pipeline. Developing and implementing a test automation strategy for continuous testing is challenging; yet models such as the test automation pyramid and the core test practice of test optimization can help. Let’s explore how the test pyramid and test optimization enables the development of an effective test automation strategy for continuous testing.
The Tools of Test Optimization
The test pyramid is a “shift left” pattern whereby most of the automated test cases are concentrated in the unit, API, component and integration layers with fewer automated test cases used to test the GUI layer. Implementing this pattern has several advantages. Bugs are caught earlier in the process when it is less costly to fix them. As automation for unit and integration testing is quicker and easier to develop and maintain; the cost of automation is reduced. Return on investment is increased by reducing cost while increasing velocity and quality.
Test optimization creates the foundation for an effective automation strategy for continuous testing. Test optimization refers to the practice of selecting the right tests that provide the test coverage you need with the fewest number of test cases. The practice has many applications from selecting tests for a full regression suite to choosing tests for targeted testing and regression of new features. It is a dynamic, on-going process, especially when it is applied as part of a continuous testing framework. Ideally, test optimization should take place prior to automation and it should continue throughout the continuous testing process.
Optimizing Test Scope
When we think about test optimization, we often concentrate on selecting test cases for automation, but it is important to begin at a higher level; optimize the test scope. To optimize the test scope:
- Understand all integrations involved in critical user workflows including the technologies employed in those applications (web, mobile, message/API-layer etc.).
- Implement service virtualization for integrations and components that aren’t readily available to run end-to-end test continuously.
- Use the most realistic data possible for each test, include synthetic data generation where needed.
- Include exploratory testing to find user-experience and other bugs not easily found by automated tests.
Optimizing Test Cases
With a clear understanding of the test scope, optimizing the test cases is the next step in the process. It includes analyzing test cases not only for their quality and level of detail but also using the following criteria:
- Tests should be focused on the functionality they are validating.
- Test should be as simple as possible; the more complex the test, the more validation is required for the test code itself.
- Even though service virtualization can be used to fill in for missing components in integration testing, test should as independent as possible.
Optimizing Test Suites
The next step in test optimization is to build test suites. Test suites, whether they are for regression or for specific feature testing should be optimized. Test suites should be designed to provide the maximum amount of coverage with the fewest number of test cases to increase both quality and velocity. Here are some guidelines to consider when optimizing test suites:
- Include test cases that validate high risk areas of code.
- Include test cases that validate customer-centric features.
- Include test cases that validate critical business workflows
- Use Functional Automation tools that create multi-purpose scripts:
Cross-Platform Testing
- Scripts should run on Desktop, Web, Mobile & Server
Cross-Browser Testing
- Scripts should run on multiple browsers including Chrome, Edge, Safari and any others required
Performance Testing
- Scripts should collect performance statistics including page loads etc.
Optimizing Regression Test Suites
Regression Test Selection (RTS) is one of the best approaches to creating an optimized regression suite. There are several methods of RTS; some involve complicated rankings based on factors including complexity of code and business criticality. In general, RTS involves technical and business risk analysis to develop a test suite that is likely to find the highest number of critical bugs with the fewest number of test cases. It is important to note that the focus is on finding the critical bugs that will negatively impact the business customer as opposed to finding more but less consequential bugs. Artificial Intelligence tools are becoming more effective at predicting areas of code that are likely to have critical bugs and are now being used for test optimization.
Maintaining your Optimized Regression Test Suites
All approaches emphasize that RTS is a dynamic process; test suites must constantly be re-evaluated, as new features are developed and released, and this is particularly challenging in the context of continuous testing and continuous delivery. One of the best ways to continually optimize test suites within continuous delivery frameworks is to maintain traceability between the test suite and the features currently under development. This takes collaboration and communication among the developers and testers to ensure not only a thorough understanding of the features under development, but also, what other areas of the code are potentially impacted. Feature checklists are helpful in maintaining traceability and test coverage and can be managed manually or through a variety of collaboration tools.
To maintain the effectiveness of the regression suite, it is important for testers to understand the strategic direction of the application under test. Often, features will be deprecated from the code or no longer supported due to strategic business decisions. A communication process is needed so that the tests related to these features are removed from the regression suite in a timely manner.
A critical part of maintaining an optimized test suite is evaluating its effectiveness. Defect removal efficiency and defect leakage metrics can be useful, however; they must be focused on the severity of the defects as the purpose of optimized test suites is to find the most critical defects not the most defects. Metrics of the types of defects found by the regression suite are often helpful for further optimization.
Continuous Test Automation Strategy is a Must for Continuous Delivery
For successful continuous testing within a continuous delivery pipeline, a test automation strategy specifically developed for continuous testing is a must. The strategy must include not only the development of a flexible automation framework, but must include continuously optimizing the test scope, test cases and the regression test suite.
