Audit Testing: A Consideration of Utility and Meaning

As the IT industry evolves, it introduces new disciplines to meet new needs. As these disciplines mature, the methods which they implement expand. Audit testing is one of the methods the testing discipline can use to examine a testing process and produce usable feedback with less resource expenditure than a more exhaustive testing effort may require.

What is “Audit Testing”?

The most common example of an audit test is the financial audit. In it, you examine the financial records, some individual transactions, and the process used to obtain and record them. Similarly, when we audit the testing process, we look at results of a test cycle, the process by which the results were obtained, and the tools and components a test has used to obtain these results.

Who performs an audit?

Most commonly an audit is executed by an external resource to verify validity and adherence to standards. An external auditor is usually seen as a more credible source because the only interest they should have is a thorough and accurate assessment. An internal audit makes use of resources already at hand within a company. The potential benefits include knowledge of the tools and methods in use and a more complete understanding of the goals and direction of the project. An internal audit may not be viewed with the same amount of credit an external audit would carry due to potential for conflicting interests such as company reputation and the political situation in the company.

Why perform an audit?

Audit testing does not exhaustively test a product to uncover every potential issue and defect and so does not incur the cost in time and personnel that such a test would suggest. Instead, audit testing aims to examine a testing process already in place for coverage and accuracy of the process.

An audit provides several benefits to a project. Common reasons for initiating an audit include:

• Lack of experience with a kind of project.
• Questionable history with a type of project.
• New business partnership between companies.
• Peace of mind in regards to the testing team.
• Finding problems caused by the complexity of the project.
• Verifiable controls for quality assurance.
• Improvement of internal resources.
• Complex high level standards associated when working with organizations such as government agencies, medical devices, military projects and financial institutions.

Consider also the potential impact of an insufficient quality assurance process. Even if the product is improved after release, the company may receive a negative impact to their reputation from their target audience. A product that had a strong potential for return on investment may face complete rejection from the audience due to initial experience, especially if it has a competing product or service on the market.

When should an audit test be implemented?

Audit testing is most commonly implemented towards the end of, or just after a testing cycle. Audit testing can, and in many cases should, be implemented during any or all phases of a cycle: before, during, and after.

Reasons for implementation during each phase:

1. Before the testing begins.

Prior to the start of testing there are no results to examine, and as such many companies would consider audit testing as non-applicable and wasteful of resources. In this stage the elements to be examined focus on the tools that are intended for use and the processes adhered to. When an audit is to be conducted prior to the start of a test cycle, the following process is likely:

1. Familiarization with the goals of the project, standards the project must adhere to, and all relevant documentation.
2. Examination of the testing processes and tools the processes will utilize.
3. Review the communication protocols and test management plan.
4. Interview relevant personnel.
5. If practical, run simulated test case.
6. Submit the report.

Considerations for pre-cycle audit testing include:

• Especially helpful with little testing experience or new product directions.
• Does not interfere with the testing
• Helps prevent excessive waste of personnel/time resources on faulty methods/tools
• Familiarization for future audits
• Does not have results of the cycle to confirm.

2. During the testing

                Arguably the most impactful time to run an audit is during the actual test cycle. It is here that the auditor has results they can confirm by asking the testers to show how they came to them. This allows the auditor to evaluate the tools and the process, as they are utilized, for the most comprehensive feedback potential. When an audit is to be conducted while a test-cycle is active, the following process is likely:

1. Familiarization with the goals of the project, standards the project must adhere to, and all relevant documentation.
2. Examination of the testing processes and tools the processes are utilizing.
3. Review the communication protocols and test management plan.
4. Interview relevant personnel.
5. Examine a representative sample of test case results and the processes used to obtain them.
6. Evaluate communication and test management in action.
7. Examine processes for potential defect evaluation and correction where applicable.
8. Submit report.

Considerations for during-cycle audit testing include:

• Results exist for comparison and evaluation
• Potential for positive impact on the test cycle in progress
• Simplifies scheduling with all relevant members present during times of testing.
• Can see conflicts between written plan/management and execution in action.
• Adds time and expense to a project

3. After the audit testing.

The time when the need of an audit is most noticeable is often after the completion of a test cycle. It is here that a mismatch between expectations and delivery is most visible and the utility of an audit is most easily justified. When an audit is to be conducted post-test cycle, the following process is likely:

1. Familiarization with the goals of the project, standards the project must adhere to, and all relevant documentation.
2. Examination of the testing processes and tools that were used for the test cycle.
3. Review the communication protocols and management plans that were used.
4. Interview relevant personnel.
5. Examine a representative sample of test case results and the processes that were utilized to get them.
6. Examine how many defects were found and corrected. Also, examine how many defects that were found remain.
7. Examine how many defects were not found by the test cycle that have been discovered post-test cycle and their severity.
8. Submit report.

Considerations for after-cycle auditing include:

• All results for comparison and consideration are present.
• Mismatch between expectations and results are most visible and defined.
• Does not interfere with the timeframe of the test cycle.
• Can be used to confirm a positive result/expectation parameter.
• Has no impact on the results of the completed test cycle.

4. Combination of phases.

While an audit of each separate phase has the potential for positive impact on a project, a combination of two or even all three phases can provide the most complete coverage and the greatest potential for success. The resources consumed in a combination of phases may not be feasible to every project, however, and when planning a project the goal of maximizing return and minimizing required resource should not be dismissed. An audit that takes place over more than one phase does not need to repeat every step in the process for each stage unless a major change has taken place. A table listing the previously mentioned steps, what phases they take place in, and if they need to be repeated without a major change can be seen below. Stages the steps apply to are marked with an X. Need for repeated steps when an audit takes place over several stages are marked Y for yes or N for no.

Considerations for combination of phase auditing include:

• Projected timeframe of the test cycle.
• Allocation of resources.
• Availability of personnel.
• Targeted impact of the audit.
• Current development of project in question.
• Potential for diminishing returns on each auditing cycle

How will the audit communicate the results?

Once an audit has transpired, the auditing entity needs to communicate its findings in a meaningful manner to the appropriate people. This is often done as a series of written statements declaring intent, entity responsible for initiating the audit, reference to material covered, and result of the audit. When appropriate, a disclaimer is included depending on the circumstances of the audit. The findings of an audit are not intended as a definitive analysis of the subject being audited. It is intended to sample the subject matter, and to present the trends found therein, either through a statistical presentation, or as a more informal report. For instance, if an audit were performed on test cycle A, the report may include a statement such as: “We find test cycle A, in accord with current testing standards, likely to acceptably find all essential bugs and the majority of minor defects prior to the release of this project in the given time frame required by management”. Other potential reports may convey:

• Statistical chance of a result.
• Likely failure of a project to meet expectations.
• Requirements for successful audit are not met.
• Inconsistency within a project in regards to expectations or stated process.
• Suggested improvements.

A quick reference of positive and negative considerations for an audit test process:

Summary

Audit testing is a progeny of the expanding IT industry, necessitated by the expansion of its toolbox and scope. It is not intended to completely replace a full test cycle, instead aiming to supplement and confirm the delivery of the testing process. The impact of audit testing is dependent on when it is implemented, and on a thorough follow-up. For maximum return, an external auditor should be used with access to the tools, processes, results, personnel, documentation, standards, and expectations of the project. The results of an audit deal in likelihood and statistics and do not convey 100% coverage of the product.