The SRAO project maintains a suite of Validation Tests in the Jira system that are used to validate the system prior to release. The state diagram for validation tests is shown at right. This page describes the intended meaning of each state. Because the SRAO project uses a rolling wave project management process many requirements are only partially fulfilled in any given wave, and future waves may require previously validated requirements to be revalidated. Thus the validation state at any time is a snapshot of the state of the system.
Validation tickets are linked to the requirements they validate in the Cameo Model and are updated through an import step. The relationship between tests and requirements may be many-to-many, a requirement is validated if and only if all of the validation tests associated with it are in the passed or closed states.
Test State Descriptions
- TO DO: This is the initial state for each test, indicating that the test has been defined, but is not yet ready to be executed, usually because the software the test is designed to validate has not been made available yet.
- READY TO TEST: Designates that all pre-requisites for the test have been met and testing may commence. Transition to this state may be the result of a formal Test Readiness Review or a less formal process.
- TEST IN PROGRESS: Indicates that testing is underway.
- FAILED: A test in the failed state indicates that the software is not validated for the functionality tested.
- PASSED: The software under test fully meets the requirements of the test.
- PARTIAL: Some subtests (either formal subtests in the Jira sense or informal subtests within the test case) are in the passed or closed state and others are not. The functionality tested is "partially"
validated. - CLOSED: Because subsequent releases of the software may introduce defects to previously validated functionality, until a regression test is created that ensures a defect is not introduce tests remain in the passed state. Fully validated tests, with regressions are placed in the closed state and do not need to be manually executed in the future.
- CANCELLED: The test is deemed to no longer be useful or required and will not be executed in the future. For example if a scope change at the project level removes requirements from the project, the tests validating that functionality should be cancelled.
Parent States
Often we group sets of sub-tests together for tracking and organization reasons. The parent states are derived from the states of the sub-tests as follows:
- CANCELLED: All subtests are in the CANCELLED state. Otherwise CANCELLED subtests are not considered in determining the state of the parent.
- TO DO: All subtests are in the TO DO state.
- READY TO TEST: All subtests are in the READY TO TEST or TO DO state and at least one subtest is in the READT TO TEST state.
- TEST IN PROGRESS: All subtests are in the TO DO, READY TO TEST, or TEST IN PROGRESS state and at least one subtest is in the TEST IN PROGRESS state.
- PASSED: All subtests are in the PASSED or CLOSED states.
- CLOSED: All subtests are in the CLOSED state.
- FAILED: At least one subtest is in a failed state and no subtests are in the PASSED, CLOSED, PARTIAL, or TEST IN PROGRESS states
- PARTIAL: All other cases.