To both assess the efficacy of the SRDP capabilities, the usage patterns, and drive decision making based on informed data we will rely on metrics throughout the project.
Unless otherwise noted collected metrics should be available on weekly, monthly, semester and cumulative timescales.
Status Progression
DISCUSS Ongoing discussion
PROPOSED Ready to become a requirement, approvals pending
RQT: WAVE 1 Now an official requirement
COMPLETE Implemented and in use
Archive Content Metrics
| Description | Purpose | Traceability | Status |
|---|---|---|---|
| Report the number and volume of calibration results ingested to the archive | Track usage of space and coverage of SRDP product in the archive. | TUC02.5.1 | |
Archive Usage Metrics
Unless otherwise noted all requirements should be reported both as the number of primary products and the total data volume.
| Description | Purpose | Traceability | Status |
|---|---|---|---|
| Number of registered users that download each type of product. | This metric as different from the total download volume allows us to determine how widespread the SRDP user community is. | TUC12.5.1 | RQT: WAVE 1 |
| Total number of product downloads of each type (including anonymous users). This should be both a total number and a number per collection. | Total volume of data and products going out is one of the key performance indicators for the project. | TUC12.5.1 | RQT: WAVE 1 |
Number of products downloaded by a member of the observing team, both during and after the product proprietary period | Understand data reuse and other usage patterns. | TUC12.5.1 | RQT: WAVE 1 |
| Method of Data Delivery | Understand how users prefer to access data, deprecate unused methods. | TUC12.5.1 | RQT: WAVE 1 |
Capability Usage Metrics
| Description | Purpose | Traceability | Status |
|---|---|---|---|
| Report the number of times the standard calibration pipeline was triggered. | Determine overall usage of the pipeline. Normalize error reporting and other pipeline related reports. | TUC02.5.1 | RQT: WAVE 1 |
| Number and type of customization choices made by the user as part of the download from the archive. | Understand usage patterns, gauge interest for various options Understand resource implications of capabilities | TUC12.5.1 | RQT: WAVE 1 |
Workflow Metrics
| Description | Purpose | Traceability | Status |
|---|---|---|---|
| Report the number of times (and reason) that the standard products required intervention before passing QA. | Gauge efficienciency and maturity of pipeline. Estimate effort required for QA in future performance periods. | TUC02.5.1 | RQT: WAVE 1 |
| The number of times that a pipeline failed to produce a valid product and needed to switched to manual. | Understand the success rate of the pipeline to mediate between additional modes and increased robustness. | New | DISCUSS |
Product Quality Metrics
| Description | Purpose | Traceability | Status |
|---|---|---|---|
| Report the number of times that calibrations are reported by the PI as deficient. | Gauge efficienciency and maturity of pipeline. Estimate effort required for QA in future performance periods. Monitor the rate of bad data released by the SRDP project. | TUC02.5.1 | RQT: WAVE 1 |
Deployment Metrics
| Description | Purpose | Traceability | Status |
|---|---|---|---|