Review Purpose
The TTA Tools System Description introduces multiple new concepts and requirements to the NRAO Proposal and Review process. Version 0.2 is the implementation of proposal submission and review that permits users to experience them in practice and evaluation efficacy. The TTA Tools are available at https://tta.nrao.edu/
In this review, the google forms linked below will ask for feedback in the following categories:
- Predictability. Is the user experience predictable?
- Process. Is the process reasonable?
- Usability. Too many clicks, tool tips, missing info?
- Bugs. Report any issues.
- Documentation. Are there any gaps in the documentation?
Feedback form for Conflicts and Individual Science Reviews (available immediately)
Feedback form for Consensus Reviews (available at the start of the Consensus meeting)
Logistics
We have divided the testing into five stages (see below). Stages 1–2 will primarily be done offline (i.e., we do not need to organize a Zoom meeting). For stages 3-4, however, we want to have our software front-end developer watch how the user interacts with the software. These cases are listed under Scheduled Zoom Meetings. During the Consensus review (stage 5) we will all meet via Zoom.
Test Schedule:
| Stage | Action | Who | When |
|---|---|---|---|
(1) Release Proposals for Review | TTA Member configures SRP/vets science category. Sends email. | TTA Member | 11/06/23 |
| (2) Conflict Declaration | SRP member declares conflict of interest. | SRP Members | 11/06/23 to 11/08/23 |
| (3) SRP Chair Assignment | SRP chair assigns reviewers to proposals. | SRP Chair | 11/08/23 |
| (4) Individual Science Review | SRP members perform Individual Science Reviews. | SRP Members | 11/xx/23 to 11/17/23 |
| (5) Consensus Review | SRP meets to perform Consensus Review. | All | 11/17/23 |
Scheduled Zoom Meetings:
| Who | What | When (EST) | Zoom Contact |
|---|---|---|---|
| Craig | Conflict/SRP Assignment/ISR | 11/08/23 1 - 2 EST (11 - 12 MST) | |
| Tony | Individual Science Review | TBD | |
| Trey | Individual Science Review | TBD | |
| Cherry | Individual Science Review | TBD | |
| Joan | Individual Science Review | TBD | |
| All | Consensus Review | 11/17/23 4-6pm EST (2 - 4 MST) |
Testing Plan and Roles:
We plan to simulate the review process for one Science Review Panel (SRP). To do this a TTA Member will generate about 25 mock proposals to be considered for review by one SRP. To be realistic the SRP will be comprised on one chair and five members. Here are the roles:
- TTA member (admin): Dana
- SRP chair: Craig Heinke
- SRP members: Tony Remijan, Trey Wenger, Cherry Ng, Joan Wrobel, Allie Costa
Proposals prepared for the review:
The proposals included in this review have a Proposal ID of Sem-28A.
The proposals creation portion of this application is not fully complete; as such, the content of the proposals has been significantly limited for this review.
How to review simulated proposals:
Though you cannot evaluate the proposals for the actual scientific merit, please provide a range of scores to the proposals such that they will be normalized realistically for the Consensus Review.
Please also considering fabricating a brief comment for each proposal so the Consensus Reviews appear mildly realistic. Here are a few examples you could replicate as desired into the review comments. Even if the comment is a single sentence, it will make the Consensus UI appear more realistic than random letters. It is fine if there is a lot of duplication across proposals and reviewers.
- This proposal is well justified.
- This proposal could have justified the use of resources more.
- This proposal will not address the science goals it proposes.
- This proposal can provide valuable results for the community.
- This proposal did not account for the available archival data and it is not clear if it would provide novel results.