Review Purpose
The TTA Tools System Description introduces multiple new concepts and requirements to the NRAO Proposal and Review process. Version 0.2 is the implementation of proposal submission and review that permits users to experience them in practice and evaluation efficacy. Please provide the following feedback broken down by Review Stages 2–4 (see below)The TTA Tools are available on the internal network at https://tta.nrao.edu/
In this review, the google forms linked below will ask for feedback in the following categories:
- Predictability. Is the user experience predictable?
- Process. Is the process reasonable?
- Usability. Too many clicks, tool tips, missing info?
- Bugs. Report any issues.
- Documentation. Are there any gaps in the documentation?
Feedback form for Individual Science Reviews (available immediately)
Feedback form for Consensus Reviews (available at the start of the Consensus meeting)
Logistics
The different review stages are listed below. Review stages We have divided the testing into five stages (see below). Stages 1–4 will primarily be done offline (i.e., we do not need to organize a Zoom meeting). For select individuals, however, we want to have our software front-end developer watch how the user interacts with the software. These cases are listed belowunder Scheduled Zoom Meetings. During the Consensus review (stage 5) we will all meet via Zoom.
Test Schedule:
| Stage | Action | Who |
|---|
| When |
|---|
(1) Release Proposals for |
Review | TTA Member configures SRP/vets |
science category. Sends email. | TTA |
Member | 9/18 |
| /23 | |
| (2) Conflict Declaration | SRP member declares conflict |
of interest. | SRP |
Members | 9/18/23 - 9/21/23 |
| (3) |
Members
| SRP Chair Assignment | SRP chair assigns reviewers |
to proposals. | SRP Chair |
Chair
| 9/22/23 | |
| (4) Individual Science Review | SRP members perform |
Individual Science Reviews. | SRP |
Members | 9/23/23 - 9/28/23 |
| (5) |
Members
| Consensus Review | SRP meets to perform |
Consensus Review. | All |
- TTA member configures SRP/vets proposal science category.
- TTA member sends emails.
- SRP members declare conflicts of interest.
- Zoom Meeting: Emily (1 hr)
- SRP chair assigns reviewers to proposals.
- Zoom Meeting: Loreto (1 hr)
- SRP members perform Individual Science Reviews.
- Zoom Meeting: Emily (1 hr)
| 9/29/23 |
Scheduled Zoom Meetings:
| Who | What | When | Zoom Contact |
|---|---|---|---|
| Emily | Conflict Declaration | 9/21/23 1-2pm EDT (11-12pm MDT) | |
| Loreto | SRP Assignment | 9/22/23 10-11am EDT (8-9pm MDT) | |
| Emily | Individual Science Review | 9/26/23 1 - 2 EDT (11 - 12 MDT) | |
| All | Consensus Review | 9/29/23 3-5pm EDT (1-3pm MDT) | https://go.nrao.edu/nraozoom08 |
Testing Plan and Roles:
We plan to simulate the review process for one Science Review Panel (SRP). To do this a TTA Member will generate about 25 mock proposals to be considered for review by one SRP. To be realistic the SRP will be comprised on one chair and five members. Here are the roles:
- TTA member (admin): Dana
- SRP chair: Loreto
- SRP members: Allie, Jeff, Amy, Robert, Emily
Proposals prepared for the review:
The proposals included in this review have a Proposal ID of Sem_26A.
The proposals creation portion of this application is not fully complete; as such, the content of the proposals has been significantly limited for this review.
How to review simulated proposals:
Though you cannot evaluate the proposals for the actual scientific merit, please provide a range of scores to the proposals such that they will be normalized realistically for the Consensus Review.
Please also considering fabricating a brief comment for each proposal so the Consensus Reviews appear mildly realistic. Here are a few examples you could replicate as desired into the review comments. Even if the comment is a single sentence, it will make the Consensus UI appear more realistic than random letters. It is fine if there is a lot of duplication across proposals and reviewers.
- This proposal is well justified.
- This proposal could have justified the use of resources more.
- This proposal will not address the science goals it proposes.
- This proposal can provide valuable results for the community.
- This proposal did not account for the available archival data and it is not clear if it would provide novel results.