Telescope Time Allocation Tool Version 0.2 Internal Review

Review Purpose

The TTA Tools System Description introduces multiple new concepts and requirements to the NRAO Proposal and Review process.  Version 0.2 is the implementation of proposal submission and review that permits users to experience them in practice and evaluation efficacy. In this review, the google forms linked below will ask for feedback in the following categories:

• Predictability. Is the user experience predictable?
• Process. Is the process reasonable?
• Usability. Too many clicks, tool tips, missing info?
• Bugs. Report any issues.
• Documentation.  Are there any gaps in the documentation?

Feedback form for Individual Science Reviews (available immediately)

Feedback form for Consensus Reviews (available at the start of the Consensus meeting)

Documents:

• System Description
• Review Process Users' Guide

Logistics

The different review stages are listed below.  Review stages 1–4 will primarily be done offline (i.e., we do not need to organize a Zoom meeting).   For select individuals, however, we want to have our software front-end developer watch how the user interacts with the software.  These cases are listed below.  During the Consensus review we will all meet via Zoom.

1.  Release Proposals for Review
   
   1. TTA member configures SRP/vets proposal science category.
   2. TTA member sends emails.
2.  Conflict Declaration
   1. SRP members independently declare conflicts of interest by 21 Sept
   2. Zoom Meeting with Emily to observe her workflow (1 hr)
3.  SRP Chair Assignment
   1. SRP chair assigns reviewers to proposals.
   2. Zoom Meeting: Loreto (1 hr)
4. Individual Science Review
   1. SRP members independently enter and complete Individual Science Reviews by 28 Sept.
   2. Zoom Meeting with Emily to observe her workflow (1 hr)
5. Consensus Review
   1.  Panel meets to perform Consensus Reviews.
   2. Zoom Meeting: All (2 hr)

Testing Plan and Roles:

We plan to simulate the review process for one Science Review Panel (SRP).  To do this a TTA Member will generate about 25 mock proposals to be considered for review by one SRP.  To be realistic the SRP will be comprised on one chair and five members.  Here are the roles: 

• TTA member (admin): Dana
• SRP chair: Loreto
• SRP members: Allie, Jeff, Amy, Robert, Emily

Proposals prepared for the review:

The proposals included in this review have a Proposal ID of Sem_24B.

The proposals creation portion of this application is not fully complete; as such, the content of the proposals has been significantly limited for this review.

How to review simulated proposals:

Though you cannot evaluate the proposals for the actual scientific merit, please provide a range of scores to the proposals such that they will be normalized realistically for the Consensus Review.

Please also considering fabricating a brief comment for each proposal so the Consensus Reviews appear mildly realistic. Here are a few examples you could replicate as desired into the review comments. Even if the comment is a single sentence, it will make the Consensus UI appear more realistic than random letters. It is fine if there is a lot of duplication across proposals and reviewers.

• This proposal is well justified.
• This proposal could have justified the use of resources more.
• This proposal will not address the science goals it proposes.
• This proposal can provide valuable results for the community.
• This proposal did not account for the available archival data and it is not clear if it would provide novel results.