Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Review Purpose

The TTA Tools System Description introduces multiple new concepts and requirements to the NRAO Proposal and Review process.  Version 0.2 is the implementation of proposal submission and review that permits users to experience them in practice and evaluation efficacy.  Please provide the following feedback broken down by Review Stages 2–4 (see below)The TTA Tools are available on the internal network at https://tta.nrao.edu/

In this review, the google forms linked below will ask for feedback in the following categories:

  • Predictability. Is the user experience predictable?
  • Process. Is the process reasonable?
  • Usability. Too many clicks, tool tips, missing info?
  • Bugs. Report any issues.
  • Documentation.  Are there any gaps in the documentation?

Feedback form for Individual Science Reviews (available immediately)

Feedback form for Consensus Reviews (available at the start of the Consensus meeting)



Logistics

The different review stages are listed below.  Review stages We have divided the testing into five stages (see below).  Stages 1–4 will primarily be done offline (i.e., we do not need to organize a Zoom meeting).   For select individuals, however, we want to have our software front-end developer watch how the user interacts with the software.  These cases are listed belowunder Scheduled Zoom Meetings.  During the Consensus review (stage 5)  we will all meet via Zoom.

 Release


Test Schedule:

StageActionWhoWhen

(1) Release Proposals for Review

TTA

member

Member configures SRP/vets

proposal

science category.  Sends email.

TTA

member sends emails.

Member

9/18/23
(2) Conflict
 Conflict
Declaration

SRP

members declare conflicts

member declares conflict of interest.

  • Zoom Meeting: Emily (1 hr)
  •  SRP

    SRP Members

    9/18/23  - 9/21/23
    (3) SRP Chair Assignment

    SRP chair assigns reviewers to proposals.

  • Zoom Meeting: Loreto (1 hr)
  • SRP Chair

    9/22/23
    (4) Individual Science Review

    SRP members perform Individual Science Reviews.

  • Zoom Meeting: Emily (1 hr)
  • SRP Members

    9/23/23 - 9/28/23
    (5) Consensus Review
     Panel

    SRP meets to perform Consensus

    Reviews.
  • Zoom Meeting: All (2 hr)
  • Review.

    All9/29/23


    Scheduled Zoom Meetings:

    WhoWhatWhenZoom Contact
    EmilyConflict Declaration

    9/21/23 1-2pm EDT (11-12pm MDT)


    LoretoSRP Assignment

    9/22/23 10-11am EDT (8-9pm MDT)


    EmilyIndividual Science Review

    9/26/23 1 - 2 EDT

    (11 - 12 MDT)


    AllConsensus Review9/29/23 3-5pm EDT (1-3pm MDT)https://go.nrao.edu/nraozoom08

    Testing Plan and Roles:

    We plan to simulate the review process for one Science Review Panel (SRP).  To do this a TTA Member will generate about 25 mock proposals to be considered for review by one SRP.  To be realistic the SRP will be comprised on one chair and five members.  Here are the roles: 

    Roles:

    • TTA member (admin): Dana
    • SRP chair: Loreto
    • SRP members: Allie, Jeff, Amy, Robert, Emily

    Proposals prepared for the review:

    The proposals included in this review have a Proposal ID of Sem_26A.

    The proposals creation portion of this application is not fully complete; as such, the content of the proposals has been significantly limited for this review.

    How to review simulated proposals:

    Though you cannot evaluate the proposals for the actual scientific merit, please provide a range of scores to the proposals such that they will be normalized realistically for the Consensus Review.

    Please also considering fabricating a brief comment for each proposal so the Consensus Reviews appear mildly realistic. Here are a few examples you could replicate as desired into the review comments. Even if the comment is a single sentence, it will make the Consensus UI appear more realistic than random letters. It is fine if there is a lot of duplication across proposals and reviewers.

    • This proposal is well justified.
    • This proposal could have justified the use of resources more.
    • This proposal will not address the science goals it proposes.
    • This proposal can provide valuable results for the community.
    • This proposal did not account for the available archival data and it is not clear if it would provide novel results.