TTA Tools Sprint Feedback Page

Just a note - when someone is an external reviewer, they see the "select a panel" screen instead of being taken directly to their SRP

Dana (Firefox, Fedora)

Goal: One last run through the system to make sure we are ready for
      the V0.2 Internal Testing.

----------------------- 
User: Dana Balser_TTA |
-----------------------

Solicitations
-------------

1. Generate new Solicitation: "dsbTest61", "dsb61", "PPR", and "25".
   Looks good.

2. Close Solicitation.  Looks good.  Got the 25 emails.

Reviews
-------

1. Configure SRPs.  Add panel "GWT Panel" with "GWT" as the science
   category.  Select reviewers: Lorant (chair), Allie, Mark, Amy,
   Ryan, and Emmanuel.  Looks good.

2. Vet Science Categories.  Looks good.

3. Save and Finalize.  Seems okay.  I am brought back to the Summary
   page.  Received the 6 emails, one for each SRP memeber, that the
   panel was open for business.

-------------------------
User: GWT Panel Members |
-------------------------

1. Declare conflicts.  UX is better.  I can see the certified badge
   right away along with the new page to start my reviews.  

------------------------
User: Lorant Sjouwerman |  Chair
-------------------------

1. Assign Reviewers.  Overall okay.  Still a bit hard to absorb.

-------------------------
User: GWT Panel Members |
-------------------------

4. Individual Reviews.  Login as each reviewer and use the file
   download/upload functionality to enter reviews.  We probably want
   to add the proposal title to the file since this will help to
   recall the proposal in question.  Once all members have finalized
   their reviews I get the email notification.

----------------------- 
User: Dana Balser_TTA |
-----------------------

Reviews
-------

1. Start Consensus.  

-------------------------
User: Lorant Sjouwerman |  Chair
-------------------------

Reviews
-------

1. Consensus Reviews.  The "T's" now show up for the chair and I can
   see the two Tertiary reviewers.

2. Revote.  Seems like this is working now.  When I change an
   individual normalized score the mean and SRP Score are updated.
   The orignial individual normalized score is displayed.  We do not
   see the orignial mean normalized score but that is probably okay.
   If I change the SRP score directly I can still revote and update
   the mean, but not the SRP score.  Looks good.

3. Download Reviews.  This is now gone as requested.   

------------------
User: Allie Costa:
------------------

Reviews
-------

1. Concensus Reviews.  This all seems to work okay.  I can edit
   Primary and Secondary text only.  I cannot change Individual
   Normalzied score or SRP score.  If I complete a review then I can
   no longer make changes but the chair can do so.

-------------------------
User: Lorant Sjouwerman |  Chair
-------------------------

Reviews
-------

1. Complete and Finalize.  I go through and check that even when a
   review is completed I can change the review as chair.  Once all of
   the reviews are completed I see the Finalize button.  Once I click
   this button then I cannot make any changes as chair.  I also receive
   an email notification.

Conclusion: Looks good to go for internal review.

Dana (Firefox, Fedora)

Goal: Test run to see if we are ready for the V0.2 Internal Testing.

--------------------------------- 
User: Dana Balser_TTA |
----------------------------------

Solicitations
------------------

1. Generate new Solicitation: "dsbTest60", "dsb60", "PPR", and "25".
   Looks good.

2. Close Solicitation.  Looks good.

Reviews
------------

1. Configure SRPs.  Add panel "GWT Panel" with "GWT" as the science
   category.  Select reviewers: Lorant (chair), Allie, Mark, Amy,
   Ryan, and Emmanuel.  Looks good.

2. Vet Science Categories.  Looks good.

3. Save and Finalize.  Okay, but not clear that I have finalized
   unless I leave the page and come back.

-------------------------------------
User: Lorant Sjouwerman |  Chair
--------------------------------------

1. View Summary page.  First impression.  Not bad.  I can view all 6
   reviewers on one page.  Still a bit busy but without much effort I
   can see the status: the folks that have not declared their conflict
   yet and some that have been automatically conflicted (see here).

2. Declare conflicts.  Once I hit the button the badge still says
   uncertified.  I go to the Summary page and still see that Lorant's
   reviews are uncertified.  When I refresh the browser I can now see
   that Lorant's reviews are certified but the proposal list is
   missing (see here).  The proposals are also missing on Individual Reviews
   tab. If I login as another reviewer (Allie Costa) I cannot see the
   list of proposals to perform conflict declaration (see here).  If I go to the
   Proposals tab as a TTA Member (Dana Balser_TTA) I do not see the
   list of proposals.  After refreshing the browser a few times I can
   see the proposals.  Then if I go back to the Reviews tab I can see
   the list of proposals on the Summary page and the proposal-reviewer
   matrix on the Reviews->Reviews->Summary page.  If I login as the
   chair (Lorant Sjouwerman) I can now see the proposal-reviewer
   matrix on the Summary page.  Next login as each SRP member and
   declare conflicts. Again I do not initially see the certified badge
   after clicking the button but otherwise it seems to work.

3. Assign Reviewers.  The UX is okay but the number of P,S,T, and N
   values are hard to absorb.  Partly because the numbers are small
   and hard to read.  Also, the overall matrix is complex once the
   P,S,T, and N boxes are shown.  Lots of small boxes with different
   colors. 

4. Individual Reviews.  Login as each reviewer and use the file
   download/upload functionality to enter reviews.  The Individual
   Reviews page is improved.  Conflicts are clear and I can filter.  I
   like the UX when finalizing the ISR---the popup with high/low
   score.  Once the reviews are finalized for a given reviewer the UX
   on the Summary page becomes less cluttered as the P,S,T, and N
   boxes go away and the cell is labeled as Primary, Secondary, etc.
   This is not the case for proposals with review type N.  At this
   point, however, there are just too many colors (see here).  Maybe just remove
   all colors associate with P,S,T, and N.

--------------------------------- 
User: Dana Balser_TTA |
---------------------------------

Reviews
------------

1. Start Consensus.  As discussed the location of this functionality
   will change as having a button on the Summary page is not what we
   want.  When the button is clicked the matrix changes a bit.  The
   P,S,T, and N buttons go away for those cells with N and the words
   None are listed.  This is because once consensus is started you can
   no longer change the review type.  Okay.

-------------------------------------
User: Lorant Sjouwerman |  Chair
-------------------------------------

Reviews
-------

1. Consensus Reviews.  Overall the layout is pretty good.  Couple of
   issues.  Lorant's Tertiary reviews are not listed in the column
   next to the proposal ID.  I have "P", "S", and "Conflict Symbol",
   but no "T" for SRP chair.  Too, there is only one Tertiary reviewer
   name listed under that proposal title.

2. Download Reviews.  I get the consensus reviews for all
   Solicitations instead of just "dsbTest60".  We will want additional
   info here (e.g., mean, std., etc.).
   

------------------------
User: Allie Costa:
-------------------------

Reviews
------------

1. Consensus Reviews.  For an SRP member I do get "T" in the column
   next the proposal ID.  No Primary, Secondary, Tertiary names under
   the Title. This seems reasonable since the chair is running the
   meeting.  I notice that I cannot see Consensus comments or
   Individual comments for proposals for which I am conflicted.  I
   cannot edit Consensus comments if T or N but can view them.  This
   appears correct.

2. Enter Consensus Comments.  Enter draft comments manually for
   Primaries.  I have to enter Internal comment to save which should
   not be a constraint.  Perform the same task for the other SRP
   members.

-------------------------------------
User: Lorant Sjouwerman |  Chair
-------------------------------------

Reviews
-----------

1. Enter Consensus Comments.  Enter draft comments manually for
   Primaries.

2. Edit SRP Score.  This seems straight forward.  I am not able to
   edit scores for which Lorant is conflicted which is the correct
   behavior.  

3. Re-vote.  The interface is a bit clunky.  Not much real estate, and
   when I click save I am put back in Summary view.  In practice I
   would need to edit several of these at a time.  Too, this does not
   seem to work.  When edit the Individual score for a reviewer and
   click save the SRP Score does not update.  When I go back to see
   the Individual Score it is now blank.  If I do not click Save then
   the Individual Score persists but the SRP score is still not
   updated.

4. Complete/Finalize.  I have to complete all reviews before I can see
   the Finalize button.  I had to login as a TTA Member to Complete
   proposals for which the chair (Lorant) was conflicted.

Conclusion: Close to being ready for internal testing.  Would be good
to investigate the missing proposals.  Or at least to try to reproduce
it.  We should probably fix the re-vote functionality unless I am
doing something wrong.

• STT-1330
  • new Chair UI in ISR phase
    • Conflicted status is displayed, regardless of auto or user declared
    • message saying "Awaiting Conflict Declaration" is shown when not certified. Prevents chair from assigning a type.
    • updates once reviewer has done their certifications. Shows certified badge.
    • reorders list of proposals base do who logged in
    • still shows review state of proposals in addition to the assignment of the review type. it is less cluttered though.
      • I assume that if I were to change the RT of a proposal i would need to hit finalization. I need instructions written on how I will test this.
    • I can save just one person's RT even if the others are conflicted,
    • Once i finalize, in the summary view I can no longer change the review type of the reviewer as the chair.
    • It just shows a static badge of review type and the review state of the proposal
      • need to test closed, blank, None, completed.
    • Review Types None with any RS of blank, saved,completed can continue to be edited by srp reviewer and the chair can assign a RT still. Hopefully these are both closed when consensus starts.
    • As a TTA member, I can go in still and change the review types even though they are finalized.
    • When Consensus started, the UI changed for TTA and Chairs
      • Now all RT modifications are gone. Just badges showing the Review Type, the Review State are available, conflict state.
• The number of reviewer badge is now correctly populating for non chairs and non tta members.
• the chair name is not.
• Finalize reviews is not showing up the moment I have access to the ISRs. All ISRs are Blank, None so it could and used to. It shows up once there is the right states now I think.

Consensus

• As a conflicted reviewer, I cannot see the proposal anymore except for the abstract, title and ID. that's good.
• As the chair, I cannot edit a conflicted consensus review
• I can see all the reviewers (As chair) and their roles on an proposal.
  • as a reviewer i only see my role.
  • the filtering on RT is gone.
• As a tertiary, i cannot interact with a proposal but i can see everything.

• STT-1175 New OSR UI

Dana (Firefox, Fedora)

Goal: Check entire path from Solicitation creation to Finalize
Consensus Reviews.  I am not going to worry about trying to break the
system but just see if I get go from A to B and the overall look/feel
and functionality.  Let's setup just one SRP with a typical proposal
load (25) and panel membership (5 reviewers + 1 chair).

TTA member sets things up
========================

----------------------- 
User: Dana Balser_TTA |
-----------------------

Solicitations
-------------

1. Click "Generate Test Solicitation" button.  Enter "dsbTest58",
   "dsb58", "PPR", and "25".  Click "Submit".  Seemed to work.  I did
   not have to refresh browser to see the Solicitation show up in the
   list.

2. Click "Close".  Seemed to work.  Close icon disappeared.

Proposals
---------

I can see the 25 proposal in the dsbTest58 Solicitation.  I also
received 25 email notifications.  Good.

Reviews
-------

1. Select "dsbTest58".  I can now see the list of 25 proposals with
   subtabs: Summary, Configuration, Reviews, and Allocation Disposition.

2. Configuration->Panels.  Enter "GWT Panel" and click "New Science
   Review Panel".  Click GWT Panel.  I can see the "delete" button to
   delete the panel.  Select "GWT" science category.  Search/Select
   reviewers: Lorant (chair), Allie, Mark, Amy, Ryan, and Emmanuel.
   Lorant has one conflict and Allie has no conflicts.

3. Configuration->Vetting.  Click "auto assign panels".  The "GWT
   Panel" was selected for all propsoals as expected.  Click "Vet
   All".  All the vetted checkboxes are checked.  Click "Save
   Proposals" button.  Nothing seems to happen. When I do a
   browswer refresh the "Save Proposals" button is gone and I can see
   a "Finalize" button.  Click "Finalize" button.  I am sent back to
   the the Summary page.  If I go back to Configuration I can see the
   "Finalized" icon (button is gone).  Also, I received 6 email
   notifications, presumably sent to each reviewer, indicating that
   the SRP is open for business.

Declare conflicts of interest
=========================

-------------------------
User: Lorant Sjouwerman |  Chair
-------------------------

Review
------

1. Needed to click the following to get to the conflict declaration:
   Reviews, dsbTest58, GWT panel, Individual Reviews, Lorant
   Sjouwerman.  Lots of clicks but the flow was logical.  I then added
   one conflict for proposal 20.

2. Click "Certify Conflict Declarations" button.  I now see the
   certify badge but I do not see any conflicts in the list which is
   not correct. I see two column of badges to the right.  The first
   column are all blank, presumably indicating the review is blank.
   The second column are all None, presumably indicating that I am
   assigned the review type of None (the default).  But from this view
   I do not know which proposal I am conflicted on.

3. On the Summary page, however, I can see that Lorant is certified
   and see the conflicts.

-------------------
User: Allie Costa |
-------------------

Review
------

1. Declared no conflicts.  Since Allie is not a chair she only sees
   the Individual Reviews and Consensus Reviews tabs which makes
   sense.  I also only see Allie Costa and not the list of other
   reviewers which is reasonable.

---------------------
User: Mark Claussen |
---------------------

Review
------

1. Declared no conflicts.

------------------------
User: Amy Mioduszewski |
------------------------

Review
------

1. Declared a conflict for dsb58-10.

------------------
User: Ryan Lynch |
------------------

Review
------

1. Declared no conflicts.

------------------------
User: Emmanuel Momjian |
------------------------

Review
------

1. Declared no conflicts.

GWT Chair assigns reviewers to proposals
====================================

-------------------------
User: Lorant Sjouwerman |  Chair
-------------------------

Review
------

As already discussed the summary page is too busy but the details look
correct before the chair begins assigning reviews.

1. Summary tab: assign reviewers to proposals: P, S, or T.  Then click
   "Save".  I get the following error:

"A backend error occurred at 'api/individual_science_reviews' : Expected individual science review conflict declaration to be available. Found a conflict state of Conflicted for ISR id 172"

2. Upon inspection I had assigned P,S, or T to a reviewer on a
   proposal for which they were conflicted.  After changing these
   assignments I clicked the "Save" button and get the following
   error:

"A backend error occurred at 'api/individual_science_reviews' : Expected individual science review conflict declaration to be available. Found a conflict state of AutomaticallyConflicted for ISR id 246"

3. Found one more assignment with a conflict.  Fixed this and then
   click the "Save" button.  Works this time.  

Perform Individual Reviews
========================

-------------------------
User: Lorant Sjouwerman |  Chair
-------------------------

Review
------

Inspection:

- The individual review tab for the chair list the reviewers with some
info.  The tab is highlighted which is good.  When I click on
"Lorant Sjouwerman" I go to his reviews which is fine but the
"Individual Reviews" tab is no longer highlighted.

- It would be useful to filter/order the proposal list.

1. Individual review.  Use the file download/upload to enter reviews.
   The download seemed to work fine.  I cannot recall the exact
   requirements of the file format.  No field to complete or an
   indication of conflicts.  When I try to upload I get the following
   error (appears I have to include a review for all proposals which I
   do not think is the behavior we want):

A backend error occurred at 'api/science_reviewer/43/import_isrs' : CSV errors found when attempting to import Individual Science Reviews: Row 2: No comment for the SRP found, Row 15: No comment for the SRP found, Row 10: No comment for the SRP found, Row 9: invalid individual score found - score must be between 0.1 and 9.9 in increments of a tenth, Row 15: invalid individual score found - score must be between 0.1 and 9.9 in increments of a tenth, Row 5: No comment for the SRP found, Row 18: No comment for the SRP found, Row 10: invalid individual score found - score must be between 0.1 and 9.9 in increments of a tenth, Row 3: No comment for the SRP found, Row 9: No comment for the SRP found, Row 18: invalid individual score found - score must be between 0.1 and 9.9 in increments of a tenth, Row 19: No comment for the SRP found, Row 8: invalid individual score found - score must be between 0.1 and 9.9 in increments of a tenth, Row 19: invalid individual score found - score must be between 0.1 and 9.9 in increments of a tenth, Row 8: No comment for the SRP found, Row 3: invalid individual score found - score must be between 0.1 and 9.9 in increments of a tenth, Row 2: invalid individual score found - score must be between 0.1 and 9.9 in increments of a tenth, Row 5: invalid individual score found - score must be between 0.1 and 9.9 in increments of a tenth 

2. Enter reviews manually.  I can enter reviews for proposals for
   which I have not been assigned or are conflicted which is not
   correct. Once I complete all of the assigned reviews there is no
   initial indication that I am done.  If I go to the Individual
   Review tab with list of reviewers, however, I can see the finalized
   badge, but this needs to be more prominent.  (I do get an email
   notification).  Would be useful to allow one to order the proposals
   (e.g., proposal ID, review type, status).  Need to see which
   proposals I am conflicted on.

-------------------
User: Allie Costa |
-------------------

Review
------

1. Enter reviews manually.  I do not receive an email
   notification. If I login as the GWT chair, however, I can see
   that Allie's reviews have been finalized on the Individual Review
   summary view.
   

---------------------
User: Mark Claussen |
---------------------

Review
------

1. Enter reviews manually.I do not receive an email
   notification.

------------------------
User: Amy Mioduszewski |
------------------------

Review
------

1. Enter reviews manually.  I do not receive an email
   notification.

------------------
User: Ryan Lynch |
------------------

Review
------

1. Enter reviews manually.  I do not receive an email
   notification.

------------------------
User: Emmanuel Momjian |
------------------------

Review
------

1. Enter reviews manually.  I do not receive an email
   notification.

Inspection:

- The only indication in the UI that all reviews have been finalized
is for the chair to go to the Individual Reviews tab in the summary
view.  Maybe when we rework the Summary tab this will be more clear.
But there was also no email notification.

- After discussion with Allie and Reid the issue is that finalize is
checking against a state of Blank without cross-checking the Review
Type.  So all reviews must be completed.  This may explain the
behavior of the file import.

-------------------------
User: Lorant Sjouwerman |  Chair
-------------------------

Review
------

1. Click "Dowload Reviews" button.  Edit file offline.  Add a review
   (text + score) for every proposal.

2. Click "Upload Reviews" button, select file, and then click
   "Submit".  This appears to work.  Interesting, for the proposals
   which had a state of "Completed" from the previous work are now
   "Saved" after the upload.  The state field does not exist in the
   file format.  I now, however, see the "Finalize Reviews" button.

3. Click "Finalize Reviews" button.  I see the state change to
   "Finalize" only for those proposals for which the Review Type is
   primary, secondary, or tertiary.

-------------------
User: Allie Costa |
-------------------

Review
------

1. Use file Download/Upload to perform the remaining reviews.  I was
   able to include commas in the review text, which were enclosed in
   quotes, and this worked fine.

2. Click "Finalize Reviews" Button.  It would be good to have some
   indication on this page that all reviews are finalized.  I did not
   receive an email notification, but maybe this is the requirement.

---------------------
User: Mark Claussen |
---------------------

Review
------

1. Use file Download/Upload to perform the remaining reviews.

2. Click "Finalize Reviews" Button.

------------------------
User: Amy Mioduszewski |
------------------------

Review
------

1. Use file Download/Upload to perform the remaining reviews.

2. Click "Finalize Reviews" Button.

------------------
User: Ryan Lynch |
------------------

Review
------

1. Use file Download/Upload to perform the remaining reviews.

2. Click "Finalize Reviews" Button.

------------------------
User: Emmanuel Momjian |
------------------------

Review
------

1. Use file Download/Upload to perform the remaining reviews.

2. Click "Finalize Reviews" Button.

Inspection:

- After all reviewers reviews were finalized I did receive an email
notification to this effect.  Good.  But not easy to tell in the UI
that all reviews are finalized.

- But when I login as the SRP chair and go to the Consensus tab I do not
see any proposals listed, even after a browser refresh.  After some
discussion with Allie I need to login as a TTA member and click the
"Start Consensus Review" button.  Once I do this I can see proposals on
the Consensus tab.

Perform Consensus Reviews
=========================

-------------------------
User: Lorant Sjouwerman |  Chair
-------------------------

Review
------

Inspection:

- Consensus Review tab: Not all reviewers are listed under the
  title. For example, Allie is not included for dsb58-006.  Also, the
  mean/std is zero for proposals for which Lorant is conflicted but
  these numbers should be listed.  The SRP score does not match the
  mean.  Hard to distinguish between different states (e.g., blank vs
  saved).  Different colors would be helpful.

- Consensus Review tab-->Proposal: For conflicted proposals the author
  is listed.  Should be anonymous.  Should not be able to see
  scientific justification or any Allocation Requests.

- Consensus Review tab-->Reviews: For conflicted proposals the author
  should not be able to see the Individual reviews (score or text).
  Not sure why the UI real estate is so large for the score.  The
  normalized scores look okay.  Here is what I get for Allie where the
  UI has a normalized score of 3.4 for dsb58-014.

Proposal ID   Raw     Norm
   dsb58-001  4.0000  3.4495
   dsb58-006  7.0000  6.4397
   dsb58-003  2.0000  1.4560
   dsb58-022  7.0000  6.4397
   dsb58-004  8.0000  7.4365
   dsb58-021  5.0000  4.4463
   dsb58-005  4.0000  3.4495
   dsb58-020  3.0000  2.4528
   dsb58-009  6.0000  5.4430
   dsb58-010  4.0000  3.4495
   dsb58-018  6.0000  5.4430
   dsb58-011  8.0000  7.4365
   dsb58-012  7.0000  6.4397
   dsb58-016  7.0000  6.4397
   dsb58-013  9.0000  8.4332
   dsb58-014  4.0000  3.4495
   dsb58-002  6.0000  5.4430
   dsb58-019  3.0000  2.4528

1. Enter Consensus comments for all proposals for which Lorant is
   Primary.  In order to save comments I have to include Internal
   Comments which is not the desired behavior.

-------------------
User: Allie Costa |
-------------------

Review
------

1. Enter Consensus comments for all Primary proposals.

---------------------
User: Mark Claussen |
---------------------

Review
------

1. Enter Consensus comments for all Primary proposals.

------------------------
User: Amy Mioduszewski |
------------------------

Review
------

1. Enter Consensus comments for all Primary proposals.

------------------
User: Ryan Lynch |
------------------

Review
------

1. Enter Consensus comments for all Primary proposals.

------------------------
User: Emmanuel Momjian |
------------------------

Review
------

1. Enter Consensus comments for all Primary proposals.

2. I was unable to see Consensus comments for proposals for
   which Emmanuel was conflicted.

3. I was able to see but not enter Consensus comments for proposals for
   which Emmanuel was Tertiary or None

4. I was able to edit Consensus comments for which Emmanuel was
   Secondary.

5. I was never able to edit the SRP score.

-------------------------
User: Lorant Sjouwerman |  Chair
-------------------------

Review
------

1. Modify the SRP Score and click "Save".  Seems to work.

2. Change the Normalized Individual score for a reviewer.  Allows the
   chair to change the value and displays the original value (called
   "As Finalized").  The Mean value or SPR Score is not updated.

----------------------- 
User: Dana Balser_TTA |
-----------------------

Review
------

Inspection:

- Interestingly Dana, that's me, is listed as conflicted on all
  proposals.  This is not correct.

- I am able to change the SPR Score and do the re-voting (change the
  Normalized Score) with the same behavior as the chair above.

-------------------------
User: Lorant Sjouwerman |  Chair
-------------------------

Review
------

1. Mark as Complete all reviews for proposals for which Lorant is not
   conflicted.  Indeed, I cannot complete proposals for which Lorant
   is conflicted.  (As a regular SRP member I can only Complete
   proposals for which I am either Primary or Secondary as expected.)

2. After the last review is Completed I see the "Finalize Consensus
   Reviews" button.  When I click the button the state goes to
   "Finalized".  I can also now see the Normalized Linear Rank Score.
   This is not really a score so I would call it "Normalized Linear
   Rank".  The linear ranks are correct based on the SPR Scores. I received an email notification.

General Issues:

1. Need to redo the Summary page.

2. Need to be able to export Summary page info.  This is one of the most
   useful tools for the chair when running the SRP meeting.

3. Need more functionality on the Individual Review page.
   Filter/Order features and an indication of conflicted proposals.

4. Need to sort out the behavior of the File Download/Upload process.

Notes for Dana (no testing spoilers) -

FE and BE has not been hooked up for Consensus Reviews. This work is being done now, so you can start consensus but not save comments or change scores yet.

Allie

• Refresh Bugs:
  • When the Summary tab first loads in a Solicitation's Proposal Process, it needs a refresh the very first time it opens. After that, it is fine and I see a list of proposals. Might consider what information could go here
    • Number of proposals? Large vs regular?
    • Facility, Joint, External?
    • Metrics landing page?
  • When I first went into Reviews→ Reviews→GWT, it shows all of the proposals in the solicitation instead of just the ones assigned to GWT. It did not show the Review Type assignment matrix for them. I had to refresh and then only the proposals assigned to the panel are available for TTA member or Reviewer.
• Review Configuration – not testing because of overhaul in UI in sprint 58
• Vetting
  • Changing panel assignment resets (collapses) all of the rows. If there are unsaved SC changes or "Vetted" changes, they revert.
  • Clicking Auto Assign Panels again gives an error: Object Object
  • Clicking Finalize sends me back to home page
    • Got 6 emails about panels being open (3 per panel)
• As a TTA member I should not be able to modify the configuration, vetting, or panel assignment of proposals after the configuration is Finalized.
• As a TTA member I need to be able to add an external reviewer.
• Reviews (Chair)
  • Summary
    • See proposal matrix (skip for now)
    • Return after all conflicts are certified
    • Can assign to conflicted, story not done yet.
  • Individual Reviews
    • First conflicts – as expected
    • Redirects to ISRs; certified icon changes.
    • RS and RT shows Blank/None until they are updated, as expected
    • Can mark Completed on ISRs without comments (I think we said that was okay?)
    • Error thrown when trying to complete/save a ISR score of 0
    • Used TTA powers to do conflict declaration for 1 reviewer – got an email about it
    • got MANY emails about insufficient reviewers
    • On list of reviewers, the Finalized icon is showing as Finalized before I have actually finalized. Then it isn't showing Finalized when the ISRs are finalized.
      
      • When I look at the list of reviewers in ISRs, I want the Finalized icon to display the Review State of Finalized accurately so it is clear if reviewers have finalized.
    • As a SRP member I do no want to be able to enter scores or comments into ISRs for which I am conflicted nor should I see the proposal.
    • When I use the Finalize Review as a TTA member,
      • There isn't  visual feedback that the finalization has worked without a refresh.
        • When ISRs are Finalized, I want feedback in the UI so I know the action was successful.
        • As a reviewer, I only want the Finalize button to be available when the condition is correct so I don't use it inappropriately.
          • Finalized as the Chair and it is still letting me hit the finalize button. – It's because there is a RT of None here. There isn't the condition
          • I can finalize with no RT assigned when all of the nonconflicted proposals have RS of saved/completed. This isn't the requirement.
          • Bigger issue: I made 2 reviews with the same score and the other ones I left blank. Then I went in as a TTA member and finalized them, which set the blank ones with RT nq None to Closed. Then finalization was allowed initially, I presume, because of the 4 with RT nq None, they did not all have the same score but since the two non unique ones were Closed, they not included in future finalization calculations. This is expect but now the onle two reviews have the same score. I suspect it will NaN out in the next phase. What are exact conditions for finalizing even as a TTA member. This is protected against as a normal user already, but a TTA member could mess it up.
          • Should close out first and then calculate the finalized normalized individual score.
      • Any RT not equal to None and RS equal to Blank get state changed to closed.
      • I can still use the Finalize button as a reviewer, which gives an error.
      • I can use the Finalize button as a TTA member (as it should be) but it also gives an error.
      • Finalizing sends all ISRs with RT of None to RS of Closed. Changes Completed or Saved to Finalized if RT nq None.
    • As a TTA member, I need to be able to edit Finalized ISRs until the start of consensus.
    • As a Chair, I do not want to be able to edit the Review Type of an ISR if it Finalized.
      
      • Once the ISRs are finalized, I can still access the buttons but then the save buttons hangs. Presumably, it is because I cannot edit the RT. This needs more UX consideration
      • (STT-1200 for tta powers)
    • As a reviewer, I do not want to be able to edit ISRs with Review States of Closed.
      • Note, a TTA member can edit Closed and saving/completing takes it out of closed. There isn't a way to make it closed again though. I can change the RT so it won't go into consensus I suppose.
    • As a TTA member, I would like to be able to Close a single ISR until the start of Consensus.
    • As a reviewer, I want the name of the chair to be displayed for my panel.
    • As a reviewer, I want the icon that tracks how many reviewers to either display correctly or be taken out.
    • As a reviewer, I would like filtering and sorting on my ISRs.
  • Consensus Review (Only partial as it isn't hooked up)
    
    • As a Chair, I should not be able to see the button that starts Consensus.
      • The triggering criteria for consensus phase is correct, in that it only appears once all of the members have finalized their ISRs.
      • I also got an email saying that all ISRs for GWT panel are finalized.
      • Only the TTA member using the button actually starts consensus I think though.
      • Even after consensus was started, the button is still available.
    • As a TTA member, I do not want to see the start consensus button once consensus has started for a panel so I will not be confused by the progress of the panel.
    • As a TTA member I would like the Reviews/Reviews page to show what subphase the panel is at (ISR or Consensus).
    • As a reviewer, I would like the Reviews/Consensus tab to display an informational message until Consensus is actually started.
    • As a TTA member, I would like to be able to edit the message displayed in the consensus tab at will.
    • What is the purpose of the State filter in Reviews/Consensus? It doesn't seem to work and is populated by the Proposal State model instead of the Consensus Review State model?
      
      • As a reviewer, I want the state filter to be the consensus review state model.
    • When a TTA member I can see the names of who is primary/ secondary per proposal, which is good. As a reviewer though, it only shows "my" name.
      • As a reviewer, I want to see the name of the Primary and Secondary so I know who is leading the discussion.
      • As a reviewer, I would like the filtering on primary to display more than just my RT primary proposals.
    • Generally, the Chair view is quite right as compared to the TTA member and reviewer. It should be close to the TTA member than the reviewer at this point I think.
      
      • As a chair I need more tta member like visibility into consensus reviews so I can finalize the consensus reviews.
• Misc
  • When I came in as Toney Minter, I can suddenly see in the list that he is not assigned to. Clicking on them, however, gives an error.
  • Got an error certifying conflicts once.
  • Mixed Conflicted states on TM now.

• Demoable Stories:
  • STT-861: Hidden & Submit
    
    • Can submit hidden proposals when a solicitation deadline has passed.
  • STT-1188: Restrictions on withdrawing a proposal.
    • The default behavior is a proposal with In Review state cannot be withdrawn (e.g., OSR proposals), but PPRs are treated slightly differently. Once you Finalize the Review Configuration in a PPR proposal process, you cannot withdraw. Note, a new Proposal Process, should we make one, will default to only allowing withdrawing when Proposal State = Submitted. If we want this behavior modified for a new Proposal Process, a case statement needs to be added to the software.
  • STT-1319: OSR works again
  • STT-1318: ISR subphase works again
    • Note, the "chair" icon that shows the chair name is being hidden now because of anonymous data. Will address later as a cosmetic issue.
• Carryover
  • STT-1234:
    • Still need more time to wrap up the refactor.
  • STT-918:
    • Notifications are not working on test. Will fix for next sprint

Notes:

• we should report any "refresh" bugs, as it should be fixed now.
• In sprint 58, the review configuration UI will be overhauled so that portion of the review process likely does not need close scrutiny until that happens.
• Part of the Review config overhaul is changing the behavior of the panel creation drop down tool, which currently resets the collapsed panel with each edit.

Allie on Dev

Solicitation generation worked after a refresh to populate the proposal process field.

Did not creat a proposal but went to reviews tab.

Reviews *prior to close

*Summary

No changes - just a list of proposals in the solicitation

*Configuration

**Panels

Able to add panels, assign SC, assign and delete reviewers, assign 1 chair.

I need to refresh all the time though to see my changes. Also, every assignment of a reviewer to a panel collapses the panel view which is very obnoxious.

We still want a reviewer focused interface here.

The save changes button only appears when I make changes not related to the reviewers. Not sure if the changes to reviewers persists yet without the save. It looks like it does though. It is strange that there is different behavior between the fields (e.g., SC assignemnt) and the reviewer assignment.

Vetting doesn't look like it is persisting.

I can still go to the reviews tab though.

As a Chair, I click Reviews → solicitation → mypanel and see three tabs: Summary, individual reviews, consensus reviews. I cannot assign RT or declare conflicts.

Proposals are not populating for the chair though.

As a reviewer, I click Reviews → Solictation → mypanel and see two tabs: Indiviual Reviews, Consensus Reviews. By default, a list of proposals is shown but I cannot interact with it. When I click IR or CR, it shows nothing. Hitting a refresh shows some of the conflict declaration page. I cannot actually declare conflicts though.

I think this is all I can test now.

Allie on test

There was a stale log in apparently, that was only "fixed" by logging in as a normal user and then returning as a TTA member. Refreshing and logging out did not help.

Solicitation

**Generation**

Successfully make a Solicitation with PPR and 10 proposals.

Received 10 email notifications about their submission.

**Prior to Close**

Reviews

Can access the Reviews Tab. It lists by solicitation. There are error messages that comes up when I hit the reviews tab

"a backend error occurred at api/science_reviewers': <html> <head> <title> 404 not found </title> </head> </body> <h1> 404 not found </h1> the resource could not be found. <br/> <br/> \science_reviewers </body></html>"

"a backened error occured at api/ppr_proposal_reviews: 404 not found .. the resource could not be found... "

"a backend error occured at api/osr_proposal_reviews: 404 not found ... the resource could not be found..."

In the ahc-1 solicitation proposal process, there are four tabs: Summary, Configuration, Reviews, and Allocation Disposition. I should be able to configure the review panels even prior to the close of the solicitation.

*Summary

Lists the proposals by proposal ID and Title. No interaction beyond that though at this point.

*Configuration

Generates an error: "A backend error occured at api/individual_science_reviews: <html> ... 404 Note found .... the resource could not be found.

Generates an error: object Object

View shows two subtabs: panels and vetting. Neither are really functional beyond this. The vetting is fine as the solicitation is open.

**Close the Solicitation**

Panels and Vetting still not functional.

Proposal Creation

Created proposal for Sem25A.

As an author, I would like to only specify the Solictation when I first create a proposal so I only have to specify solicitation specific information.

As an author, I would like to have information about the available Solicitations when I create a proposal so I know which one to select.

As an author, I would like a visual clue as to why I can't access the observatory copy for a proposal in the draft state.

I have to refresh a lot but generally it working as I expect. I can resubmit.

As an author, I would like to be able to access the observatory copy after I submit without needings to first navigate out of the proposal creation screen.

As an author, I would like the view of the observatory and author copy to populate correctly when not in edit mode.

Dana (Firefox, Fedora)

Testing for issues after the refactoring.  In scope is Solicitations and Proposals. Review is not ready.  Know issues are here: Sprint 56 - Known Issues.  Per request, if an issue can be fixed by a refresh then there is no need to note this here.

---------------------------------- 
User: Dana Balser_TTA |
---------------------------------

Solicitations
-----------------

First create a new Solicitation.

1. Click "Generate Test Solicitation" button.  Enter "dsbTest56",
   "dsb56", "PPR", and "10".  I can view the Solicitation.  I received
   the 10 email notifications.  I can view the 10 proposals.

-------------------------------------
User: Lorant Sjouwerman |
--------------------------------------

Proposals
--------------

Begin by creating a new proposal in Solicitation "dsbTest56".

1. Click "Add".  Enter Title, Solicitation, and science category.

Basic Information
-------------------------

2. Add abstract and then click "Save".  So far so good.

Scientific Justification
------------------------------

3. Upload a PDF file.  No problems.
4. Click view.  I can see the PDF.

Allocation Requests
----------------------------

---> Capability Requests

5. Click "Add" under Allocation Requests.
6. Edit Allocation Request name; click "Save."
7. Click "Add" under Capability Requests.
8. Field Sources. Import from a file.  Seems to work okay.
9. Spectral Specifications.  Import from a file.  Seems to work okay.
10. Performance Parameters.  Enter values for Angular Resolution, LAS, and rms.
11. Calibration Parameters.  Select "yes" for Polarization.
12. Click "Save".

---> Observation Specifications

I can see the Observation Specification.  Overall looks okay.  I can
see the polarization calibration, together with the standard flux,
gain, and bandpass calibrations.

Let's go back to the Capability request and adjust the rms noise since
to increase the integration time. Currently rms = 0.001 Jy/beam and
integration time = 1 sec.  Change rms to 0.00025 Jy/beam.
When I do this I still get the same integration time.

Maybe some limit, so let's reduce the rms to 0.000001 Jy/beam.
Hmmm...I get the same result.

One more time.  Change the rms to 0.00000000001 Jy/beam. Okay, now I
get a big number.  Probably just needed to refresh which I did this
time.

Try to get back to rms = 0.000001 Jy/beam which should yield a more
sensible result.  Indeed, I get about 4 hours on source.  Okay.

13. Click "submit".  Seems to work.  I get an email notification.

Dana (Firefox, Fedora)

Testing mostly focused on looking at the new structure where the
Reviews tab now contains the Proposal Process.

--------------------------------- 
User: Dana Balser_TTA |
----------------------------------

Solicitations
-----------------

1. Generate test Solicitation with name="dsbTest51", proposal prefix =
   "dsb51", and count = 10.  Received notifications.
2. Click "Close" button.

Reviews (effectively the proposal process)
-----------

Initially see a list of the Solicitations (name and review process).

1. Click "dsbTest51".  See "Summary", "Configuration", "Reviews", and
   "Allocation Disposition" tabs.  Nothing is listed under Summary.

- Sub-tab: Configuration-->Panels

2. Configure panels: "GWT Panel" with one chair and five SRP memebers.
   Click "Save" button.

- Sub-tab: Configuration-->Vetting

3. Vet proposals: click "auto assign panels" button (all GWT).
4. Click "vet all" button.
5. Click "Save Proposals" button.  So far so good.
6. Click "Finalize" button.  Moved to Home tab.

If I now go back to the Reviews tab for this Solicitation I do see a
list of the proposals in the Summary sub-tab.

- Sub-tab: Reviews-->Summary

I can now see the matrix: proposals by reviewers.

- Sub-tab: Reviews-->Individual Reviews

I see a list of the reviewers with some summary info.  If I click on a
reviewer I see either the conflict declaration page or the ISRs.

Login as each reviewer and declare conflicts.  Notice that only the
chair has the Reviews-->Summary sub-tab and can also see the list of
other reviewers in the Reviews-->Individual Reviews sub-tab.  If I
login as chair I can see in the Reviews-->Summary sub-tab that all SRP
members are certified.

Next login as each reviewer and perform ISR, mark as completed, and
then finalize.  One reviewer had only on ISR assigned so when I finalized
I got the following error:

"A backend error occurred at
'api/individual_science_reviews/finalize/20' : Expected normalizable
individual scores not to all be the same, found instead [3.0] for
ScienceReviewer with User ID 6 "

So I login as the chair and assign another review and then the SRP
member in question is able to finalize their reviews.  Once this was done
I got the email notification that all ISRs are finalized.

I can now see the "Start Consensus Review" button.  Still on the
Reviews-->Summary sub-tab but this is probably out of scope here.
Missing, of course, is the TTA Member page that allow the process to
move from one review stage to the next, but overall the structure here
is reasonable.

When I click the "Start Consensus Review" button I now see the
Reviews-->Consensus Review sub-tab and can see a summary of each
proposal.

Dana (Firefox, Fedora)

--------------------------------- 
User: Dana Balser_TTA |
---------------------------------

Setup stuff.

Solicitations
-----------------

1. Generate test Solicitation with name="dsbTest50", proposal prefix =
   "dsb50", and count = 10.  Received 10 email notifications.  Good.
2. Configure panels: "GWT Panel" and "SFM Panel". Each panel has one
   chair and two SPR members.  Click "Save" button.
3. Vet proposals: 7 as GWT and 3 as SFM (changed).  Click "Save
   Proposals" button.
4. Click "Finalize" button.

Forgot to close Solicitation. F#$K!  Circle wagons.

5. Generate test Solicitation with name="dsbTest50_2", proposal prefix =
   "dsb50_2", and count = 10.  Received 10 email notifications.  Good.
6. Click "Close" button.
7. Configure panels: "GWT Panel" and "SFM Panel". Each panel has one
   chair and two SPR members.  Click "Save" button.
8. Vet proposals: 7 as GWT and 3 as SFM (changed).  Click "Save
   Proposals" button.
9. Click "Finalize" button.  Okay, worked this time.  Received an
   email notification for each reviewer.

Check email notification when less than 3 reviewers.  Test by adding
conflicts to some proposals.  Since there are some automatic conflicts
(proposal 1 and 7) and only three reviewers this would happen
regardless, but the notification should list them.

-------------------------------------- 
User: Lorant Sjouwerman |
-------------------------------------

Reviews
------------

1. Select "Available" for proposals 1-6; auto conflict on 7.

2. Click "Certify Conflict Declarations" button.  Got one email
   notification that insufficient reviewers on dsb50_2-001 and
   dsb50_2-007.  These were both due to automatic conflicts.

-------------------------------------- 
User: Emmanuel Momjian |
--------------------------------------

Reviews
-------

1. Select "Available" conflict on proposal 2; available on rest.
2.  Click "Certify Conflict Declarations" button.  Got one email
    notification with insufficient reviewers on dsb50_2-001,
    dsb50_2-002, and dsb50_2-007.  Seems correct.

------------------------------------
User: Amy Mioduszewski |  Chair
------------------------------------

Reviews
-----------

1. Select "Available" for proposals 2-7; auto conflict on 7.
2.  Click "Certify Conflict Declarations" button.  Got one email
    notification with insufficient reviewers on dsb50_2-001,
    dsb50_2-002, and dsb50_2-007.  Seems correct.

3. Assign reviewers to proposals.
4. Click Save.  

--------------------------------------
User: Lorant Sjouwerman |
-------------------------------------

Reviews
-------

1. Enter ISRs for all proposals.  I was able to do this for the
   conflicted proposal, which should not be allowed, but this has None
   so it should not count.  Mark these as complete.
2. Click "Finalize Reviews".

-------------------------------------- 
User: Emmanuel Momjian |
--------------------------------------

Reviews
------------

1. Enter ISRs for all proposals.  Mark these as complete.
2. Click "Finalize Reviews".

-------------------------------------
User: Amy Mioduszewski |  Chair
-------------------------------------

Reviews
-----------

1. Inspect reviews summary. The state is finalized for all proposals,
   even the one with a self-declared conflict.  For the automatically
   conflicted proposals the state is completed.
2. Enter ISRs for all proposals.  I was able to do this for the
   conflicted proposal, which should not be allowed, but this has None
   so it should not count.  Mark these as complete.

3. Click "Finalize Reviews".  I do receive the email notification that
   all ISRs have been finalized.

----------------------------------
User: Dana Balser_TTA |
---------------------------------

Reviews
------------

Start consensus review.

1. Click "Start Consensus Review" button.  I get the following message:

"A backend error occurred at 'api/science_review_panels/5/launch_consensus' : [object Object]"

Allie (Firefox, Redhat)

General Impressions

• I think that the buttons like Finalize, Save should always be there but just grayed out if I can't use them. This would make is obvious when I first arrive that my end goal is.
• The Conflict Declaration page has some weirdness with it. It will commonly override my inputs and go back to unknown.
• Refreshing a page either helps a broken page or breaks a good page.
• Making a new Solicitation threw the error or "Science Review Panel's IndividualScienceReviews could not be assigned: object Object"
• To finalize a Review Panel, the set of vetted Science Categories must map to a panel's Science Category. e.g., if there are 2 vetted science categories, there must be those 2 science categories assigned to any number of panels.
• If proposals are assigned to a panel and then the panel is modified, all of the  proposals will be deleted from the UI.
  • we can have a story that this work could go into.

As TTA member,

• generate new test solicitation ahcSP49_1 with 23 proposals
• 23 emails received that the proposals were submitted
• the test solicitation generator keeps the fields filled even after I have generated the solicitation.
  • • As a TTA member I would like fields to not keep the previously generated information after a successful submission.
    • As a TTA member I would like the ordering of the Solicitations to have a consistent behavior and for filtering to be available.
• Look at Reviews tab
  • • As a TTA member, I would like the reviews tab  to load optimally.
    • As A TTA member, I would like the ordering of the panels in the reviews tab to have a consistent behavior and to have filtering on the panels.
  • I have no closed my solicitation, so I do not see it on the Reviews tab, which is good. Even after hitting refresh, it does not show up. Good.
  • I logged in as a non tta member and can see the Reviews tab. Initially, it was cached such that I saw it as a TTA member. Once I hit refresh though, then I could only see the panels for which I am a member of.
    • As a product owner, I want to protect against caching issues that would allow users to see content they do not have permission to view.
  • The same happens initially when I log back in as a TTA member. I only see the panels of the regular user at first, and then it populates more panels.

In Solicitation, Panel Configuration now.

• Panels and Vetting are separate. Vetting is hit by default.
• Can change SCs. Can save. Can vet individually. Can vet all.
  • • As a TTA member I would like summary information about the proposals on the vetting page e.g., number of proposals per SC.
• Changing to the Panels Tab does not prompt a warning about losing unsaved content. It looks like the vetting info is preserved despite this but I bet when I make a save to panels, it will revert.
  • It did not. The SC I changed reverted back when I saved on the Panels page.
  • Similarly, when I did not save on the Panel page, went back to vetted and saved, the panels I had created but not saved disappeared.
    • As a TTA member, I would like notification when I am navigating away from unsaved changes in Vetting and Panel Configuration.
• There are now 13 GWT, 5 SFM, and 5 NGA proposals.
• I created a GWT panel, a SFM panel, and a NGA panel.
  • I assigned each their appropriate SC and now the counter says "3/3 Science Categories" as I would expect.
  • What does the icon next to the word "Panels" mean? It shows "2" currently.
    
• All proposals have been vetted. Panels have science categories, and each expected SC belongs to a panel. Since I have panels, I can auto assign proposals to the panels.
  
  • Auto assign looks at the vetted Science Category.
  • I should test if it needs to be saved first before it will do that.
• Now that all proposals have been assigned and all of the SC are attached to a panel, the Finalize button appears.
  • Error - Could not Finalize because solicitation is open. The system message gives the useful error message. The banner just says it cannot be finalized.
• Before I go to close the Solicitation, I looked at the Panels page again. The icon that displays how many proposals are associated with that panel is not populating until a refresh.
• Close Solicitation. Get a message asking if I'm sure. good.
• Go back to Panel Configuration and the page doesn't load correctly. Have to hit refresh. I can now successfully hit finalize and it boots me out to the home page.
  
  • I receive a number of notifications via email that the panels are open.

Go to Reviews page

• In GWT panel. I see everyone that I assigned to it. As a TTA member I have full access.
  • When I go to look at a SRP member, I can see ISRs and Conflict. At the bottom of the Conflict page, there's a banner that says Science Review. I don't think that should be there.

GWT Chair: Amy

• On the Reviews tab, I can see the summary view of all the panels associated with the Solicitation because I am the chair of one of the panels. When I click on them, it gives an error that says "You do not appear on the requested panel". They don't have access but they shouldn't see the other panels. Wait - now I see only the panels I am on. This is not consistent!
  
  • • As a SRP member I want the Reviews tab to only show the panels I am on.
• In the GWT panel, as chair I can see the matrix view. I cannot see the ISRs or conflicts of the other members like the TTA member can (good), I can only do my own.
• First it forces conflict declaration
  • I can see the title, proposal id, and abstract of the proposals. I can mark my status. There are 2 that I am auto conflicted on.
    • An error message warns if I do not give a reason for my conflict.
  • I mark all but 1 as available.  The certification button appears and I can use it. It leaves me on this screen. I can click the "back to panel" button, but it isn't obvious.
    
  • I am now certified.
  • I go back to the panel and then click my name again so I can see my ISRs. It takes me back to the certify conflicts page and I can change my conflicts but it gives an error that it cannot be certified.
    • Bug - After certification, a srp member can only see their conflict declaration page instead of the ISR page.
  • I log in as a TTA member and I can change the conflict and certify without issue.
    • I get notifications stating that the changes were made. There is one for each proposal and not just the ones I changed.
    • As a TTA member I would like to only receive one email per certification change (bulk or only for the one that changed).
• The matrix screen shows that I am available for 10 proposals.

Log in as GWT panel SRP member

• I go to the Reviews page and now only see the panels I am actually on.
• Clicking on the GWT panel takes me to conflict declaration. I do so and certify. It auto puts me to the ISRs, which is good and now I can review them. !! If I refresh the page, I get the conflict certification back !!
  
  • I see the review type and the review state. There is not marker for conflicted and I can see conflicted proposals.
  • I can save and mark as completed.
  • Errors are appropriately given  if my score is not between 0.1 and 9.99.
  • I can complete and save if no comments are made.
  • The Review States change appropriately.
    • As a SRP member, I would like the view to not collapse on a save.
    • As a SRP member, I would like the "Mark Complete" button to not look like it was already used before I use it.
• I certify and start saving ISRs before the Review Type is assigned. The Finalize button does not appear for me? I would think it does.

Log in as GWT Chair

• Wanted to go assign types
• It says everyone is uncertified but I went through and certified!
  • • As a SRP Chair, I do not want to be able to assign a Review Type to a Conflicted reviewer
    • As a SRP Chair, I want to be prevented from assigned multiple Primary Review Types per Proposal. Same for Secondary.
  • The conflicted cells should just be red or something.
• I really can't test any more because I cannot see ISRs.
• When I saved the Review Types, it suddenly populated the review types for all of the other ISRs that I hadn't done?
• I tried to save and got a weird error.
  • A backend error occured at 'api/individual_science_reivews': all isrs for the science review panel must be submitted together. 52 ISRs submitted, 2 expected.

Dana (Firefox, Fedora)

--------------------------------- 
User: Dana Balser_TTA |
---------------------------------

Solicitations
-----------------

First create a new Solicitation.

1. Click "Generate Test Solicitation" button.  Enter "dsbTest49",
   "dsb49", "PPR", and "10".  I received the 10 email notifications.
   I could see the Solicitation and proposals without refreshing the
   browser.  The PIs now do not include TTA Group members (good).  All
   science categories are still GWT.
2. Click "Review Configuration" button for dsbTest49.  I now see two
   tabs: Panels and Vetting.  I have not yet closed the Solicitation.
3. In Panels tab: configure GWT Panel and SFM Panel.  At the top I see
   "Panels 2  ...  0/1 Science Categories".  Not sure what 0/1 means.
4. Click "Save" button.  I now see at the top:
   "Panels 2 ... 2/1 Science Categories".  Still confused.
5. Before vetting let's close the Solicitation.  This is what we would
   do in practice.  Need to click "Solicitation" tab to get back to
   the list view.  Then for dsbTest49 click the "Close" button.  This
   seemed to wipe everything out: I can no longer see the two panels
   that were created.

Start over.  Redo steps 1-4 with "dsbTest49_2" and "dsb49_2".  Let's
first close the Solicitation and then start to configure, etc.

6. Click "Close" button.  This does not appear to work either.  There
   does not appear to be any proposals in the configuration step.

Start over again.  Redo steps 1-4 with "dsbTest49_3" and "dsb49_3".
This time do not close the Solicitation at all and see what happens.

7. Start vetting.  Change the last three proposals from science
   category GWT to SFM.  I still need to select the Panel for each
   proposal.  (I notice later there is an Auto Assign Panels button.)
   Click vet all button which check all of the boxes.  At the top I
   see: "Proposals 10 ...  0 vetted 0 assigned".
8. Click "Save Proposals" button.  I now see at the top:
   "Proposals 10   ... 10 vetted 10 assinged".  I also see
   the "Finalize" button appear.  If I go back to the Panels tab I see:
   "Panels 2 ... 2/2 science categories".
9. Click "Finalize" button.  I get an error message that says:
   "Configuration could not be finalized".  Looking at the system
   messages it appears this is causing problems because the
   Solicitation is open.  Let's try closing the Solicitation.
10. For dsbTest49_3 click the "Close" button.  When I go back to the
    Configure Review view I do not see any proposals but the Finalize
    button is still present.  Click "Finalize" button.  Seems to work
    but when I look at the system messages it appears to fail.  I do,
    however, get 8 email notifications which probably correspond to
    the 6 reviewers on GWT panel plus the 2 reviewers on the SFM
    panel.

Reviews
------------

I do see dsbTest49_3 listed so maybe this worked.  The summary has 6
reviewers/7 proposals for GWT and 2 reviewers/3 proposals for SFM,
which is correct.  Login as one of the reviewers (who has a conflict).

------------------------------------- 
User: Lorant Sjouwerman |
--------------------------------------

Reviews
-----------

For dsbTest49_3, I can see both GWT Panel and SFM Panel.  Seems like I
should only see GWT since Lorant Sjouwerman is not a reviewer on SFM.
If I click on SFM, however, I do get a system messaging saying that I
am not on SFM Panel.

1. Click on GWT Panel.  This gives me the Conflict Declarations page.
2. I am automatically conficted on 001.  I select available for the rest
   except for 007 where I declare a conflict.
3. Click "Certify Conflict Declarations".  I now see the certified
   badge.  I should be conflicted on 001 and 007 but it seems I can
   enter an ISR.  I might put the proposal info on the left and the
   review stuff on the right (in English we read/write from
   left-to-right).
4. Enter ISRs for 002-006 (save all but only complete 005).

------------------------------------- 
User: Amy Mioduszewski |
-------------------------------------

Reviews
------------

Login as chair to inspect the status of the reviews.

1. Select GWT Panel.  I can see the matrix.  Indicates that Lorant
   Sjouwermant is not certified but should indeed be certified.

Oops! I should have assinged reviews first.  Well...a good test
anyhow.  The reviewer should not be able to input ISRs until they are
assigned.  Let's assign reviewers.

2. Select either P,S,T, or N for each proposal/reviewer.  Click
   "Save".  Seems to work but the row/column stats do not make sense.

------------------------------------- 
User: Lorant Sjouwerman |
--------------------------------------

Reviews
------------

Login as a reviewer again to have a look.  I see that the conflict
declarations were saved but I am now uncertified and there is no
button to save conflict declarations.  If I make a change the button
appears but then I get an error message that I have already certified.

---------------------------
User: Dana Balser |
---------------------------

Reviews
-----------

1. Declare conflicts: select available for all but 007 which is auto
   conflict.
2. Click "Certify Conflict Declarations".  I can now see the review
   assignments.  I only see proposal details for 007, the proposal for
   which I am conflicted, which is the opposite of what I expected.

Dana (Firefox, Fedora)

--------------------------------- 
User: Dana Balser_TTA |
---------------------------------

Solicitations
-----------------

First create a solicitation and generate proposals.

1. Click "Generate Test Solicitation" button.
2. Enter Solicitation Name = "dsbTest48", Proposal Code Prefix =
   "dsb48", Proposal Count = 10, PPR.  I see the Solicitation is
   created and receive the 10 proposal submission emails.
3. Click "Close" button.

Proposals
--------------

1. Select "dsbTest48" Solicitation filter.  I can see the proposals
   went from "Submitted" to "In Review" from step (2) to step (3)
   above. Looks good.

Reviews
------------

Now configure the reviews / vet science categories.  I notice here
that the Solicitation name is now all in caps; we should probably
retain case.