Prototype Consensus, Finish out ISRs and hook up BE/FE ISRs + Conflicts
Feb 28
Sprint 49
Getting issues...
Due March 14: Finish hooking up ISRs (Completed/Finalized), Initiate Consensus, move Review Configuration into Solicitation tab, make views possible in UI for different roles.
Sprint 50
Getting issues...
Initiate Consensus, see (SRP) scores as Conflicted and Non Conflicted SRP member, notification on finalized ISRs and less than 3 non conflicted reviewers.
Sprint 51
Getting issues...
Consensus Comments and consensus related Scores, ability to export a file for ISR comments and score
one email per reviewer per change is sent out to the mailing list (tta-notifications-test)
NOT IMPLEMENTED
notifications to tta member/ srp chair are blocked until Identity management system is up
REVIEW
The ticket says that "Conflict Reason, and updated Conflict State" should be included in email but it is currently not
right now, there is a bulk update on conflict certification on the front end, so there are emails for every proposal instead of just the ones that were changed.
Notification when there are less than N available reviews for a proposal.
When a reviewer certifies conflicts, all of the proposals that do not have at least N available or unknown reviewers per proposal are sent in a single email.
The list in the email will have all of the proposals affected. It is triggered when SRP members certify. It does not trigger when the auto conflicts are managed.
Create a simple solicitation with hardcoded Science Categories and Hardware. The Proposal Process can be specified. The number of proposals it creates with 1 of 3 Science categories can be specified. The names and information on the proposal is randomly generated.
Author Copy is viewable and modifiable when in Draft State and Submitted if the Solicitation is Open
Observatory Copy is generated when Proposal is Submitted
Author and Observatory Copies are both viewable if
the Solicitation is Open
the Proposal State is Submitted
Can edit Author Copy after a submit if solicitation open
Can resubmit if Solicitation is open and PPR
Cannot edit Observatory Copy ever
Cannot edit or view an Author Copy when proposal is In Review
Can view only Observatory Copy when Proposal State is In Review
A Proposal with a Proposal State of Submitted should not be able to be deleted.
As TTA member:
Can edit Author Copy when the Proposal has any State (only up to In Review possible to test though).
Can Save and it persists. Should not affect Obs Copy if one exists.
Can Submit it only when call is open.
BUG
Can edit the Observatory Copy, if one exists but save applies to Author Copy only. There is no way to submit, so changes do not affect downstream (review phase) information.
Can withdraw a proposal in the Hidden State.
Withdrawn proposal still shows up in Review phase.
A Hidden proposal cannot be submitted, regardless of the Solicitation being open or not. It should be allowed to be submitted at anytime.
REVIEW
navigating between author and observatory copy is not intuitive
Information in Observatory Copy is not displaying correctly or ever in OS case
BUG
Saving changes to author copy shows a warning on the list of proposals and allows resubmission only after navigating away and then back again. The button is not always available when it should be.
Submitting a proposal works but the view of the proposal is missing afterwards and until a refresh is done.
Proposal State Changes
Allowed:
Draft → Submitted
Draft → Hidden
Submitted → Withdrawn
Submitted → In Review
In Review → Withdrawn
In Review → Completed
Hidden → Submitted (TTA only)
Not Allowed:
(Submitted, In Review, Completed, Closed) → Hidden
Hidden → Withdrawn
Sprint 42
Sprint
Sprint 42
WORKING
submission either goes to submitted or in review (DDT); Proposal State change to Completed is out of scope for v0.2
Withdrawing a proposal works: the author can see the Observatory Copy but not edit either it or the author copy. Cannot even access the author copy. Tooltip shows the action is not available.
BUG
can delete a proposal which causes the state to change to hidden; however, author can see the Observatory Copy!
Draft → Withdrawn is currently allowed but it should not be.
REVIEW
Withdraw button is only available in the edit mode of a proposal for the TTA member; it should be higher.
Print a single or multiple proposals
A user can print a proposal or more
the science justification is included in the view
all of the components of the proposal are concluded in the view (need definition)
There is an interface with which to vet Science Categories
Can see the Title, Abstract, and Requested Science Category per proposal
the Vetted SC = the Requested SC at first, but it can be changed.
There is a place to record notes; they persist.
The proposal can be opened for viewing
There is a check box per proposal to indicate it has been vetted; there is a "vet all" button.
Changes persist
proposals can be assigned to panels
Hidden proposals are not shown in Vetting; i.e., only those with Proposal States of In Review
TO FIX
Drop down menu on available Science Categories should either be linked to Solicitations or the drop down menu on the available science categories on the vetting should be linked to the master list.
TTA Member can access vetting comments after Configuration in Finalized and forever more.
PPRs - Review Configuration
Description
JIRA
Implemented
Last Working
Tested
Status
Panels
Create SRP panels with unique names
Delete SRP Panels
Associate panels with 1 or more Science Categories (edge cases)
Assign 1 or more SRP Chairs
Assign multiple unique SRP Members to panel
SRP members can only belong to one panel
Auto bulk assign proposals to panels based on SC (in vetting UI)
Manually assign proposals to panels (in vetting UI)
Changes to panel persist
Feasibility groups (back burner)
Upload and use a configuration file
Filtering on Panels/proposals
Can delete reviewers
Guidance Text to tell TTA member missing components necessary to Finalize (vet all, assign panels, create panels per SC, chair assigned to panel)
Automatic proposal to panel assignment based on Science Category
Delete reviewers
SRP members can only belong to one panel
Assign multiple chairs
NOT IMPLEMENTED
feasibility groups
TO FIX
Drop down menu on available Science Categories should either be linked to Solicitations or the drop down menu on the available science categories on the vetting should be linked to the master list.
Finalize Review Configuration
Finalize can occur when
all proposals are vetted
the Solicitation is Closed
In line prompt to confirm Finalization
Finalizing should prevent future edits to configuration.
Can update Conflict state and Certify (preferable that it is done after user certifies)
Conflicted ISRs do not go to Consensus
PPRs - Review Type Assignments
Description
JIRA
Implemented
Last Working
Tested
Status
SRP Chair
can see a "matrix" of Reviewers in their panel, their Conflict States, the Review States of their ISRs
can assign Review Type per ISR
Only 1 Primary, 1 Secondary, * Tertiary, * None per Proposal
Default is None
Can change Review Types unless ISR is Finalized
Cannot be able to assign types of Conflicted reviewers
Can change Review Type of External Reviewer
External Reviewer's name is redacted but can see the Review Type and Review State for the external ISR
Can upload a configuration file
Proposal ID, Proposal Title, Pop-out to Proposal
Sprint 46/47
Sprint 61
V0.2
Sprint 61
V0.2
WORKING
can see a "matrix" of Reviewers in their panel and their Conflict State
Only one Primary and one Secondary can be saved to a proposal.
Any number of None or Teritary can be saved to a proposal.
Cannot assign Review Types to a conflicted reviewer
Can change Review Type of External Reviewer
External Reviewer's name is redacted but can see the Review Type and Review State for the external ISR
NOT IMPLEMENTED
Can upload a configuration file
TTA Member
Can change any Review Type even if the Review State is Finalized until Consensus starts
Can change the Review Type of external ISRs until Consensus of the hosting panel begins
Can upload a configuration file
Sprint 45/47
Sprint 61
V0.2
Sprint 61
V0.2
WORKING
Can change RT of external ISR as long as Consensus for the hosting panel has not started
Can change RT of any ISR even in Finalized State until the start of Consensus
NOT IMPLEMENTED
Can upload a configuration file
PPRs - ISRs
Description
JIRA
Implemented
Last Working
Tested
Status
SRP Reviewer
See list of proposals (after conflict certification)
Review Type is displayed
Default is None; otherwise P,S,T
Review State is displayed
default is Blank; otherwise Saved, Completed, Finalized, Closed
Conflict State is displayed
Conflict/Available
If not Conflicted, have an ISR in which to enter comments and a score and see the proposal
can Save (Review State → Saved) per ISR
validate if score not equal to 0; otherwise it cannot save
does not validate comments
can Complete (Review State → Completed) per ISR
can Finalize (bulk action) if any ISRs with Review not equal to None are not Closed or Finalized
If Conflicted, only see title, abstract, proposal ID and cannot enter any information
All comments and scores are saved (though not propagated to consensus; see later)
Normalized Individual Scores are calculated on Finalize
Cannot provide comments or scores for ISRs with a Review State of Closed
Edge cases:
If an external proposal is added after reviewer has finalized, they should be able to review and finalize that one proposal (finalize is still bulk in theory)
If consensus has started for their panel, the Finalized Normalized Individual Score in that panel may be inconsistent with their new Normalized Individual Score (since it has the extra proposal). This is acceptable.
Instructions present
Finalize Reviews button always there but not active
See list of proposals (after conflict certification)
Review Type is displayed
Default is None; otherwise P,S,T
Review State is displayed
default is Blank; otherwise Saved, Completed, Finalized, Closed
Conflict State is displayed
Conflict/Available
If not Conflicted, have an ISR in which to enter comments and a score and see the proposal
can Save (Review State → Saved) per ISR
validate if score not equal to 0; otherwise it cannot save
does not validate comments
can Complete (Review State → Completed) per ISR
can Finalize (bulk action) if any ISRs with Review not equal to None are not Closed or Finalized
If Conflicted, only see title, abstract, proposal ID and cannot enter any information
All comments and scores are saved (though not propagated to consensus; see later)
Normalized Individual Scores are calculated on Finalize
If an external proposal is added after reviewer has finalized, they should be able to review and finalize that one proposal (finalize is still bulk in theory)
If consensus has started for their panel, the Finalized Normalized Individual Score in that panel may be inconsistent with their new Normalized Individual Score (since it has the extra proposal). This is acceptable.
Cannot provide comments or scores for ISRs with a Review State of Closed
Instructions present
Finalize Reviews button always there but not active
TO FIX
External Reviewer
External Reviewer is someone who is on any one panel associated with the Solicitation in question
An external reviewer can be assigned an external ISR for any proposal as long as the Panel associated with that proposal has not started Consensus
An external reviewer can be assigned an ISR even if their host panel is in Consensus.
External Reviewers do not have to declare conflicts on external ISRs. When assigned, the conflict state is automatically set to Available
External Reviewers can see the external ISR in their list of ISRs.
ER can provide comments, scores, save, complete, and finalize their external ISRs
ER do not have access to the Consensus UI or any responsibility at Consensus for the panel associated with the external ISR.
ER scores and comment are available in Consensus and corretly integrated
External Reviewer is someone who is on any one panel associated with the Solicitation in question
An external reviewer can be assigned an external ISR for any proposal as long as the Panel associated with that proposal has not started Consensus
External Reviewers do not have to declare conflicts on external ISRs. When assigned, the conflict state is automatically set to Available
ER do not have access to the Consensus UI or any responsibility at Consensus for the panel associated with the external ISR.
ER scores and comment are available in Consensus and corretly integrated
External Reviewers can see the external ISR in their list of ISRs.
ER can provide comments, scores, save, complete, and finalize their external ISRs
An external reviewer can be assigned an ISR even if their host panel is in Consensus.
TO FIX
Import/Export ISRs
a CSV file is expected but it can use as a delimiter any of the following:
, ; | tab
putting double quotes around the comment will keep the commas within the comment safe
if reviews exist when asked to export, it will export the reviews
cannot upload a file with any blank fields (fails and warning issued)
if a user changes the name of the reviewer within the file, it gives an error
protects against Bobby Drop Tables
on a successful upload, the Review State of the ISR will changed to Saved
reuploading will update the comments/score
Don't need every ISR in file to work
Sprint 57?
Sprint 61
V0.2
Sprint 61
V0.2
WORKING
TTA Member
can change Review State to Closed
Auto changes Review Type to None
If a TTA member Finalizes a SRP member's ISRs (for the reviewer), it will finalize the ISRs that have a valid comment and score (Review State of Saved, Completed) and Review Type not equal to None. If a comment+score is not valid, regardless of the Review Type, it will Close the ISR (unsure if it still ignores review types of None)
can change Review Type until Consensus is initated for the SRP member's panel.
can change Review State until Consensus is initiated for the SRP member's panel.
When TTA member uses Finalize, as a bulk action for all ISRs for that reviewer
Review Types of Primary, Secondary, Tertiary but have Review State of Blank are set to Close
Review Type of None is ignored, regardless of Review State
Review Types of P,S,T that have Review States in (Saved, Completed) are transitioned to Finalized
an change Review State to Closed
Auto changes Review Type to None
If a TTA member Finalizes a SRP member's ISRs (for the reviewer), it will finalize the ISRs that have a valid comment and score (Review State of Saved, Completed) and Review Type not equal to None. If a comment+score is not valid, regardless of the Review Type, it will Close the ISR (unsure if it still ignores review types of None)
When TTA member uses Finalize, as a bulk action for all ISRs for that reviewer
Review Types of Primary, Secondary, Tertiary but have Review State of Blank are set to Close
Review Type of None is ignored, regardless of Review State
Review Types of P,S,T that have Review States in (Saved, Completed) are transitioned to Finalized
can change Review Type until Consensus is initated for the SRP member's panel.
can change Review State until Consensus is initiated for the SRP member's panel.
NOT IMPLEMENTED
A separate Close functionality will be per ISR
TO FIX
Adding external reviewers as TTA member
If an external reviewer is added, the Conflict state is automatically Available.
TTA member can change Review Type until Consensus starts
Sprint 59
Sprint 61
V0.2
Sprint 61
V0.2
PPRs - Consensus
Description
JIRA
Implemented
Last Working
Tested
Status
SRP Member
As a conflicted SRP member, only see the SRP scope, proposal ID, title, abstract, scores, and Rank
As a non-conflicted SRP member, I want to see the comments, scores, and proposals for which I am not conflicted
Comments are not connected to reviewers
Normalized Individual Score
Finalized Normalized Individual Score
Mean Normalized Score
Standard Deviation of the Mean Normalized Score
SRP Score
Rank
As a non-conflicted SRP member and with a Review Type of Primary or Secondary, I want to enter consensus comments.
All members that are not conflicted can view Consensus comments.
Be able to see the updated SRP score, if it is changed by the chair or via changes to the Finalized Normalized Individual Score. Push only for now.
See Feasibility Reviews (on hold)
As a Primary or Secondary, I want to complete a consensus comment
Can enter comments for PI, Technical, Data Management + an internal for each
Score
Needs to be Binary
Can Save
Can Finalize which "dangles" but next step is out of scope. It does change the review state to Finalized, as expected. Proposal State doesn't change yet (not in scope).
Create 1 to 3 Allocation Requests and randomly select the Facility from the ones defined in the Solicitation.
restricted to VLA, GBT
Capability Request per Allocation Request
Per Allocation Request, create 2 to 10 Capability Requests. The Capability should be randomly selected from what is available for the Facility in the Solicitation.
restricted to VLA Continuum, GBT Spectral Line
Field Sources per Capability Request
Per Capability Request, randomly select 1 Test Field Sources available to the Capability.
Use calibratorList.json to select 1 field source + spectral spec pairs per Capability Request
If VLA Continuum
set other parameters to 0 if not specified in json
If GBT Spectral Line
set Peak Line Flux Density to 10% of peak continuum flux density
set Line Width to 50 km/s
set other parameters to 0 if not specified in json
Spectral Specifications per Capability Request
Per Capability Request, randomly select between 1 Test Spectral Specifications available to the Capability.
see above for center frequency
If VLA Continuum
set spectral resolution 1000 km/s
set bandwidth to 2000 km/s
If GBT Spectral Line
set spectral resolution to 10 km/s
set bandwidth to 300 km/s
Performance parameters per Capability Request
Per Capability Request, set Test Performance Parameters
If VLA Continuum:
randomly select a desired VLA Configuration from the ones available in the Capability (A, B, C, D), once a config is selected B_max and B_min is defined
B_max = (36.4, 11.1, 3.4, 1.03) km
B_min = (0.68,0.21,0.035,0.035) km
calculate Angular Resolution as θ = 1.2(c_km/freq_Hz)/B_max where c is the speed of light = 2.998e5 km/s
calculate largest angular scale as θ = (c_km/freq_Hz)/B_min where
set RMS sensitivity as 10% of the peak continuum flux density
If GBT Spectral Line
calculate the Angular Resolution as θ = 1.2(c_km/freq_Hz)/D where D = 0.1 km
set RMS Sensitivity as 30% of the peak line flux density
Calibration parameters per Capability Request
Per Capability Request, randomly select a Test Calibration Parameter
If VLA Continuum:
randomly set Polarization Calibration boolean to True
If GBT Spectral Line:
randomly set Flux Density Calibration to True
randomly set Test Source to True
randomly set Polarization Calibration to True
Version 0.3 Outline
Description
JIRA
Slice 1 - v0.2.2
Slice 2 v0.2.2
Scheduler
Source Conflict identification
Scheduler Tools
Construct Preliminary Allocation Dispositions (priorities and time)
the number of proposals submitted, approved (priority A, B, F), filler (C), rejected (N), and oversubscription (submitted/approved); and by proposal hours: the requested time, the available time, the approved time (priority A, B, F), filler time (C), rejected time (N), and the pressure (requested hours/available hours)
csv with
ALLOCATION REQUEST ID, PRINCIPAL INVESTIGATOR, NORMALIZED LINEAR-RANK SCORE, REQUESTED TIME, and APPROVED TIME (broken down by semester) for each SCHEDULING PRIORITY (A, B, C [filler], F [fixed], and N [rejected])
Approve each OSD
Deploy the Recommended Allocation Dispositions for Director's Review
Update/Test a solicitation when a different one is Open and has submitted proposals
NOT IMPLEMENTED
Solicitation Tab
Sprint 37.3
Sprint 37.3
Only available to TTA Member
Navigation
Description
JIRA/Behavior
Implemented
Last Working
Tested
Status
As an Author, I would like to view my proposals.
in a list
in a detailed view (Card)
<Sprint 29
Sprint 37
Sprint 37
WORKING
Saving within a CR
Capability Requests have their own save button. Though information looks like it is kept on the frontend, it is not in the backend without hitting save.
Change CR1 → don't save → navigate to CR2, change CR2 → save
Only changes to CR2 are kept
CR1 changes are not saved
Change CR1 → don't save → navigate to CR2, change CR2 → don't save → navigate to CR2 (changes are still present in front end) → save
Changes to CR1 are persisted
Changes to CR2 are not persisted
A refresh without hitting save does not persist work
