Page History

• Date: //date
• Time: 9:00 ET
• Location: nraozoom04 
• Sprint Calendar

Attendance:

• Jeff Kern
• Dana Balser
• Mark Whitehead 
• Allison Costa
• Mark Ferguson 

Agenda:

• Supplemental Diagrams in Confluence now for reference:
  • https://open-confluence.nrao.edu/x/WwCtBg

• Is this actually automatic or does the TTA member initiate this?

  Jira
  server DMS JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId eb2e750b-a83a-387e-8345-36eee8a98f01 key STT-904

  • This should be automatic. The TTA member can edit Consensus Reviews until Disposition Letters are sent.
• Currently, do we use the Review State of Finalized/Closed to prevent edits to ISRs?

  Jira
  server DMS JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId eb2e750b-a83a-387e-8345-36eee8a98f01 key STT-879

  • (How) Do we prevent modification of ISRs with Review Type = None currently?
  • Suggestion: At initiation of Consensus sub-phase, all ISRs are immutable for the SRP member
  • We should disable the edit widget in the UI for SRP members once they Finalize but allow it if they get an external ISR.
    • E.g., SRP member finalizes. Primary Secondary and Tertiary ISRs now have a Review State of Finalized; they are not modifiable as a rule by SRP member. All of the Review Type = None have their edit widgets disabled.
    • If the SRP member gets an external ISR, it will have a Review Type of Primary Secondary or Tertiary but a Review State of Blank so they should be able to edit it. Trigger on Review Type and the presence of any Review State of Finalized.
      
    • As an SRP member, I want all of the Review Type = None ISRs to not be modifiable once I finalize my ISRs.

• What do we mean by "Reviews" in the UI currently? (Allie) -Skipped for now
  
  • Proposal Process
  • "Review Process" 

• Here is  is a link to some GBO DSS documentation to clarify the distinction between DSS and PHT.
  • We do want to subsume the responsibilities of the GBT PHT and (vla/vlba) PHT.
  • The DSS would provide an input for the available time model to our TTA tools (via the simulator) but this concept is similar to other needed inputs.
  • The TTA tools output to the DSS. This will keep a linear flow and separation between the DSS and the TTA tools.
  • Note, we recognize a need for a project completion tool, but it is not in scope for the TTA tools. Thus, we should focus on the import of the required data to build the available time model.

Further discussion on

• TTA member can modify ISRs (Review State, comments and score) up until Consensus starts for a panel
  • Note, this means they cannot Close an ISR once Consensus starts. We agree on this behavior.
    
• TTA member can change Review Types until the NLR is generated.
  • We decided, based on the discussion below on review states and review types, that TTA member should not be able to change Review Type after a Consensus panel is started. We had allowed this ability before hand because if the chair made a mistake, it was a way for a panel member to enter in consensus comments. We have decided that if the chair made a mistaken, they can enter the comments in lieu of a SRP member. If all else fails, the TTA member can enter the comments.
    
• TTA member can modify Consensus reviews (Review State, comments , score) up until Disposition Letters are sent. We agree on this behavior.
  

Consequences of changing ISR Review States and Review Types

Edited based on decision above restricting a TTA member's ability to modify review type in consensus.

Summary:

• The TTA member should always have access to the bulk Finalize button, regardless of the triggering criteria, in order to "fix" the NIS if an out of process action is taken. 

  Jira
  server DMS JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId eb2e750b-a83a-387e-8345-36eee8a98f01 key STT-1212

  was created to address this.
  
• A TTA member should avoid changing the individual Review State of an ISR to Finalized unless they REALLY understand the process. Finalizing should really only be done via the bulk action to ensure the NIS is properly calculated.
• A TTA member should avoid changing the Review Type from None if the Review State is Closed, as it would give the reviewer access to editing consensus comments.
• An additional criteria on the Review Type= P, S, or T has been to be added to 

  Jira
  server DMS JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId eb2e750b-a83a-387e-8345-36eee8a98f01 key STT-1166

  
  

Image Added

Recall:

• Finalizing ISRs is a per reviewer action. It changes the Review State to Finalized for To Finalize, the ISRs with Review type Type of  PrimaryPrimary, Secondary, or Tertiary must be in Review States of Saved or Completed.
  • For ISRs with Review Types of P, S, or T and Review States of Saved or Completed, the Finalize action changes the Review State to Finalized. 
  • If the Review Type is None, the Review State does not change (Mark - currently review state is changed to saved/completed anytime an update is made).
    
  • If the Review State is Finalized or Closed, finalizing does not change the ISR's state.
• Finalizing calculates the Normalized Individual Score (NIS) using all ISRs with . The implication is that a Review Type = None cannot have a Review State of Finalized . (Discussion for later if the TTA member intervenes with a manual change.) and a Review Type of P, S, or T.

  Jira
  server DMS JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId eb2e750b-a83a-387e-8345-36eee8a98f01 key STT-1166

• To Finalize, the ISRs with Review Type of Primary, Secondary, Tertiary must be in the Saved/Completed state.
• and 

  Jira
  server DMS JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId eb2e750b-a83a-387e-8345-36eee8a98f01 key STT-879

  .
  
  • ISRs with Review States of Finalized or Closed cannot be modified by SRP members
  Finalizing calculates the Normalized Individual Score (NIS) and prevents the SRP member from editing the ISRs. There is currently nothing stopping an SRP member from editing ISRs that have a Review State other than Finalized
  • .
• To start Consensus, all ISRs in a panel with Review Types of P,S, or T must have Review States State of Finalized.
• At the start of Consensus, the NIS per reviewer are copied into Finalized NIS.
• (Proposed) TTA member cannot change Review State after Consensus starts.

Use Case 1

A SRP member has Finalized their ISRs. It is prior to Consensus . They would like to make an edit.Reviews. One of their ISRs has

• a If the Review Type is None, then the .
• a Review State can only be of Blank/, Saved/Completed/Closed.
  (1) If the Review Type is not changed, then they are currently , or Completed (only states allowed if in process).

1. If the TTA member changes nothing,
   1. the SRP member is (as of Sprint 50) able to continue editing

1. 1. their ISR and save edits into perpetuity.
   2. Even though the comments and score won't go to Consensus, it is best to have a hard stop to their edits at some point.
   3. Proposed Fix: Initiating Consensus should prevent edits by SRP members to any ISRs

1. 1. associated with that panel. should disable the edit widget in the UI for SRP members once they Finalize but allow it editing for an external ISR, see discussion above.
2. If a TTA member only

1. changes the Review Type to P, S, or T

1. ,
   1. the SRP member can modify their review as needed. The Finalize button will trigger.
   2. The correct NIS's will propagate into Consensus and the TTA member cannot start Consensus until the SRP member finalizes (again). This is the outcome/system behavior we desire.

1. If the Review State is changed to Closed, then no further action is needed.
   1. This is the outcome/system behavior we desire because nothing actually change for Consensus.
2. If the TTA changes the Review State to Finalized
   1. The NIS does not include this Individual Score in its calculation. This may not be what the TTA member intended though!
   2. The ISR comments will go to the Consensus comments because the action to send to consensus is separate than NIS calculation. Going to change the criteria STT-1166 to include a Primary, Secondary, Tertiary condition for ISR comments going to Consensus.
   3. TTA member should avoid changing a Review State of a single ISR to Finalized. They should use the bulk Finalize action instead.

Use Case 2

A SRP member has Finalized their ISRs. It is prior to Consensus Reviews. One of their ISRs has

• a Review Type is always forced to None and further edits are prevented.
• (4) If the a Review State is changed from Closed, then 1 or 2 appliesof Closed.

1. If nothing changes,
   1. the SRP member cannot edit this ISR and it doesn't not go to Consensus.
2. If the Review Type

1. changes to P, S,

1. T

1. 1. the SRP member can edit Consensus Comments but yet their scores and comments would not affect consensus.
   2. This is perhaps a useful state in some ways but very out of process. TTA member beware.
   3. If it changes to Tertiary, SRP member cannot edit Consensus comments.
      1. TTA member should avoid changing the Review Type from None of a Closed ISR.
2. TTA member changes the Review State to Blank, Saved, or Completed
   1. the SRP can edit their ISRs but the changes do not go to Consensus or affect the NIS.
3. TTA member changes Review State to Finalized,
   1. Individual Score will never be included in future calculations of the NIS, which is perhaps not the intent so TTA member should beware!
      1. TTA member should avoid changing a Review State of a single ISR to Finalized. They should use the bulk Finalize action instead.
   2. ISR comments will go to Consensus because the criteria is different as of Sprint 50. Going to change the criteria STT-1166 to include a Primary, Secondary, Tertiary condition for ISR comments going to Consensus.

Use case 3

A SRP member has Finalized their ISRs. It is prior to Consensus Reviews. One of their ISRs has

• a Review Type is Primary, Secondary, or Tertiary.
  
• a Review State of Finalized (only one allowed in process)
  

1. No changes made
   1. SRP member cannot edit their ISR and NIS has been calculated properly. ISR comments go to Consensus.
2. TTA member only changes the Review Type between P, S, and T,
   1. no changes occur that affect ISRs, the NIS, or the ISR comments.
   2. It only changes the ability to modify to Consensus Reviews.
   3. This is the outcome/system behavior we desire.
3. TTA member only changes the Review State to Blank/Saved/Completed
   

1. 1. Consensus cannot be initiated until the SRP member elects to Finalize their ISRs again

1. 1. . Note, it does not matter if the Review Type is changed between P, S, or T. 
   2. The correct NIS's will propagate into Consensus and the SRP member has to Finalize again.
      

1. TTA member

1. only changes the Review Type to None:
   1. The previously calculated NIS does not reflect this change as there is currently not an automatic recalculation of the NIS.
   2. Furthermore, the Finalize button will never trigger

1. 1. for the SRP member, so there is no way for the SRP member to force the recalculation.
      1. TTA member should avoid changing a Review State of a single ISR to Finalized. They should use the bulk Finalize action instead.
   2. ISR comments will go to Consensus because the criteria is different as of Sprint 50. Going to change the criteria STT-1166 to include a Primary, Secondary, Tertiary condition for ISR comments going to Consensus.
      
2. TTA member changes the Review Type to None and the Review State to Blank, Saved, or Completed:
   1. The previously calculated NIS does not reflect this change as there is currently not an automatic recalculation of the NIS.
      
   2. Furthermore, the Finalize button will never trigger for the SRP member, so there is no way for the SRP member to force the recalculation.
   3. ISR comments will not go to Consensus.
      
   4. TTA member should generally always use the bulk finalize option after changing Review States or Review Types.
3. If a TTA member changes the Review State to Closed,
   1. the Review Type is always forced to None

1. 1. .
   2. Further edits are prevented

1. 1. because of the Closed state
   2. The previously calculated NIS does not reflect this change as there is currently not an automatic recalculation of the NIS.
      
   3. Furthermore, the Finalize button will never trigger for the SRP member, so there is no way for the SRP member to force the recalculation.
   4. ISR comments will not go to Consensus.
   5. TTA member should generally always use the bulk finalize option after changing Review States or Review Types.

The TTA member should generally always use the Finalize button after modifying the Review States or Review Types out of process to resolve inconsistencies in the existing NIS calculation.

1. Recall that if the TTA member uses the Finalize button for an SRP member:
   1. all of their ISRs with Review Type of P,S, or T with review states of Blank → Closed
   2. ISRs with Review Types of P, S, T and states of Saved/Completed → Finalized
   3. ISRs with Review Types of None have no state change
      
   4. ISRs with Review States of Finalized and Closed have no state change
      
   5. Only ISRs with Review States of Finalized and Review Types of P, S, or T are used to calculate NIS.

We decided that the TTA member should not be able to modify the Review Type after consensus has started. If an edit needs to be made to a consensus comment and only the chair or tta member can do it because of some outstanding issue, then this is an acceptable solution. We expect to rarely have to exercise it.

In If in Consensus, an ISR review state, comments, and score cannot be modified by anyone, but a TTA member is able to modify the Review Type.  At the start of Consensus, the NIS per reviewer is copied into the FNIS.

• Edge case 1
• The TTA member is prevented from Closing an ISR after Consensus starts.
• Changes to the Review Type really should only determine the SRP member's ability to enter Consensus Comments.
• Weird edge case:
  • SRP member A for Panel 1 Finalizes finalizes their ISRs. Their Consensus Meeting starts . The for Panel 1.
  • For a good reason, the TTA member changes one of their Review Types SRP member A's ISR's Review Type to None, which was previously had a Secondary. The ISR Review State = Finalized and Review Type = None. This is not allowed normally and should not matter to the Consensus Review.Type = Secondary. 
  • This change means that SRP member A cannot enter or complete Consensus Comments for that proposal. This change cannot affect the NIS or FNIS or any other score or availability of ISR comments because all of those objects are defined at the start of Consensus.
• Edge case 2:
  • The SRP member A is subsequently assigned an external ISR after edge case 1 occurs. They do their external ISR and hit finalize.
    
    • The ISR with a Review State of Finalized and a Review Type of None will be excluded from the calculation of the new NIS. The new NIS will be discrepant from the NIS shown in Panel A but this is acceptable
    If the SRP member is then assigned an external review from Panel 2. Should it matter to calculating the NIS that they now have a Review State = Finalized & Review Type = None for an ISR in Panel 1?
    • The only ISRs with Review State of Finalized go into calculating the Normalized Individual Score (NIS).

Open Actions: