Feedback on Threat Calculator implementation in Biotics

Please use this forum to add comments and questions about the Biotics Threat Calculator
implementation introduced in yesterday's webinar. We would like all feedback by the end of the day on Monday, February 4.


Here are notes from the webinar along with detailed implementation proposals from NatureServe Central. We will update the implementation details if warranted by feedback from users.



Proposed Threat Calculator implementation in Biotics – Demo

The recording can be downloaded from Training Webinars transfer site or accessed via this link: https://register.gotowebinar.com/recording/6283836892622703106. The chat log is below.


Reminder that this was a demo of functionality not yet released. The plan is for it to go out in the next release (Feb. 27) after the feedback you've provided has been incorporated.

 

1. Individual threat impact calculation (calculates when scope and severity are entered)

 

a. How should application respond if existing Calculated Impact value is incorrect given the existing Scope, Severity, and Timing values? Note that any of the 4 fields could be incorrect; Impact is not necessarily the incorrectly entered value.

 

example: Scope and Severity calculate to Low rather than Medium

 

example: Scope and Severity are blank but Calculated Impact is populated

 

Reason this can happen: Until now, Calculated Impact was a manually editable field. So data could have been incorrectly entered or accidentally updated, or a batch update from Excel Rank Calculator could have been incorrect.


What we heard:

After some consideration of an automatic update of Calculated Impact, there seemed to be a consensus that an automatic update should not happen, especially given the fact that it could be Scope or Severity that are incorrect. Instead, there should be a warning with an explanation.

 

Request: Is it possible to create QC queries to find cases where previously entered calculated impact is incorrect for that scope/severity/timing? So we don’t have to wait to find these cases till next edit? Answer: Yes!

 

Suggested implementation specifics:

Display a Warning when the record is first opened, and insert red text above the grid explaining the problem. This would work similarly to our other warnings, as illustrated in this example unrelated to ranking or threats, but without the checkbox:


b. Is a Date Stored field needed?

What we heard: Nothing! So we will not add this. Rationale: Threats are evaluated as part of rank review and while we do not track dates for individual factor changes, we do track any factor changes in the rank factor date/author field.



2. Calculation of Overall Threat Impact (adjust as new individual threats are added to the grid)and Store as Calculated Threat button [note: field label will change to Stored Calculated Overall Threat Impact).


What we heard:

1. Some found the 3 overall threat impact fields confusing. NatureServe Central note: please look at the last comment in https://bioticssupport.natureserve.org/support/discussions/topics/322929 and see if this helps. We think putting a box around the Automatically Calculated and Stored Calculated fields, like we did for the rank calculations (see screenshot below) may help too.


2. Some thought Stored Calculated Overall Threat Impact should automatically update. NatureServe Central note: We think a better understanding of why there are 3 fields (see previous bullet) might help show why we are resistant to this idea.


3. Some wanted to change wording on the button to Accept, others disagreed. NatureServe Central note: we prefer to keep it consistent with rank calculation button

 

Suggested implementation specifics:

NatureServe would prefer to keep this as demoed, except with a box added around the Automatically Calculated and Stored Calculated fields, to keep it consistent with the rank calculation button and fields. For comparison, here is a rank calculation screenshot:



3. Rollup of lower level threats

If there is exactly one level 2 threat under a level 1 threat, its scope, severity, and timing values will automatically update in top-level threat record. The top-level threats can be edited independently; when this is done, the lower level threats are not updated. 

 

If there is exactly one level 3 threat under a level 2 threat, its scope, severity, and timing values will automatically update in in the level 2 threat record (and the level 1 threat record will likewise be updated if there is only one level 2 threat). 

 

a. How should application behave when there is more than one level 2 or level 3 threat under a top-level threat?

 

What we heard

1. Ideas for how to present on the screen that manual rollup is needed (NatureServe Central prefer options d or e):

a. Warning or pop-up reminder to check top level threat when lower level values change

b. Pop-up as soon as a second lower level threat is added

c. Remove values from top level threat when lower level values change 

d. Make top level red when lower level values change

 

Suggested implementation specifics:

We suggest a hybrid of above (if possible): a pop-up as soon as a second lower level threat is added, and make top level red when lower level values change.


b. How should application behave when a higher level threat is modified such that it no longer coincides with lower level threats? (Rollup works from lower level threats up, but not in reverse.)

 

Suggested implementation specifics:

We suggest a warning message: “Check for consistency among threat levels” upon the editing of a level 1 threat which has level 2,3, threat(s).



4. Level 1 Threats rows are the ones used in the Overall Threat calculation. Should they stand out in some way?

 

What we heard:

Yes! Highlight or bold 

 

Suggested implementation specifics:

We would like to proceed with bold, because highlighting used when user selects a row. We are hoping that this is feasible for the developers!



ITEMS NOT DIRECTLY RELATED TO this implementation:

is there a past webinar on implementation of other rank calculator elements: We have not found one yet. We'll post here if/when we do.

 

Tracking threats over time: A tracking table would be unwieldy. Audit log queries can be used instead. We will provide one.

 

EO Threats: use Data Manager forum to continue this discussion [not applicable to threats calculator implementation at element level]

 

Online Help Fix: 2nd paragraph in Scope, Severity and Timing needs to be removed, along with some additional edits.

 

Generation Length: may already be a JIRA ticket, but NatureServe Central Science staff are all in agreement that this should be added to CAG and maybe CAx tables. Data model changes not fully worked out yet.

 

Addition of threat check box to element reference grid: data model change that can be requested.  Note that references can and should be cited in the threat summary and added to the element references grid.

 

EORank Calculator: Geoff Hammerson and BC developed an Element Occurrence Viability Calculator for grizzly bear populations that can be shared with those who want it. It was modified from the Excel rank calculator.It uses Population Size, Threat, Isolation, and Trend to calculate an R(1-5) rank. R5 being Excellent Viability - Secure, to R1 being Probably Not Viable - Critically Imperiled.



Chat log

lisa st.hilaire
02:09:58 PM
Because everything else automatically calculates, it would   make sense if it automatically recalculated
lisa st.hilaire
02:10:05 PM
AND maybe also a popup saying that it did so
Jasa Holt
02:10:18 PM
We'd like a warning message.
Frank Price
02:10:21 PM
If it's wrong, overwrite or at least flag
Steve Young
02:10:27 PM
Why would it calculate incorrectly?
Rayo McCollough
02:10:38 PM
I'd say flag incorrect records (maybe highlighted in red)   so we can change it.
Lindsey Wise
02:10:51 PM
those errors would only be introduced with bulk uploads,   right? seems fine to auto-fix those in Biotics based on changing methodology.
Leah Ramsay
02:10:57 PM
Q1: Incorrect calculated: Automatic fix would be best.   Pop-up w/ explanation would help.
Karen Coleman
02:11:06 PM
Maybe qc queries? Then we don't have to wait until we find them individually...?
Robert Gottfried
02:11:43 PM
So I don't think that it should change the value   automatically because we might not recognize the change and not understand   why things are suddenly different.Warning message would be good.If it did automatically update we'd need a pop-up telling us that a   change was made.
Judith Ratcliffe
02:12:06 PM
Is it possible to have a location to document the changes   made over time - as with the Srank changes?
Steve Young
02:12:20 PM
Sorry, I am new to this but what do you mean they came in   incorrectly?
Amanda Treher
02:12:39 PM
warning.... could be data entry error in scope/severity or   caclulated impact....worth a glance
Frank Price
02:13:42 PM
changes are already in the audit log!
Robert Gottfried
02:14:07 PM
Uggh, seeing threat change over time like SRank changes   would be a mess. Hard pass.
Judith Ratcliffe
02:14:10 PM
We will likely use comments field then to record   changes...is there a character limit
Judith Ratcliffe
02:15:43 PM
good point about where the potential error may have arisen
Judith Ratcliffe
02:19:30 PM
it would be problematic if there were no scope severity   then the calculated would be changed to blank if it's automated, when it   would be important to keep the information in the calculated impact with a   warning
Steve Young
02:23:21 PM
How can you have a very high secondary threat and   negligible primary threat?
Steve Young
02:25:57 PM
Is the calculated threat field automatically populated or   do you have to always enter it manually?
Frank Price
02:25:58 PM
can you flag cases like this where higher level threat has   been edited so it's not in agreement with lower level?
Steve Young
02:26:59 PM
The stored one should be automatic then changed if needed.
lisa st.hilaire
02:27:07 PM
This isn't live yet, right?
Steve Young
02:27:49 PM
Less typing!
Steve Young
02:28:06 PM
Why confusing?
lisa st.hilaire
02:28:56 PM
I'm not seeing the automatically calculated threat impact,   nor the store calculated overal threat impact button.
Steve Young
02:29:55 PM
Could the value go in automatically and then change the   button to accept value?
lisa st.hilaire
02:30:08 PM
OH YES
lisa st.hilaire
02:30:11 PM
Oh, OK
lisa st.hilaire
02:30:57 PM
I think "accept value" is more intuitive than   "store as calculated threat"
lisa st.hilaire
02:31:18 PM
Maybe "accept value and store as calculated   value"
Lindsey Wise
02:31:48 PM
how the autocalculate and store value button is set up now   is fine. "accept value" is not the right label, that is better   applied to the assigned overall threat value.
Lindsey Wise
02:32:15 PM
(I agree w Kristin's comments)
lisa st.hilaire
02:32:16 PM
gotcha
Steve Young
02:32:30 PM
Three fields is very confusing!
Frank Price
02:32:43 PM
"Accept value" gives me no idea what will happen   if i click it, "Store" is better because that's what happens
Steve Young
02:35:21 PM
Could you explain again what the difference is between calculated   and assigned. What is a scenario that would have different values?
Leah Oliver
02:37:34 PM
Could Kristin add another lower level threat to Threat   Category 2, so we can see how this works?
Leah Ramsay
02:37:54 PM
Roll-up: There are incidences where there are more than   one secondary threat within a catagory and there needs to be thought given as   to whether the total is additive or not (often depends on the areas that the   lower level cover).
Leah Oliver
02:38:05 PM
ACK!
Steve Young
02:38:12 PM
Still makes more sense to have stored and assigned field   auto filled and then change if need be. Seems like changing it would not   happen very often.
lisa st.hilaire
02:38:20 PM
Can you overwrite the automatic rollup?
Leah Ramsay
02:38:23 PM
OOPS -never mind she explained!
lisa st.hilaire
02:39:37 PM
Maybe as soon as you grab a second secondary threat a   popup warning?
Robert Gottfried
02:40:06 PM
If there are multiple second leve threats the top level   threat should go blank and maybe a popup to remind that the top level must be   entered.
Jasa Holt
02:40:12 PM
with 2 or more level threats, the summation should be done   case by case based on expert opinion
Leah Ramsay
02:40:19 PM
Idea -maybe put in the worst case scenario first as the   roll-up
Judith Ratcliffe
02:40:28 PM
add a warning that the additional 2nd level has not been   auto calculated?
Steve Young
02:40:45 PM
How many total secondary threats are there? Would we have   to look at them each time?
Leah Ramsay
02:40:56 PM
then a pop-up to review
Frank Price
02:41:01 PM
top level should go blank when multiple lower entered
Robert Gottfried
02:41:53 PM
So third level rolls up to second, but multiple thirds   don't roll up to second, right?
Leah Oliver
02:43:06 PM
Will the warning about lower level threats not being   rolled up be implemented by the time this is released?
Karen Zyko
02:43:19 PM
after lower levels added, higher level turn red rather   than blank
Robert Gottfried
02:43:44 PM
Ooooh, I like that feature. Adding second and primary manually was   always a pain.
Steve Young
02:43:51 PM
Could you show the values of scope, severity and timing   slower?
Steve Young
02:46:45 PM
What does timing mean? Example?
Robert Gottfried
02:47:42 PM
Timing, is the threat occuring now, is it something   happening in the near future, or distant future.
Frank Price
02:48:24 PM
there's an error in this help - "...for the threat   with the greatest overall impact..." is now out of date, right?
Frank Price
02:49:19 PM
I think same help language in scope and severity too
Leah Ramsay
02:49:27 PM
Because generation time is so important in calculating   severity (and timing) could that be stored somewhere
Leah Oliver
02:50:16 PM
That information is typically put into the CAG, but I   would put that info in the comments in the threat grid.
Leah Ramsay
02:50:47 PM
it is critical in assessing the threats - so good
Steve Young
02:51:00 PM
Do you calculate scope by the percentage of EOs it   affects?
Leah Ramsay
02:51:27 PM
YES -I like the bold idea!
Robert Gottfried
02:51:50 PM
Yes
Amanda Treher
02:51:54 PM
you could
Karen Zyko
02:53:17 PM
is there a past webinar on implementation of other rank   calculator elements?
Leah Oliver
02:53:44 PM
One other question - will the top level threats be   highlighted or bolded in the release? That would help visually to check one's work.
Steve Young
02:54:08 PM
Would you have to calculate all the threats Could you   calculate the threat for each EO and then use those numbers to calculate the   element threat?
lisa st.hilaire
02:54:39 PM
OMG Whitney, go home and drink hot liquids, with alcohol!
Steve Young
02:54:41 PM
Is the grid availble for each EO?
Leah Oliver
02:54:53 PM
I like highlighting.
lisa st.hilaire
02:55:08 PM
Bold!
Steve Young
02:55:56 PM
That would be helpful to have it for each EO.
Lindsey Wise
02:56:18 PM
don't need a threat grid for EOs. there's a threat comment   field, that's good enough. plus threats factor into EO rank and EO rank   comments.
Leah Ramsay
02:56:23 PM
(the EO threat calculator has been modified for EOs by   Geoff Hammerson and BC for Grizzly Bears if others want to use that
Frank Price
02:56:24 PM
rolled up EO threats <> threats to whole element
Robert Gottfried
02:56:28 PM
This is element ranking not EO Ranking. For each EO would be uber-confusing.
Leah Ramsay
02:56:53 PM
oops EO RANK calculator soory
lisa st.hilaire
02:56:55 PM
agreed
Leah Ramsay
02:57:55 PM
yes -but it does use the threat calc
Steve Young
02:58:08 PM
Not uber confusing. We have always wanted to see trends of   threats over time and in more detail   and that is not possible.
Leah Ramsay
02:58:25 PM
References: Now the references used in threats would go   under "rank factors" which is fine but maybe having another column   for references associated with threats would be helpful. (lower priority!)
Steve Young
02:58:56 PM
How would you know when a threat appears and disappears   for an EO?
Lindsey Wise
03:00:20 PM
people could use the data manager forum on sharepoint to   continue the eo threats discussion
Peter Hazelton
03:00:45 PM
Is there a SQL script to upload the threat grid from the   Excel threat calculator?
Leah Ramsay
03:01:44 PM
Ask Debbie Webb in BC about uploading -she has been doing   some

 


Lisa, thank you for providing that screenshot with the word "Stored" added. We did mention that it would be included in the summary, but it was very subtle. We ran out of time to update the screenshot itself.

Wow! What a summary! Generally I agree with all of the suggested implementations noted above from NatureServe. I especially like the idea of the box around calculated threat impacts like for ranks. One suggestion, something Kristen mentioned yesterday, that might be helpful is if the second rank of the three had the word "stored" included, so Stored Calculated Overall Threat Impact.

image

Thank you for the webinar! I'm just getting into the rank calculator spreadsheet for the butterflies for ME and look forward to having this all in Biotics.


Login or Signup to post a comment