Observations toolbox and survey - updated 8/9/2024
Whitney Weber
started a topic
over 4 years ago
This topic will be used to report updates to the Observations survey and toolbox. Follow the topic to receive announcements when new versions are released. For more information regarding the toolbox, survey, and Biodiversity Observation Data Standard, see this Solution.
NEWSFLASH: NatureServe can now clone the (Observation, QC, and paged version) surveys and related materials (dashboards, map, app) from the NatureServe AGOL Organization to yours. See this solution for details. This saves tons of time and allows you to hit the ground running - collecting data in the field the same day!
updated 5/22/2023 to fix bug preventing Lifestage, Count, Sex group from displaying appropriately
updated 8/2024 - minor clarification
Collect Data via Survey123.docx - updated 8/22/2022 to include information for using offline basemaps
Survey (v 1.4)
Updated to Survey123 v 3.17 - 5/22/2023
Fixed bug preventing Lifestage, Count, Sex group from displaying appropriately - 5/22/2023
Location Use Class not displayed if not applicable - previously showing for all animals, even when there was no LUC value to select - 7/11/2022
Error displayed for duplicate Associated Species, preventing saving. - 7/11/2022
Available via web app
Increased performance and ease of updating data lists
Option to sketch or use vertex draw methods for lines and polygons
Detected & Dataset fields added to reflect Biotics
QC fields added for use with Dashboard
Removed Dataset and Reference sections as they are now accommodated within the Toolbox
Added fields: elevation (auto-calculated for automatically recorded point location, editable if point location manually edited), specimen collected?, family (hidden), egt_uid (hidden) (updated 5/25/2022) - see this solution if you had a previous version of the survey for details on which fields to add.
Changed Museum/Herbarium field to select_one dropdown list from open text (updated 6/7/2022)
Specimen Labels (Labels_Observations.docm)
updated 6/7/2022 - altered to include Scientific Name author and use new County/Municipality and State/Province fields.
Toolbox (v 1.4)
Observations BLP Pro.pyt:
updated 11/29/2023: added Locational Uncertainty Distance Unit as a parameter, increased EO_ID field length, removed additional 1 m buffer
updated 10/9/2023: FIXED problem, caused by updated version of Pro, causing false report of success resulting in no fields/data being added.
Observations.atbx:
updated 8/9/2024:writing v_note data to orig_vnote field; populating v_note data with OBJECTID field for joining purposes - allows joining from Visit rather than SF (previously joined via DIG_COM) as can have multiple Visits per SF; updated SQL statements in Script 9 as dataset/global_id not appropriately updated in all instances.
updated 4/18/2024:accommodate possibility of not having ALL points, lines & polys; minor bug fixes, empty lists, and adding data to correct sf layer; accommodate use of Enterprise instead of AGOL as survey location.
1. Create GDBs and Files (updated 8/18/2022): provides option to link to the survey via AGOL directly via the logged in ArcGIS Pro user; if AGOL credentials are provided, a profile is created to store the credentials securely, for subsequent use.
Added 8/18/2022: 12. Update hosted Feature Layer (AGOL) with Biotics IDs (VISIT_ID, SOURCE_FEATUE_ID, EO_ID). This data within the hosted feature layer will clearly communicate whether the data has been loaded into Biotics and if so, with the specific record it is associated. This could be used to create a direct link to the Biotics record, which would of course prompt for a Biotics login.
Prompts for which geodatabase will be used to populate the ArcGIS Online (AGOL) feature layer: Observations.gdb for newly processed data or Observations_MASTER.gdb if you have existing data for which there are not IDs in the feature layer in AGOL already. NOTE: running the script using the Observations_MASTER.gdb as the source of data will take longer (more data) and hence should only be run once. Subsequent runs should use the Observations.gdb as only the new/edited data will need to be populated within the AGOL feature layer.
Adds new data to the feature layer in AGOL
For data which has been edited and hence there could be multiple records, the most recent data according to DESTINATION_DATE is used.
1. Create GDBs and Files (updated 2/28/2022): improved attachment downloads to include spreadsheets so can easily crosswalk attachments (of any type) to survey response
1. Create GDBs and Files (updated 5/25/2022): fixed bug when downloading attachments from zip files. Improved it to include spreadsheets for crosswalking attachments, as available when providing AGOL credentials. Also added calculation of county, state, and QRCode URL for records with specimen_collected = 'Yes'.
1. Create GDBs and Files (updated 6/7/2022): changed to feature service which includes US Counties/States and Canadian Municipalities/Provinces
3. Prep Data for Bulk Load Prep Toolbox (updated 5/25/2022): updated to export specimens_collected.csv containing records with specimen_collected = 'Yes', for use with Labels_Observations.docm to generate specimen labels.
Write an Observations_config.csv file to record inputs and minimize input prompts for user. (updated 3/19/2022)
Delete intermediate geodatabase created when downloading directly from ArcGIS Online (updated 3/30/2022)
Connect directly to Biotics to query for Visit/SF/EO IDs to:
identify updated Observations via parent/globalid
indicate IDs of resulting records created/modified in Biotics to provide link between Observation and Biotics record
Identify and generate SQL statements to add Visit record to an existing SF rather than create a new SF
Identify and populate Bulk Replace geodatabase with data which should replace an existing SF (spatial feature and/or tabular data) rather than create a new one
Populate destination table in Observations.gdb with Visit/SF/EO IDs & status (Included, Replaced, Modified)
Check Observations_MASTER.gdb for records with same globalid/parentglobalid as those in Observations.gdb. Deletes any such records in MASTER.gdb, assuming that edits have been made, to prevent duplication. Appends all observations to Observations_MASTER.gdb
9. Write SQL Statements: Auto-generate SQL statements to populate following fields (until BX-4414 implemented) - updated 8/9/2024 as some dataset/global_id was not updated when appropriate.
SOURCE_FEATURES, where SOURCE_FEATURE_ID IS NOT NULL:
- Data Sensitive (SOURCE_FEATURE.d_data_sensitive_id)
- Data Sensitive Category (SOURCE_FEATURE.d_data_sensitive_category_id)
- Data Sensitive Reasons (SOURCE_FEATURE.data_sensitive_reasons)
- Dataset (SOURCE_FEATURE.d_dataset_id) - when both REPLACE_SF AND REPLACE_VISIT BECAUSE ONLY VISIT FOR SF
- Global ID (SOURCE_FEATURE.global_id) - when both REPLACE_SF AND REPLACE_VISIT BECAUSE ONLY VISIT FOR SF
VISIT, where SOURCE_FEATURE_ID IS NOT NULL:
- Detected (Visit, once Detected field included in Biotics, see BX-2139)
- Dataset (VISIT.d_dataset_id)
- Global ID (VISIT.global_id)
- ALL VISIT FIELDS - when REPLACE_VISIT IS NOT NULL - Visit Date
EO, as appropriate:
- Data Sensitive (EO.d_data_sensitive_id)
- Data Sensitive Category (EO.d_data_sensitive_category_id
- Data Sensitive Reasons (EO.data_sensitive_reasons
- Condition of EO (EO.condition_of_eo)
- Confidence Extent (EO.d_confidence_extent_id
- Survey Date (EO.survey_date)
- Last Observation Date (EO.last_obs_date)
- First Observation Date (EO.first_obs_date
EO_REFERENCE, where EO_ID IS NOT NULL:
- Reference (EO_REFERENCE.reference_id)
- Primary Reference (EO_REFERENCE.eo_reference_primary_ind) , according to EO_REF_PRIM data
6. Create, Replace, Add - altered to prevent errors when not all of the feature classes (point, line, poly) had data - 6/22/2023
8. Populate Destination table - altered to prevent errors when certain fields were not included in the survey - 6/22/2023
Download an empty zipped Observations_v14 geodatabase, should you wish to import data from other sources into the geodatabase for use with the Observations toolbox, in order to Bulk Create, Bulk Replace, Add Visits of SFs/EOs in Biotics. This can be done using Append (ArcGIS Pro) (aka Esri's Simple Data Loader in ArcMap).
Whitney Weber
This topic will be used to report updates to the Observations survey and toolbox. Follow the topic to receive announcements when new versions are released. For more information regarding the toolbox, survey, and Biodiversity Observation Data Standard, see this Solution.
NEWSFLASH: NatureServe can now clone the (Observation, QC, and paged version) surveys and related materials (dashboards, map, app) from the NatureServe AGOL Organization to yours. See this solution for details. This saves tons of time and allows you to hit the ground running - collecting data in the field the same day!
Download/replace all files in the Observations.zip (v 1.4) file (updated 8/9/2024) to ensure you get all updated files. Following is a list of SOME of the changes:
Documentation:
Survey (v 1.4)
Specimen Labels (Labels_Observations.docm)
Toolbox (v 1.4)
- Data Sensitive (SOURCE_FEATURE.d_data_sensitive_id)
- Data Sensitive Category (SOURCE_FEATURE.d_data_sensitive_category_id)
- Data Sensitive Reasons (SOURCE_FEATURE.data_sensitive_reasons)
- Dataset (SOURCE_FEATURE.d_dataset_id) - when both REPLACE_SF AND REPLACE_VISIT BECAUSE ONLY VISIT FOR SF
- Global ID (SOURCE_FEATURE.global_id) - when both REPLACE_SF AND REPLACE_VISIT BECAUSE ONLY VISIT FOR SF
VISIT, where SOURCE_FEATURE_ID IS NOT NULL:
- Detected (Visit, once Detected field included in Biotics, see BX-2139)
- Dataset (VISIT.d_dataset_id)
- Global ID (VISIT.global_id)
- ALL VISIT FIELDS - when REPLACE_VISIT IS NOT NULL - Visit Date
EO, as appropriate:
- Data Sensitive (EO.d_data_sensitive_id)
- Data Sensitive Category (EO.d_data_sensitive_category_id
- Data Sensitive Reasons (EO.data_sensitive_reasons
- Condition of EO (EO.condition_of_eo)
- Confidence Extent (EO.d_confidence_extent_id
- Survey Date (EO.survey_date)
- Last Observation Date (EO.last_obs_date)
- First Observation Date (EO.first_obs_date
EO_REFERENCE, where EO_ID IS NOT NULL:
- Reference (EO_REFERENCE.reference_id)
- Primary Reference (EO_REFERENCE.eo_reference_primary_ind) , according to EO_REF_PRIM data
Download an empty zipped Observations_v14 geodatabase, should you wish to import data from other sources into the geodatabase for use with the Observations toolbox, in order to Bulk Create, Bulk Replace, Add Visits of SFs/EOs in Biotics. This can be done using Append (ArcGIS Pro) (aka Esri's Simple Data Loader in ArcMap).
3 people like this