This topic will be used to report updates to the Observations survey and toolbox. Follow the topic to receive announcements when new versions are released. For more information regarding the toolbox, survey, and Biodiversity Observation Data Standard, see this Solution.

NEWSFLASH: NatureServe  can now clone the (Observation, QC, and paged version) surveys and  related materials (dashboards, map, app) from the NatureServe AGOL  Organization to yours. See this solution for details. This saves tons of time and allows you to hit the ground running - collecting data in the field the same day!

Download/replace all files in the Observations.zip (v 1.4) file (updated 11/18/2024) to ensure you get all updated files. Following is a list of SOME of the changes:

Documentation:

• Observations_V14.docx - 
  • updated 10/17/2023
    
  • updated 5/22/2023 to fix bug preventing Lifestage, Count, Sex group from displaying appropriately
  • updated 8/2024 - minor clarification
  • updated 11/18/2024 - re: new Dataset input in tool 1
• Collect Data via Survey123.docx - updated 8/22/2022 to include information for using offline basemaps

Survey (v 1.4)

• Updated to Survey123 v 3.17 - 5/22/2023
• Fixed bug preventing Lifestage, Count, Sex group from displaying appropriately - 5/22/2023
• Location Use Class not displayed if not applicable - previously showing for all animals, even when there was no LUC value to select - 7/11/2022
• Error displayed for duplicate Associated Species, preventing saving. - 7/11/2022
• Available via web app
• Increased performance and ease of updating data lists
• Option to sketch or use vertex draw methods for lines and polygons
• Detected & Dataset fields added to reflect Biotics
• QC fields added for use with Dashboard
• Removed Dataset and Reference sections as they are now accommodated within the Toolbox
• Added fields: elevation (auto-calculated for automatically recorded point location, editable if point location manually edited), specimen collected?, family (hidden), egt_uid (hidden) (updated 5/25/2022) - see this solution if you had a previous version of the survey for details on which fields to add.
• Changed Museum/Herbarium field to select_one dropdown list from open text (updated 6/7/2022)
  

Specimen Labels (Labels_Observations.docm)

• updated 6/7/2022 - altered to include Scientific Name author and use new County/Municipality and State/Province fields.

Toolbox (v 1.4)

• Observations BLP Pro.pyt:
  • updated 11/18/2024: added Dataset input to Script 1 to allow further filtering. Changed from DIG_COM to v_note to record globalid as need to crosswalk back from Visit record (NOT SF) to the Observation record. v_note overwritten with orig_vnote data in subsequent SQL statements.
  • updated 11/29/2023: added Locational Uncertainty Distance Unit as a parameter, increased EO_ID field length, removed additional 1 m buffer
    
  • updated  10/9/2023: FIXED problem, caused by updated version of Pro, causing false report of success resulting in no fields/data being added.
    
• Observations.atbx: 
  • updated 8/9/2024: writing v_note data to orig_vnote field; populating v_note data with OBJECTID field for joining purposes - allows joining from Visit rather than SF (previously joined via DIG_COM) as can have multiple Visits per SF; updated SQL statements in Script 9 as dataset/global_id not appropriately updated in all instances.
  • updated 4/18/2024: accommodate possibility of not having ALL points, lines & polys; minor bug fixes, empty lists, and adding data to correct sf layer; accommodate use of Enterprise instead of AGOL as survey location.
    
• 1. Create GDBs and Files (updated  8/18/2022): provides option to link to the survey via AGOL directly via the logged in ArcGIS Pro user; if AGOL credentials are provided, a profile is created to store the credentials securely, for subsequent use.
  
• Added 8/18/2022: 12. Update hosted Feature Layer (AGOL) with Biotics IDs (VISIT_ID, SOURCE_FEATUE_ID, EO_ID). This data within the hosted feature layer will clearly communicate whether the data has been loaded into Biotics and if so, with the specific record it is associated. This could be used to create a direct link to the Biotics record, which would of course prompt for a Biotics login.
  • Prompts for which geodatabase will be used to populate the ArcGIS Online (AGOL) feature layer: Observations.gdb for newly processed data or Observations_MASTER.gdb if you have existing data for which there are not IDs in the feature layer in AGOL already. NOTE: running the script using the Observations_MASTER.gdb as the source of data will take longer (more data) and hence should only be run once. Subsequent runs should use the Observations.gdb as only the new/edited data will need to be populated within the AGOL feature layer.
  • Adds new data to the feature layer in AGOL
  • For data which has been edited and hence there could be multiple records, the most recent data according to DESTINATION_DATE is used.
    
• 1. Create GDBs and Files (updated 2/28/2022): improved attachment downloads to include spreadsheets so can easily crosswalk attachments (of any type) to survey response
• 1. Create GDBs and Files (updated 5/25/2022): fixed bug when downloading attachments from zip files. Improved it to include spreadsheets for crosswalking attachments, as available when providing AGOL credentials. Also added calculation of county, state, and QRCode URL for records with specimen_collected = 'Yes'.
• 1. Create GDBs and Files (updated 6/7/2022): changed to feature service which includes US Counties/States and Canadian Municipalities/Provinces
• 1. Create GDBs and Files (updated 11/18/2022): added Dataset input to allow for further filtering of data
• 3. Prep Data for Bulk Load Prep Toolbox (updated 5/25/2022): updated to export specimens_collected.csv containing records with specimen_collected = 'Yes', for use with Labels_Observations.docm to generate specimen labels.
• Write an Observations_config.csv file to record inputs and minimize input prompts for user. (updated 3/19/2022)
• Delete intermediate geodatabase created when downloading directly from ArcGIS Online (updated 3/30/2022)
• Connect directly to Biotics to query for Visit/SF/EO IDs to:
  • identify updated Observations via parent/globalid
  • indicate IDs of resulting records created/modified in Biotics to provide link between Observation and Biotics record
• Identify and generate SQL statements to add Visit record to an existing SF rather than create a new SF
• Identify and populate Bulk Replace geodatabase with data which should replace an existing SF (spatial feature and/or tabular data) rather than create a new one
• Populate destination table in Observations.gdb with Visit/SF/EO IDs & status (Included, Replaced, Modified)
• Check Observations_MASTER.gdb for records with same globalid/parentglobalid as those in Observations.gdb. Deletes any such records in MASTER.gdb, assuming that edits have been made, to prevent duplication. Appends all observations to Observations_MASTER.gdb
• 9. Write SQL Statements: Auto-generate SQL statements to populate following fields (until BX-4414 implemented) - updated 8/9/2024 as some dataset/global_id was not updated when appropriate.

  • SOURCE_FEATURES, where SOURCE_FEATURE_ID IS NOT NULL:- Data Sensitive (SOURCE_FEATURE.d_data_sensitive_id)
    - Data Sensitive Category (SOURCE_FEATURE.d_data_sensitive_category_id)
    - Data Sensitive Reasons (SOURCE_FEATURE.data_sensitive_reasons)
    - Dataset (SOURCE_FEATURE.d_dataset_id) - when both REPLACE_SF AND REPLACE_VISIT BECAUSE ONLY VISIT FOR SF
    - Global ID (SOURCE_FEATURE.global_id) - when both REPLACE_SF AND REPLACE_VISIT BECAUSE ONLY VISIT FOR SF
    
    VISIT, where SOURCE_FEATURE_ID IS NOT NULL:
    - Detected (Visit, once Detected field included in Biotics, see BX-2139)
    - Dataset (VISIT.d_dataset_id)
    - Global ID (VISIT.global_id)
    - ALL VISIT FIELDS - when REPLACE_VISIT IS NOT NULL - Visit Date
    
    EO, as appropriate:
    - Data Sensitive (EO.d_data_sensitive_id)
    - Data Sensitive Category (EO.d_data_sensitive_category_id
    - Data Sensitive Reasons (EO.data_sensitive_reasons
    - Condition of EO (EO.condition_of_eo)
    - Confidence Extent (EO.d_confidence_extent_id
    - Survey Date (EO.survey_date)
    - Last Observation Date (EO.last_obs_date)
    - First Observation Date (EO.first_obs_date
    EO_REFERENCE, where EO_ID IS NOT NULL:
    - Reference (EO_REFERENCE.reference_id)
    - Primary Reference (EO_REFERENCE.eo_reference_primary_ind) , according to EO_REF_PRIM data
    
    

• 6. Create, Replace, Add - altered to prevent errors when not all of the feature classes (point, line, poly) had data - 6/22/2023
• 8. Populate Destination table - altered to prevent errors when certain fields were not included in the survey - 6/22/2023

Download an empty zipped Observations_v14 geodatabase, should you wish to import data from other sources into the  geodatabase for use with the Observations toolbox, in order to Bulk  Create, Bulk Replace, Add Visits of SFs/EOs in Biotics. This can be done  using Append (ArcGIS Pro) (aka Esri's Simple Data Loader in ArcMap).