4. Exploring, Visualizing, and Plotting BioSCape Field and AVIRIS-NG Data#
4.1. BioSCape Data Skills Workshop: From the Field to the Image#
BioSCape, the Biodiversity Survey of the Cape, is NASA’s first biodiversity-focused airborne and field campaign that was conducted in South Africa in 2023. BioSCape’s primary objective is to study the structure, function, and composition of the region’s ecosystems, and how and why they are changing.
BioSCape’s airborne dataset is unprecedented, with AVIRIS-NG
, PRISM
, and HyTES
imaging spectrometers capturing spectral data across the UV, visible and infrared at high resolution and LVIS
acquiring coincident full-waveform lidar. BioSCape’s field dataset
is equally impressive, with 18 PI-led projects collecting data ranging from the diversity and phylogeny of plants, kelp and phytoplankton, eDNA, landscape acoustics, plant traits, blue carbon accounting, and more
This workshop will equip participants with the skills to find, subset, and visualize the various BioSCape field and airborne (imaging spectroscopy and full-waveform lidar) data sets. Participants will learn data skills through worked examples in terrestrial and aquatic ecosystems, including: wrangling lidar data, performing band math calculations, calculating spectral diversity metrics, machine learning and image classification, and mapping functional traits using partial least squares regression. The workshop format is a mix of expert talks and interactive coding notebooks and will be run through the BioSCape Cloud computing environment.
Date: October 9 - 11, 2024 Cape Town, South Africa
Host: NASA’s Oak Ridge National Laboratory Distributed Active Archive Center (ORNL DAAC), in close collaboration with BioSCape, the South African Environmental Observation Network (SAEON), the University of Wisconsin Madison (Phil Townsend), The Nature Conservancy (Glenn Moncrieff), the University of California Merced (Erin Hestir), the University of Cape Town (Jasper Slingsby), Jet Propulsion Laboratory (Kerry Cawse-Nicholson), and UNESCO.
Instructors:
In-person contributors: Anabelle Cardoso, Erin Hestir, Phil Townsend, Henry Frye, Glenn Moncrieff, Jasper Slingsby, Michele Thornton, Rupesh Shrestha
Virtual contributors: Kerry Cawse-Nicholson, Nico Stork, Kyle Kovach
Audience: This training is primarily intended for government natural resource management agency representatives and field technicians in South Africa, as well as local academics and students, especially those connected to the BioSCape Team.
4.2. Overview#
This tutorial will explore BioSCape Campaign airborne data
currently available on the BioSCape SMCE S3 storage as well as preliminary vegetation plot data
. Participants will learn to access and explore cloud-based storage files. Analysis will occur with combinations of data available on S3 storage as well as vector plot and polygon data brought to the Hub. Demonstrations will provide methods to locate AVIRIS-NG scenes associated with vegetation plot data. Vegetation spectral profiles will be extracted and graphed from vegeation plot locations.
4.2.1. Learning Objectives#
Mount BioSCape SMCE S3 BioSCape object storage and explore file content
Using BioSCape preliminary field data, model bringing your own data to a managed hub environment for visualization and analysis
Discover and visualize AVIRIS-NG imagery within vegetation plot locations
Examine and visualize AVIRIS-NG quick look and L2 Reflectance data using both GDAL and python libraries for raster analysis
Project plot locations to a local coordinate reference system
Extract and graph vegetation plot-level spectral profiles from AVIRIS-NG data
4.2.2. Load Python Modules#
import s3fs
import rioxarray
from os import path
import matplotlib.pyplot as plt
import rasterio
from os import path
import geopandas as gpd
import xarray as xr
import folium
from osgeo import gdal
import numpy as np
from pyproj import Proj
gdal.UseExceptions()
s3 = s3fs.S3FileSystem(anon=False)
4.3. Examine Field Data Sites: Vegetation Site Data#
4.3.1. Bringing your own data to the Cloud#
We know that researchers often need to bring their own data to the cloud. For example, local files of field data in vector formats (point, polygon).
On the BioSCape SMCE Hub, you can easily upload those files into your working area. Simply click the Upload Files
button at the top of the folder/file listing panel. You can also drag and drop folders into the file listing panel.
For the purpose of this Notebook, we’re using a shape file of Vegetation Polygons as well as point locations of the Vegetation Polygon Center for a few specific vegetation plots in the Cape Peninsula.
Vegetation Polygons (BioSCapeVegPolys2024_09_12.shp)
Vegetation Polygon Centers (Focal_Veg_Center_Plots.geojson)
4.3.2. Read, convert and examinge the plot data files#
Geopandas
makes working with geospatial data in python easier. GeoPandas extends the datatypes used by pandas to allow spatial operations on geometric types.
# read the BioSCapeVegPolys shape file and convert to geojson using geopandas
veg_polys_gdf = gpd.read_file('data/BioSCapeVegPolys2024_09_12.shp')
veg_polys_gdf.to_file('veg_polys_json.geojson', driver='GeoJSON')
veg_polys_json_gdf = gpd.read_file('veg_polys_json.geojson')
vegpolys_cen_new = gpd.read_file('data/Focal_Veg_Center_Plots.geojson')
^^^ You should see a new geojson files in your folder area^^^
# Take a look at the data in the Focal_Veg_Center_Plots file
vegpolys_cen_new
BScpPID | Region | Botanst | BtnsVDT | Name | BtnstSN | QltyFlg | LctnFlg | TwnsndN | PTPlot1 | PTPlot2 | PTPlot3 | PTPlot4 | TwnsndLn | TwnsndLt | geometry | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | T093 | CapePoint | Ross Turner | 2023-05-16 | CapePoint_93 | None | None | None | None | None | None | None | None | None | None | POINT (18.41285 -34.25000) |
1 | T081 | CapePoint | Ross Turner | 2023-05-24 | CapePoint_81 | None | None | None | None | None | None | None | None | None | None | POINT (18.44767 -34.28852) |
2 | T069 | capepoint | Doug Euston-Brown | 2023-09-27 | capepoint_69 | None | None | None | None | None | None | None | None | None | None | POINT (18.40220 -33.94848) |
3 | T195 | capepoint | Doug Euston-Brown | 2023-09-28 | capepoint_195 | None | None | None | None | pt1399 | None | None | None | None | None | POINT (18.41104 -33.96294) |
4 | T201 | capepoint | Doug Euston-Brown | 2023-09-28 | capepoint_201 | None | None | None | None | None | None | None | None | None | None | POINT (18.40614 -33.95935) |
# explore the BioSCapeVegPolys data
list(veg_polys_gdf.columns)
['BScpPID', 'Region', 'Botanst', 'Name', 'Dscrptn', 'DateTim', 'geometry']
veg_polys_gdf
BScpPID | Region | Botanst | Name | Dscrptn | DateTim | geometry | |
---|---|---|---|---|---|---|---|
0 | T171 | Cederberg | Ross Turner | Cederberg 171 polygon | None | 03 May 2023 at 13:12:34 | MULTIPOLYGON (((19.02606 -32.15087, 19.02607 -... |
1 | T242 | Cederberg | Ross Turner | Cederberg_242_polygon | None | 04 May 2023 at 09:01:35 | POLYGON ((19.03114 -32.15265, 19.03114 -32.152... |
2 | T289 | Cederberg | Ross Turner | Cederberg 289 polygon | None | 04 May 2023 at 12:27:18 | POLYGON ((18.98517 -32.13685, 18.98517 -32.136... |
3 | T243 | Cederberg | Ross Turner | Cederberg_243_polygon | None | 06 May 2023 at 11:14:39 | MULTIPOLYGON (((19.01828 -32.13943, 19.01828 -... |
4 | T169 | Cederberg | Ross Turner | Cederberg_169_new_polygon | None | 07 May 2023 at 09:56:45 | POLYGON ((18.94214 -32.14125, 18.94214 -32.141... |
... | ... | ... | ... | ... | ... | ... | ... |
189 | T015 | anysberg | Paul Emms | anysberg_T015_patch | None | 30 Oct 2023 at 16:29:27 | POLYGON ((20.76362 -33.44222, 20.76360 -33.442... |
190 | T004 | anysberg | Paul Emms | anysberg_T004_patch | None | 31 Oct 2023 at 13:30:32 | POLYGON ((20.67799 -33.47535, 20.67799 -33.475... |
191 | T106 | anysberg | Paul Emms | anysberg_T106_patch | None | 31 Oct 2023 at 14:54:12 | POLYGON ((20.61907 -33.47147, 20.61904 -33.471... |
192 | T108 | anysberg | Paul Emms | anysberg_T108_patch | None | 01 Nov 2023 at 10:47:50 | MULTIPOLYGON (((20.80748 -33.44389, 20.80748 -... |
193 | T104 | anysberg | Paul Emms | anysberg_T104_patch | None | 02 Nov 2023 at 16:29:57 | POLYGON ((20.56262 -33.47006, 20.56263 -33.470... |
194 rows × 7 columns
# examine the coordinate reference system of the polygon file
veg_polys_gdf.crs
<Geographic 2D CRS: EPSG:4326>
Name: WGS 84
Axis Info [ellipsoidal]:
- Lat[north]: Geodetic latitude (degree)
- Lon[east]: Geodetic longitude (degree)
Area of Use:
- name: World.
- bounds: (-180.0, -90.0, 180.0, 90.0)
Datum: World Geodetic System 1984 ensemble
- Ellipsoid: WGS 84
- Prime Meridian: Greenwich
4.4. Visualize data#
4.4.1. Folium
is a powerful Python library that creates interactive Leaflet maps. It allows visualization of data in Python on an interactive map#
Overlay geographic data like country borders or city boundaries from GeoJSON or TopoJSON files.
Folium integrates seamlessly with Jupyter Notebooks, making it easy to create and display maps in your data analysis workflow.
The
explore()
method returns a folium.Map object, which can also be passed directly (as you do with ax in plot()). You can then use folium functionality directly on the resulting map.
4.4.2. Let’s plot the vegetation polygons and focal plot centers with folium#
import folium
mapObj = folium.Map(location=[-33.95, 23.52], zoom_start=7, control_scale=True)
folium.GeoJson(veg_polys_gdf,
name="BioSCape Vegetation Polygons",
tooltip=folium.GeoJsonTooltip(fields=["BScpPID"]),
style_function=lambda x: {"fillOpacity": 0}).add_to(mapObj)
folium.GeoJson(vegpolys_cen_new,
name="BioSCape Polygon Centers",
marker=folium.Circle(radius=30, fill_color="red", fill_opacity=0.8, color="red", weight=1),
color="red").add_to(mapObj)
# create ESRI satellite base map
#esri = 'https://server.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer/tile/{z}/{y}/{x}'
#folium.TileLayer(tiles = esri, attr = 'Esri', name = 'Esri Satellite', overlay = False, control = True).add_to(mapObj)
folium.LayerControl().add_to(mapObj)
mapObj
4.5. Explore a subset of polygon files that are in the Cape Peninsula region#
4.5.1. We’ll focus on 5 Cape polygons that have dominant vegetation families for the focal plots as listed below.#
T081 = Proteaceae
T093 = Asteraceae
T201 = Ericaceae
T069 = Rosaceae
T195 = Restionaceae
4.5.1.1. Start by selecting BScpPID = T081#
veg_polys_gdf.loc[veg_polys_gdf['BScpPID'] == 'T081']
BScpPID | Region | Botanst | Name | Dscrptn | DateTim | geometry | |
---|---|---|---|---|---|---|---|
16 | T081 | CapePoint | Ross Turner | CapePoint_81_polygon | None | 24 May 2023 at 08:54:24 | POLYGON ((18.44865 -34.28924, 18.44865 -34.289... |
# Plot the T081 polygon and center together
T081poly = veg_polys_json_gdf.loc[veg_polys_json_gdf['BScpPID'] == 'T081']
T081cen = vegpolys_cen_new.loc[vegpolys_cen_new['BScpPID'] == 'T081']
fig, ax = plt.subplots(figsize=(7,7))
T081poly.plot(ax=ax)
T081cen.plot(ax=ax, color='red')
<Axes: >
4.5.1.2. Let’s assign variables of the latitude / longitude coordinates of the vegetation center plot in the T081 Polygon#
T081_point = vegpolys_cen_new.loc[vegpolys_cen_new['BScpPID'] == 'T081'].get_coordinates()
T081lon = T081_point.x.values[0]
T081lat = T081_point.y.values[0]
print('T081lon: ', T081lon)
print('T081lat: ', T081lat)
T081lon: 18.447671093155833
T081lat: -34.288518109266434
4.6. Demonstrate how to discover the AVIRIS-NG scenes associated with plot locations.#
AVIRIS-NG scenes of of interest for the Dominant Family Polygons are:
ang20231109t131923_019
ang20231109t133124_003
ang20231109t133124_013
4.6.1. AVIRIS-NG Flight Lines#
4.6.1.1. A geoJSON of the AVIRIS-NG Flight Line’s Scenes are available from the JPL BioSCape Data Portal#
This file has been exported and already made available in the BioSCape hub in the
data/
subfolder.
# read and plot the AVNG coverage file
AVNG_Coverage = gpd.read_file('data/ANG_Coverage.geojson', driver='GeoJSON')
AVNG_Coverage.plot()
<Axes: >
list(AVNG_Coverage.columns)
['LOC Download',
'LOC ORT Download',
'OBS Download',
'OBS ORT Download',
'Quicklook Download',
'RFL Download',
'RFL UNC Download',
'end_time',
'fid',
'scid',
'start_time',
'style',
'geometry']
# Let's drop some unnecessary columns that result in errors in Folium visualization
AVNG_scenes = AVNG_Coverage.drop(columns=['LOC Download', 'LOC ORT Download', 'OBS Download', 'OBS ORT Download', 'Quicklook Download', 'RFL Download', 'RFL UNC Download', 'end_time', 'start_time', 'style'])
4.6.1.2. Visualize the veg polygons, veg centers, and AVIRIS-NG Flight Lines#
import folium
mapObj = folium.Map(location=[-33.95, 23.52], zoom_start=7, control_scale=True)
folium.GeoJson(veg_polys_gdf,
name="BioSCape Vegetation Polygons",
tooltip=folium.GeoJsonTooltip(fields=["BScpPID"]),
style_function=lambda x: {"fillOpacity": 0}).add_to(mapObj)
folium.GeoJson(vegpolys_cen_new,
name="BioSCape Polygon Centers",
marker=folium.Circle(radius=10, fill_color="red", fill_opacity=0.4, color="red", weight=1),
color="red").add_to(mapObj)
folium.GeoJson(AVNG_scenes,
name="AVIRIS-NG Coverage",
color="red",
tooltip=folium.GeoJsonTooltip(fields=["scid"]),
style_function=lambda x: {"fillOpacity": 0}).add_to(mapObj)
# Uncomment to see folium display
# create ESRI satellite base map
#esri = 'https://server.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer/tile/{z}/{y}/{x}'
#folium.TileLayer(tiles = esri, attr = 'Esri', name = 'Esri Satellite', overlay = False, control = True).add_to(mapObj)
#folium.LayerControl().add_to(mapObj)
#mapObj
<folium.features.GeoJson at 0x7f4a713bc2f0>
4.6.1.3. We see that there are many AVIRIS-NG scenes in the BioSCape Campaign data.#
4.6.1.4. Using a spatial join, let’s discover which AVIRIS-NG scenes contain the T081 polygon#
AVNG_flight_scenes_T081_gdf = gpd.sjoin(veg_polys_gdf.loc[veg_polys_gdf['BScpPID'] == 'T081'], AVNG_scenes, how='inner', predicate='within')
AVNG_flight_scenes_T081_gdf.scid
16 ang20231109t131923_019
16 ang20231113t132348_002
Name: scid, dtype: object
# Here's the same method for the T195 vegetation polygon
AVNG_flight_scenes_T195_gdf = gpd.sjoin(veg_polys_gdf.loc[veg_polys_gdf['BScpPID'] == 'T195'], AVNG_scenes, how='inner', predicate='within')
AVNG_flight_scenes_T195_gdf.scid
85 ang20231109t133124_013
85 ang20231113t131251_008
85 ang20231126t095812_046
85 ang20231126t100838_008
Name: scid, dtype: object
AVNG_flight_scenes_T081_gdf
BScpPID | Region | Botanst | Name | Dscrptn | DateTim | geometry | index_right | fid | scid | |
---|---|---|---|---|---|---|---|---|---|---|
16 | T081 | CapePoint | Ross Turner | CapePoint_81_polygon | None | 24 May 2023 at 08:54:24 | POLYGON ((18.44865 -34.28924, 18.44865 -34.289... | 4187 | ang20231109t131923 | ang20231109t131923_019 |
16 | T081 | CapePoint | Ross Turner | CapePoint_81_polygon | None | 24 May 2023 at 08:54:24 | POLYGON ((18.44865 -34.28924, 18.44865 -34.289... | 5667 | ang20231113t132348 | ang20231113t132348_002 |
# Lets use loc and the scid to plot just the ang20231109t131923_019
AVNG_scenes.loc[AVNG_scenes['scid'] == 'ang20231109t131923_019'].plot()
<Axes: >
4.6.1.5. Now plot it all together with the identified AVIRIS-NG scene#
# and now plot it all together with the identified AVIRIS-NG scene
import folium
mapObj = folium.Map(location=[-33.95, 23.52], zoom_start=7, control_scale=True)
folium.GeoJson(veg_polys_gdf,
name="BioSCape Vegetation Polygons",
tooltip=folium.GeoJsonTooltip(fields=["BScpPID"]),
style_function=lambda x: {"fillOpacity": 0}).add_to(mapObj)
folium.GeoJson(vegpolys_cen_new,
name="BioSCape Polygon Centers",
marker=folium.Circle(radius=10, fill_color="red", fill_opacity=0.4, color="red", weight=1),
color="red").add_to(mapObj)
folium.GeoJson(AVNG_scenes,
name="AVIRIS-NG Coverage",
color="red",
tooltip=folium.GeoJsonTooltip(fields=["scid"]),
style_function=lambda x: {"fillOpacity": 0}).add_to(mapObj)
folium.GeoJson(AVNG_scenes.loc[AVNG_scenes['scid'] == 'ang20231109t131923_019'],
name="AVIRIS-NG Coverage",
color="green",
tooltip=folium.GeoJsonTooltip(fields=["scid"]),
style_function=lambda x: {"fillOpacity": 0}).add_to(mapObj)
# create ESRI satellite base map
esri = 'https://server.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer/tile/{z}/{y}/{x}'
folium.TileLayer(tiles = esri, attr = 'Esri', name = 'Esri Satellite', overlay = False, control = True).add_to(mapObj)
folium.LayerControl().add_to(mapObj)
mapObj