Exploring, Visualizing, and Plotting BioSCape Field and AVIRIS-NG Data

Contents

4. Exploring, Visualizing, and Plotting BioSCape Field and AVIRIS-NG Data#

4.1. BioSCape Data Skills Workshop: From the Field to the Image#

Bioscape

BioSCape, the Biodiversity Survey of the Cape, is NASA’s first biodiversity-focused airborne and field campaign that was conducted in South Africa in 2023. BioSCape’s primary objective is to study the structure, function, and composition of the region’s ecosystems, and how and why they are changing.

BioSCape’s airborne dataset is unprecedented, with AVIRIS-NG, PRISM, and HyTES imaging spectrometers capturing spectral data across the UV, visible and infrared at high resolution and LVIS acquiring coincident full-waveform lidar. BioSCape’s field dataset is equally impressive, with 18 PI-led projects collecting data ranging from the diversity and phylogeny of plants, kelp and phytoplankton, eDNA, landscape acoustics, plant traits, blue carbon accounting, and more

This workshop will equip participants with the skills to find, subset, and visualize the various BioSCape field and airborne (imaging spectroscopy and full-waveform lidar) data sets. Participants will learn data skills through worked examples in terrestrial and aquatic ecosystems, including: wrangling lidar data, performing band math calculations, calculating spectral diversity metrics, machine learning and image classification, and mapping functional traits using partial least squares regression. The workshop format is a mix of expert talks and interactive coding notebooks and will be run through the BioSCape Cloud computing environment.

Date: October 9 - 11, 2024 Cape Town, South Africa

Host: NASA’s Oak Ridge National Laboratory Distributed Active Archive Center (ORNL DAAC), in close collaboration with BioSCape, the South African Environmental Observation Network (SAEON), the University of Wisconsin Madison (Phil Townsend), The Nature Conservancy (Glenn Moncrieff), the University of California Merced (Erin Hestir), the University of Cape Town (Jasper Slingsby), Jet Propulsion Laboratory (Kerry Cawse-Nicholson), and UNESCO.

Instructors:

  • In-person contributors: Anabelle Cardoso, Erin Hestir, Phil Townsend, Henry Frye, Glenn Moncrieff, Jasper Slingsby, Michele Thornton, Rupesh Shrestha

  • Virtual contributors: Kerry Cawse-Nicholson, Nico Stork, Kyle Kovach

Audience: This training is primarily intended for government natural resource management agency representatives and field technicians in South Africa, as well as local academics and students, especially those connected to the BioSCape Team.

4.2. Overview#

This tutorial will explore BioSCape Campaign airborne data currently available on the BioSCape SMCE S3 storage as well as preliminary vegetation plot data. Participants will learn to access and explore cloud-based storage files. Analysis will occur with combinations of data available on S3 storage as well as vector plot and polygon data brought to the Hub. Demonstrations will provide methods to locate AVIRIS-NG scenes associated with vegetation plot data. Vegetation spectral profiles will be extracted and graphed from vegeation plot locations.

4.2.1. Learning Objectives#

  1. Mount BioSCape SMCE S3 BioSCape object storage and explore file content

  2. Using BioSCape preliminary field data, model bringing your own data to a managed hub environment for visualization and analysis

  3. Discover and visualize AVIRIS-NG imagery within vegetation plot locations

  4. Examine and visualize AVIRIS-NG quick look and L2 Reflectance data using both GDAL and python libraries for raster analysis

  5. Project plot locations to a local coordinate reference system

  6. Extract and graph vegetation plot-level spectral profiles from AVIRIS-NG data

4.2.2. Load Python Modules#

import s3fs
import rioxarray
from os import path
import matplotlib.pyplot as plt
import rasterio
from os import path
import geopandas as gpd
import xarray as xr
import folium
from osgeo import gdal
import numpy as np
from pyproj import Proj
gdal.UseExceptions()
s3 = s3fs.S3FileSystem(anon=False)

4.3. Examine Field Data Sites: Vegetation Site Data#

4.3.1. Bringing your own data to the Cloud#

We know that researchers often need to bring their own data to the cloud. For example, local files of field data in vector formats (point, polygon).

On the BioSCape SMCE Hub, you can easily upload those files into your working area. Simply click the Upload Files button at the top of the folder/file listing panel. You can also drag and drop folders into the file listing panel.

upload

For the purpose of this Notebook, we’re using a shape file of Vegetation Polygons as well as point locations of the Vegetation Polygon Center for a few specific vegetation plots in the Cape Peninsula.

  • Vegetation Polygons (BioSCapeVegPolys2024_09_12.shp)

  • Vegetation Polygon Centers (Focal_Veg_Center_Plots.geojson)

4.3.2. Read, convert and examinge the plot data files#

Geopandas makes working with geospatial data in python easier. GeoPandas extends the datatypes used by pandas to allow spatial operations on geometric types.

# read the BioSCapeVegPolys shape file and convert to geojson using geopandas

veg_polys_gdf = gpd.read_file('data/BioSCapeVegPolys2024_09_12.shp')
veg_polys_gdf.to_file('veg_polys_json.geojson', driver='GeoJSON')
veg_polys_json_gdf = gpd.read_file('veg_polys_json.geojson')

vegpolys_cen_new = gpd.read_file('data/Focal_Veg_Center_Plots.geojson')

^^^ You should see a new geojson files in your folder area^^^

# Take a look at the data in the Focal_Veg_Center_Plots file
vegpolys_cen_new
BScpPID Region Botanst BtnsVDT Name BtnstSN QltyFlg LctnFlg TwnsndN PTPlot1 PTPlot2 PTPlot3 PTPlot4 TwnsndLn TwnsndLt geometry
0 T093 CapePoint Ross Turner 2023-05-16 CapePoint_93 None None None None None None None None None None POINT (18.41285 -34.25000)
1 T081 CapePoint Ross Turner 2023-05-24 CapePoint_81 None None None None None None None None None None POINT (18.44767 -34.28852)
2 T069 capepoint Doug Euston-Brown 2023-09-27 capepoint_69 None None None None None None None None None None POINT (18.40220 -33.94848)
3 T195 capepoint Doug Euston-Brown 2023-09-28 capepoint_195 None None None None pt1399 None None None None None POINT (18.41104 -33.96294)
4 T201 capepoint Doug Euston-Brown 2023-09-28 capepoint_201 None None None None None None None None None None POINT (18.40614 -33.95935)
# explore the BioSCapeVegPolys data
list(veg_polys_gdf.columns)
['BScpPID', 'Region', 'Botanst', 'Name', 'Dscrptn', 'DateTim', 'geometry']
veg_polys_gdf
BScpPID Region Botanst Name Dscrptn DateTim geometry
0 T171 Cederberg Ross Turner Cederberg 171 polygon None 03 May 2023 at 13:12:34 MULTIPOLYGON (((19.02606 -32.15087, 19.02607 -...
1 T242 Cederberg Ross Turner Cederberg_242_polygon None 04 May 2023 at 09:01:35 POLYGON ((19.03114 -32.15265, 19.03114 -32.152...
2 T289 Cederberg Ross Turner Cederberg 289 polygon None 04 May 2023 at 12:27:18 POLYGON ((18.98517 -32.13685, 18.98517 -32.136...
3 T243 Cederberg Ross Turner Cederberg_243_polygon None 06 May 2023 at 11:14:39 MULTIPOLYGON (((19.01828 -32.13943, 19.01828 -...
4 T169 Cederberg Ross Turner Cederberg_169_new_polygon None 07 May 2023 at 09:56:45 POLYGON ((18.94214 -32.14125, 18.94214 -32.141...
... ... ... ... ... ... ... ...
189 T015 anysberg Paul Emms anysberg_T015_patch None 30 Oct 2023 at 16:29:27 POLYGON ((20.76362 -33.44222, 20.76360 -33.442...
190 T004 anysberg Paul Emms anysberg_T004_patch None 31 Oct 2023 at 13:30:32 POLYGON ((20.67799 -33.47535, 20.67799 -33.475...
191 T106 anysberg Paul Emms anysberg_T106_patch None 31 Oct 2023 at 14:54:12 POLYGON ((20.61907 -33.47147, 20.61904 -33.471...
192 T108 anysberg Paul Emms anysberg_T108_patch None 01 Nov 2023 at 10:47:50 MULTIPOLYGON (((20.80748 -33.44389, 20.80748 -...
193 T104 anysberg Paul Emms anysberg_T104_patch None 02 Nov 2023 at 16:29:57 POLYGON ((20.56262 -33.47006, 20.56263 -33.470...

194 rows × 7 columns

# examine the coordinate reference system of the polygon file
veg_polys_gdf.crs
<Geographic 2D CRS: EPSG:4326>
Name: WGS 84
Axis Info [ellipsoidal]:
- Lat[north]: Geodetic latitude (degree)
- Lon[east]: Geodetic longitude (degree)
Area of Use:
- name: World.
- bounds: (-180.0, -90.0, 180.0, 90.0)
Datum: World Geodetic System 1984 ensemble
- Ellipsoid: WGS 84
- Prime Meridian: Greenwich

4.4. Visualize data#

4.4.1. Folium is a powerful Python library that creates interactive Leaflet maps. It allows visualization of data in Python on an interactive map#

  • Overlay geographic data like country borders or city boundaries from GeoJSON or TopoJSON files.

  • Folium integrates seamlessly with Jupyter Notebooks, making it easy to create and display maps in your data analysis workflow.

  • The explore() method returns a folium.Map object, which can also be passed directly (as you do with ax in plot()). You can then use folium functionality directly on the resulting map.

4.4.2. Let’s plot the vegetation polygons and focal plot centers with folium#

import folium
mapObj = folium.Map(location=[-33.95, 23.52], zoom_start=7, control_scale=True)

folium.GeoJson(veg_polys_gdf,
               name="BioSCape Vegetation Polygons", 
               tooltip=folium.GeoJsonTooltip(fields=["BScpPID"]), 
               style_function=lambda x: {"fillOpacity": 0}).add_to(mapObj)

folium.GeoJson(vegpolys_cen_new, 
               name="BioSCape Polygon Centers", 
               marker=folium.Circle(radius=30, fill_color="red", fill_opacity=0.8, color="red", weight=1),
               color="red").add_to(mapObj)

# create ESRI satellite base map
#esri = 'https://server.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer/tile/{z}/{y}/{x}'
#folium.TileLayer(tiles = esri, attr = 'Esri', name = 'Esri Satellite', overlay = False, control = True).add_to(mapObj)
folium.LayerControl().add_to(mapObj)
mapObj
Make this Notebook Trusted to load map: File -> Trust Notebook

4.5. Explore a subset of polygon files that are in the Cape Peninsula region#

4.5.1. We’ll focus on 5 Cape polygons that have dominant vegetation families for the focal plots as listed below.#

  • T081 = Proteaceae

  • T093 = Asteraceae

  • T201 = Ericaceae

  • T069 = Rosaceae

  • T195 = Restionaceae

4.5.1.1. Start by selecting BScpPID = T081#

veg_polys_gdf.loc[veg_polys_gdf['BScpPID'] == 'T081']
BScpPID Region Botanst Name Dscrptn DateTim geometry
16 T081 CapePoint Ross Turner CapePoint_81_polygon None 24 May 2023 at 08:54:24 POLYGON ((18.44865 -34.28924, 18.44865 -34.289...
# Plot the T081 polygon and center together
T081poly = veg_polys_json_gdf.loc[veg_polys_json_gdf['BScpPID'] == 'T081']
T081cen = vegpolys_cen_new.loc[vegpolys_cen_new['BScpPID'] == 'T081']
fig, ax = plt.subplots(figsize=(7,7))
T081poly.plot(ax=ax)
T081cen.plot(ax=ax, color='red')
<Axes: >
../../_images/40f9cc73d9e3a1d22f26f1f0945ed2ab07d1bd85ca95ef2391dfbc63502ceb9c.png

4.5.1.2. Let’s assign variables of the latitude / longitude coordinates of the vegetation center plot in the T081 Polygon#

T081_point = vegpolys_cen_new.loc[vegpolys_cen_new['BScpPID'] == 'T081'].get_coordinates()
T081lon = T081_point.x.values[0]
T081lat = T081_point.y.values[0]
print('T081lon: ', T081lon)
print('T081lat: ', T081lat)
T081lon:  18.447671093155833
T081lat:  -34.288518109266434

4.6. Demonstrate how to discover the AVIRIS-NG scenes associated with plot locations.#

AVIRIS-NG scenes of of interest for the Dominant Family Polygons are:

  • ang20231109t131923_019

  • ang20231109t133124_003

  • ang20231109t133124_013

4.6.1. AVIRIS-NG Flight Lines#

4.6.1.1. A geoJSON of the AVIRIS-NG Flight Line’s Scenes are available from the JPL BioSCape Data Portal#

  • This file has been exported and already made available in the BioSCape hub in the data/ subfolder.

# read and plot the AVNG coverage file
AVNG_Coverage = gpd.read_file('data/ANG_Coverage.geojson', driver='GeoJSON')
AVNG_Coverage.plot()
<Axes: >
../../_images/9f55382e47ffcb9bb1ecebe225d15c3abd435ef0cd523376ed41158e3f4a9200.png
list(AVNG_Coverage.columns)
['LOC Download',
 'LOC ORT Download',
 'OBS Download',
 'OBS ORT Download',
 'Quicklook Download',
 'RFL Download',
 'RFL UNC Download',
 'end_time',
 'fid',
 'scid',
 'start_time',
 'style',
 'geometry']
# Let's drop some unnecessary columns that result in errors in Folium visualization
AVNG_scenes = AVNG_Coverage.drop(columns=['LOC Download', 'LOC ORT Download', 'OBS Download', 'OBS ORT Download', 'Quicklook Download', 'RFL Download', 'RFL UNC Download', 'end_time', 'start_time', 'style'])

4.6.1.2. Visualize the veg polygons, veg centers, and AVIRIS-NG Flight Lines#

import folium
mapObj = folium.Map(location=[-33.95, 23.52], zoom_start=7, control_scale=True)

folium.GeoJson(veg_polys_gdf, 
               name="BioSCape Vegetation Polygons", 
               tooltip=folium.GeoJsonTooltip(fields=["BScpPID"]), 
               style_function=lambda x: {"fillOpacity": 0}).add_to(mapObj)

folium.GeoJson(vegpolys_cen_new, 
               name="BioSCape Polygon Centers", 
               marker=folium.Circle(radius=10, fill_color="red", fill_opacity=0.4, color="red", weight=1),
               color="red").add_to(mapObj)

folium.GeoJson(AVNG_scenes, 
               name="AVIRIS-NG Coverage", 
               color="red", 
               tooltip=folium.GeoJsonTooltip(fields=["scid"]), 
               style_function=lambda x: {"fillOpacity": 0}).add_to(mapObj)

# Uncomment to see folium display
# create ESRI satellite base map
#esri = 'https://server.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer/tile/{z}/{y}/{x}'
#folium.TileLayer(tiles = esri, attr = 'Esri', name = 'Esri Satellite', overlay = False, control = True).add_to(mapObj)
#folium.LayerControl().add_to(mapObj)
#mapObj
<folium.features.GeoJson at 0x7f4a713bc2f0>

4.6.1.3. We see that there are many AVIRIS-NG scenes in the BioSCape Campaign data.#

4.6.1.4. Using a spatial join, let’s discover which AVIRIS-NG scenes contain the T081 polygon#

AVNG_flight_scenes_T081_gdf = gpd.sjoin(veg_polys_gdf.loc[veg_polys_gdf['BScpPID'] == 'T081'], AVNG_scenes, how='inner', predicate='within')
AVNG_flight_scenes_T081_gdf.scid	
16    ang20231109t131923_019
16    ang20231113t132348_002
Name: scid, dtype: object
# Here's the same method for the T195 vegetation polygon
AVNG_flight_scenes_T195_gdf = gpd.sjoin(veg_polys_gdf.loc[veg_polys_gdf['BScpPID'] == 'T195'], AVNG_scenes, how='inner', predicate='within')
AVNG_flight_scenes_T195_gdf.scid
85    ang20231109t133124_013
85    ang20231113t131251_008
85    ang20231126t095812_046
85    ang20231126t100838_008
Name: scid, dtype: object
AVNG_flight_scenes_T081_gdf
BScpPID Region Botanst Name Dscrptn DateTim geometry index_right fid scid
16 T081 CapePoint Ross Turner CapePoint_81_polygon None 24 May 2023 at 08:54:24 POLYGON ((18.44865 -34.28924, 18.44865 -34.289... 4187 ang20231109t131923 ang20231109t131923_019
16 T081 CapePoint Ross Turner CapePoint_81_polygon None 24 May 2023 at 08:54:24 POLYGON ((18.44865 -34.28924, 18.44865 -34.289... 5667 ang20231113t132348 ang20231113t132348_002
# Lets use loc and the scid to plot just the ang20231109t131923_019
AVNG_scenes.loc[AVNG_scenes['scid'] == 'ang20231109t131923_019'].plot()
<Axes: >
../../_images/1727754debbb112d118341603bf34e9782ee027bafad1382cae66cab32293316.png

4.6.1.5. Now plot it all together with the identified AVIRIS-NG scene#

# and now plot it all together with the identified AVIRIS-NG scene
import folium
mapObj = folium.Map(location=[-33.95, 23.52], zoom_start=7, control_scale=True)

folium.GeoJson(veg_polys_gdf, 
               name="BioSCape Vegetation Polygons", 
               tooltip=folium.GeoJsonTooltip(fields=["BScpPID"]), 
               style_function=lambda x: {"fillOpacity": 0}).add_to(mapObj)

folium.GeoJson(vegpolys_cen_new, 
               name="BioSCape Polygon Centers", 
               marker=folium.Circle(radius=10, fill_color="red", fill_opacity=0.4, color="red", weight=1),
               color="red").add_to(mapObj)

folium.GeoJson(AVNG_scenes, 
               name="AVIRIS-NG Coverage", 
               color="red", 
               tooltip=folium.GeoJsonTooltip(fields=["scid"]), 
               style_function=lambda x: {"fillOpacity": 0}).add_to(mapObj)

folium.GeoJson(AVNG_scenes.loc[AVNG_scenes['scid'] == 'ang20231109t131923_019'], 
               name="AVIRIS-NG Coverage", 
               color="green", 
               tooltip=folium.GeoJsonTooltip(fields=["scid"]), 
               style_function=lambda x: {"fillOpacity": 0}).add_to(mapObj)

# create ESRI satellite base map
esri = 'https://server.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer/tile/{z}/{y}/{x}'
folium.TileLayer(tiles = esri, attr = 'Esri', name = 'Esri Satellite', overlay = False, control = True).add_to(mapObj)
folium.LayerControl().add_to(mapObj)
mapObj
Make this Notebook Trusted to load map: File -> Trust Notebook

4.7. Examine BioSCape AVIRIS-NG Files that have Vegetation Plot of Interest#

4.7.1. The ang20231109t131923_019 scene is a segment of a larger flight line, ang20231109t131923#

s3.ls('bioscape-data/AVNG/ang20231109t131923')
['bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_000',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_001',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_002',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_003',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_004',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_005',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_006',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_007',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_008',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_009',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_010',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_011',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_012',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_013',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_014',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_015',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_016',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_017',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_018',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_019',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_020',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_021']

4.7.1.1. ^^^ There are 22 scenes in this (ang20231109t131923) flight line ^^^#

# List the data files in the ang20231109t131923_019 scene

ang20231109t131923_019 =  s3.ls('bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_019')
ang20231109t131923_019
['bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_019/ang20231109t131923_019_L1B_ORT_main_46dd9a4a_LOC',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_019/ang20231109t131923_019_L1B_ORT_main_46dd9a4a_LOC.hdr',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_019/ang20231109t131923_019_L1B_ORT_main_46dd9a4a_LOC_ORT',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_019/ang20231109t131923_019_L1B_ORT_main_46dd9a4a_LOC_ORT.hdr',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_019/ang20231109t131923_019_L1B_ORT_main_46dd9a4a_OBS',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_019/ang20231109t131923_019_L1B_ORT_main_46dd9a4a_OBS.hdr',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_019/ang20231109t131923_019_L1B_ORT_main_46dd9a4a_OBS_ORT',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_019/ang20231109t131923_019_L1B_ORT_main_46dd9a4a_OBS_ORT.hdr',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_019/ang20231109t131923_019_L2A_OE_main_27577724_RFL_ORT',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_019/ang20231109t131923_019_L2A_OE_main_27577724_RFL_ORT.hdr',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_019/ang20231109t131923_019_L2A_OE_main_27577724_RFL_ORT.json',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_019/ang20231109t131923_019_L2A_OE_main_27577724_RFL_ORT_QL.tif',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_019/ang20231109t131923_019_L2A_OE_main_27577724_UNC_ORT',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_019/ang20231109t131923_019_L2A_OE_main_27577724_UNC_ORT.hdr',
 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_019/ang20231109t131923_019_RFL_UNC_COMBINED_ORT.json']
  • Flight lines are provided as small, manageable sections of data

  • Within a flight line, adjacent scenes are seamless

  • However, adjacent flight lines will have overlap

  • At the time of this workshop, data are in binary/header (ENVI) formats

Dataset

Description

L1B

Resampled calibrated data in units of spectral radiance. Observed geometry and illumination parameters

L2

Orthocorrected and atmospherically corrected reflectance data. Uncertainty data. (425 Spectral bands)

QL

A quick look geoTIFF file (3 bands)

4.8. Visualize AVIRIS-NG Reflectance Files#

4.8.1. Examine the Quicklook file with GDAL#

# Define the path for a quick look image
ang20231109t131923_019_ql_path = [x for x in ang20231109t131923_019 if x.endswith('RFL_ORT_QL.tif')][0]
ang20231109t131923_019_ql_path
'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_019/ang20231109t131923_019_L2A_OE_main_27577724_RFL_ORT_QL.tif'

There’s a bit of a catch with the cloud access methods using GDAL. The GDAL utility expects S3 links to be formated with the GDAL virtual file system (VSI) S3 path. We therefore have to use the VSI path to access the files with GDAL. Shown below, we’ll substitute the S3 link with the VSI (vsis3) link(s).

ang20231109t131923_019_ql = gdal.Open(path.join('/vsis3', ang20231109t131923_019_ql_path))
ang20231109t131923_019_ql
<osgeo.gdal.Dataset; proxy of <Swig Object of type 'GDALDatasetShadow *' at 0x7f4a722105d0> >
# Open the AVIRIS-NG ENVI file and read the file's bands, row, cols
#image_open = gdal.Open(gdal_url)

nbands_003 = ang20231109t131923_019_ql.RasterCount
ncols_003 = ang20231109t131923_019_ql.RasterXSize
nrows_003 = ang20231109t131923_019_ql.RasterYSize

print(f"Bands:\t{nbands_003}")
print(f"Columns:\t{ncols_003}")
print(f"Rows:\t{nrows_003}")
Bands:	3
Columns:	632
Rows:	715

4.8.2. Using GDAL to read the AVIRIS-NG raster file as a numerical array#

  • convert an existing Gdal Dataset or a Band into a numpy array with the ReadAsArray() function

ang_019_ql_array = ang20231109t131923_019_ql.ReadAsArray()
ang_019_ql_trans = ang_019_ql_array.transpose((1,2,0))  # the band arranged needs rearranged in order to plot
# bands 258 95 36
plt.rcParams['figure.figsize'] = [10,10]
plt.imshow(ang_019_ql_trans)
<matplotlib.image.AxesImage at 0x7f4ab2b10740>
../../_images/3e15e8815db7de7fe911c4c93b45e3040aea887d03bd33121f9ec8d39862e7d5.png
## Quick note - you can use this get expression to download data from S3 storage to your local environment
## We will not execute this block in the workshop.

# s3.get(ang20231109t131923_019_ql_path, 'ang20231109t131923_019_ql.tif')

4.9. Examine the AVNG Reflectance file (RFL) with rioxarray#

Rioxarray is a python package that is built upon xarray and rasterio python packages to facilitate the analysis of raster or xarray datasest.

We’ll demonstrate

  • how to read and visualize raster data using rioxarray

  • how to extract data from a point location.

4.9.1. First, let’s examine the header file associated with the ENVI binary/header file pair#

  • the header files are ascii text files

hdr_link = 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_019/ang20231109t131923_019_L2A_OE_main_27577724_RFL_ORT.hdr'
hdr_link
'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_019/ang20231109t131923_019_L2A_OE_main_27577724_RFL_ORT.hdr'
# Print the whole header file
#with s3.open(hdr_link, mode='r') as hdr:
#    lines = (hdr.read())
#    print(lines)

# Print the first 12 lines of the header file
with s3.open(hdr_link, mode='r') as hdr:
    for i in range(12):
        line = next(hdr).strip()
        print(line) 
ENVI
description = {
L2A Analytyical per-pixel surface retrieval}
samples = 632
lines = 715
bands = 425
header offset = 0
file type = ENVI Standard
data type = 4
interleave = bil
byte order = 0
map info = {UTM, 1, 1, 262884.8679920948, 6204108.374809049, 4.9, 4.9, 34, South, WGS-84, units=Meters, rotation=0.0}
# assign the the associated L2 Reflectance file to a variable
ang20231109t131923_019_rfl = [x for x in ang20231109t131923_019 if x.endswith('RFL_ORT')][0]
ang20231109t131923_019_rfl
'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_019/ang20231109t131923_019_L2A_OE_main_27577724_RFL_ORT'
# open the reflectance file with rioxarray
avng_rfl = rioxarray.open_rasterio(path.join('s3://', ang20231109t131923_019_rfl),
                             drivers='ENVI', chunks=True, parse_coordinates=True)
avng_rfl
<xarray.DataArray (band: 425, y: 715, x: 632)> Size: 768MB
dask.array<open_rasterio-70bd26400f8937bc0b6dff82046ded62<this-array>, shape=(425, 715, 632), dtype=float32, chunksize=(1, 715, 632), chunktype=numpy.ndarray>
Coordinates:
    fwhm         (band) float64 3kB dask.array<chunksize=(1,), meta=np.ndarray>
    wavelength   (band) float64 3kB dask.array<chunksize=(1,), meta=np.ndarray>
  * band         (band) int64 3kB 1 2 3 4 5 6 7 ... 419 420 421 422 423 424 425
  * x            (x) float64 5kB 2.629e+05 2.629e+05 ... 2.66e+05 2.66e+05
  * y            (y) float64 6kB 6.204e+06 6.204e+06 ... 6.201e+06 6.201e+06
    spatial_ref  int64 8B 0
Attributes: (12/445)
    Band_1:               channel_0 (377.1956495 Nanometers)
    Band_10:              channel_9 (422.27564950000004 Nanometers)
    Band_100:             channel_99 (873.0556495000001 Nanometers)
    Band_101:             channel_100 (878.0656495 Nanometers)
    Band_102:             channel_101 (883.0756494999999 Nanometers)
    Band_103:             channel_102 (888.0756494999999 Nanometers)
    ...                   ...
    file_type:            ENVI Standard
    header_offset:        0
    interleave:           bil
    lines:                715
    masked_pixel_noise:   2.711519718170166
    samples:              632
# Examine the coordinate reference system of the AVIRIS-NG file
avng_rfl.rio.crs
CRS.from_epsg(32734)

The EPSG code is 32734

# View the Coordinate Reference System (CRS) & spatial extent
# explore the full definition of the EPSG code
from pyproj import CRS
epsg = avng_rfl.rio.crs.to_epsg()
crs = CRS(epsg)
crs
<Projected CRS: EPSG:32734>
Name: WGS 84 / UTM zone 34S
Axis Info [cartesian]:
- E[east]: Easting (metre)
- N[north]: Northing (metre)
Area of Use:
- name: Between 18°E and 24°E, southern hemisphere between 80°S and equator, onshore and offshore. Angola. Botswana. Democratic Republic of the Congo (Zaire). Namibia. South Africa. Zambia.
- bounds: (18.0, -80.0, 24.0, 0.0)
Coordinate Operation:
- name: UTM zone 34S
- method: Transverse Mercator
Datum: World Geodetic System 1984 ensemble
- Ellipsoid: WGS 84
- Prime Meridian: Greenwich

4.9.1.1. This AVIRIS-NG scence is in UTM Zone 34S projection#

4.10. Plot a Spectral Profile from a Point of Interest#

4.10.1. We’ll plot the spectral profiles for a pixel based on a vegeation plot location.#

  • Recall we saved the latitude and longitude of the T081 vegetation plot center.

  • We’ll project point location lat/lon to UTM (zone 34S): recall, the AVIRIS-NG Scene is in UTM zone 34S (EPSG:32734)

# recall
print('Plot_T081 longitude: ', T081lon)  
print('Plot_T081 latitude: ', T081lat)
Plot_T081 longitude:  18.447671093155833
Plot_T081 latitude:  -34.288518109266434

4.10.1.1. Using Pyproj, convert lon/lat to UTM projected x/y#

Pyproj is a Python interface to PROJ (cartographic projections and coordinate transformations library)

# translate coordinates

p = Proj("EPSG:32734", preserve_units=False)
T081_x,T081_y = p(T081lon, T081lat)
print('T081 x value in UTM meters:', T081_x)
print('T081 y value in UTM meters:', T081_y)
T081 x value in UTM meters: 265069.2221736356
T081 y value in UTM meters: 6202903.906296786

4.10.2. Visualize the AVIRIS-NG Reflectance image and the T081 plot location#

# Plot the image with the T081 plot location

plt.figure(figsize=(10,10))
avng_rfl.isel(band=[57, 38, 22]).plot.imshow(rgb="band", vmin=0, vmax=.25)
plt.scatter(T081_x,T081_y, color='red')
plt.title(path.basename(ang20231109t131923_019_rfl))
plt.show()
../../_images/341fda1748440c7a03ff2f18d786c6f977e89a62169eef5f4066f03b6c48305f.png

4.10.3. For the T081 vegetation plot center location, let’s extract the reflectance value from each of the 425 bands, and plot this as a spectral profile#

# plot spectra from the y,x pixel defined in UTM coordinates
T081_ang_wl = avng_rfl.sel(y=T081_y, x=T081_x, method='nearest')

T081_ang_wl.plot.line(ylim=(-.2,.4), color = 'g')
plt.rcParams['figure.figsize'] = [8,6]
plt.xlabel("band")
plt.ylabel("reflectance")
plt.show()
../../_images/cf975d34bfbc8879c5dd402fbe03059fe7b71a46e652597c7cb0e6cd060d6c94.png
# Define a list of wavelengths that are "bad" 
import numpy as np
bblist = np.ones((425,))  # create a 1D array with values ones
# set tails and atmospheric window to zero
bblist[0:14] = 0        # tail
bblist[193:210] = 0     # atmospheric window
bblist[281:320] = 0     # atmospheric window
bblist[405:] = 0        # tail
T081_ang_wl[bblist == 0] = np.nan

plt.rcParams['figure.figsize'] = [8,6]
T081_ang_wl.plot.line(ylim=(0,.3), color = 'g')  
plt.xlabel("band")
plt.ylabel("reflectance")
plt.legend(["Proteaceae"])
plt.show()
../../_images/5bd6cfea1ae9a1373f9af03da26c9b4e05ae252219ee4222c3410bc589035d12.png

4.11. Plot the Spectral Profiles from each of the Vegeation Sample plots#

AVIRIS-NG scenes of of interest for the Dominant Family Polygons are:

  • T081 = Proteaceae —— AVNG = ang20231109t131923_019_rfl

  • T093 = Asteraceae —— AVNG = ang20231109t133124_003_rfl

  • T201 = Ericaceae ——– AVNG = ang20231109t133124_013_rfl

  • T069 = Rosaceae ——– AVNG = ang20231109t133124_013_rfl

  • T195 = Restionaceae —- AVNG = ang20231109t133124_013_rfl

# T081 from above

T093_point = vegpolys_cen_new.loc[vegpolys_cen_new['BScpPID'] == 'T093'].get_coordinates()
T093lon = T093_point.x.values[0]
T093lat = T093_point.y.values[0]

T201_point = vegpolys_cen_new.loc[vegpolys_cen_new['BScpPID'] == 'T201'].get_coordinates()
T201lon = T201_point.x.values[0]
T201lat = T201_point.y.values[0]

T069_point = vegpolys_cen_new.loc[vegpolys_cen_new['BScpPID'] == 'T069'].get_coordinates()
T069lon = T069_point.x.values[0]
T069lat = T069_point.y.values[0]

T195_point = vegpolys_cen_new.loc[vegpolys_cen_new['BScpPID'] == 'T195'].get_coordinates()
T195lon = T195_point.x.values[0]
T195lat = T195_point.y.values[0]

# Reproject point from lon/Lat to UTM
p = Proj("EPSG:32734", preserve_units=False)
T093_x,T093_y = p(T093lon, T093lat)
T201_x,T201_y = p(T201lon, T201lat)
T069_x,T069_y = p(T069lon, T069lat)
T195_x,T195_y = p(T195lon, T195lat)

avng_rfl_081_path = 'bioscape-data/AVNG/ang20231109t131923/ang20231109t131923_019/ang20231109t131923_019_L2A_OE_main_27577724_RFL_ORT'
avng_rfl_093_path = 'bioscape-data/AVNG/ang20231109t133124/ang20231109t133124_003/ang20231109t133124_003_L2A_OE_main_27577724_RFL_ORT'
avng_rfl_201_path = 'bioscape-data/AVNG/ang20231109t133124/ang20231109t133124_013/ang20231109t133124_013_L2A_OE_main_27577724_RFL_ORT'

# Read the reflectance file

#avng_rfl_081= rioxarray.open_rasterio(path.join('s3://', avng_rfl_081_path), drivers='ENVI', chunks=True, parse_coordinates=True)
avng_rfl_093 = rioxarray.open_rasterio(path.join('s3://', avng_rfl_093_path), drivers='ENVI', chunks=True, parse_coordinates=True)
avng_rfl_201 = rioxarray.open_rasterio(path.join('s3://', avng_rfl_201_path), drivers='ENVI', chunks=True, parse_coordinates=True)

T081_ang_wl[bblist == 0] = np.nan
T093_ang_wl = avng_rfl_093.sel(y=T093_y, x=T093_x, method='nearest')
T093_ang_wl[bblist == 0] = np.nan
T201_ang_wl = avng_rfl_201.sel(y=T201_y, x=T201_x, method='nearest')
T201_ang_wl[bblist == 0] = np.nan
T069_ang_wl = avng_rfl_201.sel(y=T069_y, x=T069_x, method='nearest')
T069_ang_wl[bblist == 0] = np.nan
T195_ang_wl = avng_rfl_201.sel(y=T195_y, x=T195_x, method='nearest')
T195_ang_wl[bblist == 0] = np.nan

# plot spectra from a pixel

T081_ang_wl.plot.line(ylim=(0,.4), color = 'g', label="Proteaceae")
T093_ang_wl.plot.line(ylim=(0,.4),color = 'r', label="Asteraceae")
T201_ang_wl.plot.line(ylim=(0,.4),color = 'y', label="Ericaceae")
T069_ang_wl.plot.line(ylim=(0,.4),color = 'purple', label="Rosaceae")
T195_ang_wl.plot.line(ylim=(0,.4),color = 'black', label="Restionaceae")

plt.rcParams['figure.figsize'] = [8,6]
plt.xlabel('band', fontsize=20)
plt.ylabel('Reflectance', fontsize=20)
plt.legend(loc="upper left")
plt.show()
../../_images/f43f931da4a1b85290b636fc1ca632dafe7c1fc3e661cbfc4d2ea4b767628dff.png