This tutorial will demonstrate how to download Daymet: Daily Surface Weather Data on a 1-km Grid for North America, Version 4 for a bounding box. Daymet dataset is available for North America (NA), including Hawaii (HI) and Puerto Rico (PR).
This tutorial will demonstrate how to directly access and subset Daymet dataset using NASA OPeNDAP Hyrax. Hyrax allows direct access to selected variables for the dataset within the spatial-temporal bounds without having to download the whole data files.
Requirements¶
import earthaccess
from pydap.net import create_session
import geopandas as gpd
import xarray as xr
from pathlib import Path
from shapely.geometry import box
import warnings
warnings.filterwarnings('ignore')Access to NASA OPeNDAP requires NASA Earthdata authentication. We recommend authenticating your Earthdata Login (EDL) information using the earthaccess python library as follows:
# works if the EDL login already been persisted to a netrc
try:
earthaccess.login(strategy="netrc")
except FileNotFoundError:
# ask for EDL credentials and persist them in a .netrc file
earthaccess.login(strategy="interactive", persist=True)Searching for Daymet Files¶
Daymet data files (or granules) are natively in netCDF4 format, and each file has one year’s worth of data. Data files are organized by variables (each for dayl, prcp, tmin, tmax, srad, swe, vp) and regions (each for US, PR, HI). For this tutorial, we will download data for the year 2024. We will first search for all the granules in the area of interest within the bounds of 170 W, 67.5N and 136.5W, 71.5N degrees.
# bound
BOUND = (-170, 67.5, -136.5, 71.5)
gdf = gpd.GeoDataFrame(geometry=[box(*BOUND)], crs="EPSG:4326")
# plot
# gdf.explore(color="red", alpha=0.3)We will now search for all the Daymet granules overlapping the above bounding box using the Earthaccess Python library. We will also limit the period of our search to the year 2024 (Jan 1 to Dec 31, 2024). The earthaccess’s search_data allows searching dataset granules using NASA’s CMR API. It requires the dataset digital object identifier (DOI), time, and the bounding box.
# Daymet DOI
DOI = '10.3334/ORNLDAAC/2129'
# dates
DATES = ("2024-01-01", "2024-12-31")
# search data
granules = earthaccess.search_data(
doi=DOI,
bounding_box=BOUND,
temporal=DATES,
)
print(f"Total granules found: {len(granules)}")Total granules found: 7
Now, the day granule information, including data URLs, is stored in granules. The OPeNDAP URLs are in the RelatedUrls fields of the granule metadata. Let’s extract the OPeNDAP URLS.
# opendap urls
hyrax = [g._filter_related_links('USE SERVICE API')[0].replace("https", "dap4") for g in granules]
# print first 3
hyrax [:3]['dap4://opendap.earthdata.nasa.gov/collections/C2532426483-ORNL_CLOUD/granules/Daymet_Daily_V4R1.daymet_v4_daily_na_tmin_2024.nc',
'dap4://opendap.earthdata.nasa.gov/collections/C2532426483-ORNL_CLOUD/granules/Daymet_Daily_V4R1.daymet_v4_daily_na_vp_2024.nc',
'dap4://opendap.earthdata.nasa.gov/collections/C2532426483-ORNL_CLOUD/granules/Daymet_Daily_V4R1.daymet_v4_daily_na_prcp_2024.nc']Project the bounds to Daymet CRS¶
Daymet is natively in lambert_conformal_conic (LCC) coordinate reference system. We can now reproject the bounding coordinates of the above geopandas object (gdf) to bounds_lcc for requesting to OPeNDAP later.
#Daymet LCC proj
DAYMET_PROJ = (
"+proj=lcc +ellps=WGS84 +a=6378137 +b=6356752.314245 "
"+lat_1=25 +lat_2=60 +lon_0=-100 +lat_0=42.5 "
"+x_0=0 +y_0=0 +units=m +no_defs"
)
# convert to LCC proj and retrieve bounds
gdf_lcc = gdf.to_crs(DAYMET_PROJ)
bounds_lcc = gdf_lcc.total_bounds
# print the four corners in LCC
bounds_lccarray([-2823222.26105783, 3101471.12476104, -1401567.56219731,
4328450.66251338])Retrieve Daymet subset with OPeNDAP¶
Access to NASA OPeNDAP requires NASA Earthdata authentication. Please set up a new account if you do not already have a NASA Earthdata account. We will use earthaccess module to set up the authentication and return a session instance with an authorized bearer token.
NOTE: it is recommended to use pydap’s create_session method, which is also a requests session, but it is configured to stream larger data arrays.
session = create_session(session=earthaccess.get_requests_https_session())Efficiently communicate with the remote OPeNDAP server¶
Hint 1: When “opening” multiple remote files with Xarray, Xarray automatically downloads any coordinate data from each remote file. In our case, this means that both
lat and lon — each a huge 2D array that is identical across all files — will unnecessarily be downloaded, resulting in a negative impact on our performance. To avoid this
we need to drop these coordinates from all but a single remote file. Xarray has a useful method called drop_variables, but it is only usable after “opening” the
dataset, i.e. after these have already been downloaded. Luckily, OPeNDAP and the DAP4 protocol can be exploited to drop the variables before downloading. This is done
by manipulating the URL so that only the variable of interest is requested from the remote file.
Hint 2: When the remote file is huge, the subset can be large too. It helps to split the download into multiple chunks, download them in parallel, and reassemble them locally.
This can be done with Xarray! But to achieve that we need to define a chunk parameter when “opening” the dataset. This chunk parameter should not be larger than our desired subset
(otherwise we will download too much data). Below, we make it so that chunk matches the horizontal extent of our subset, but split the download along the time dimension into Nt chunks.
#### ==== HINT 1
# Create a single dataset containing all coordinate/dimension data.
ds_coords = xr.open_dataset(hyrax[0], engine='pydap', session=session)[['x','y','lat','lon','time']]
# In this step below, we drop all coordinates from each remote file, except the variable of interest
urls = [url+f"?dap4.ce=/{Path(url).stem.split("_")[-2]}" for url in hyrax]
#### ==== HINT 2
# Now, identify the subset and Chunk the dataset to optimize and split streaming the data is smaller
# requests
x_idx = ds_coords.indexes["x"].get_indexer([bounds_lcc[0], bounds_lcc[2]], method="nearest")
y_idx = ds_coords.indexes["y"].get_indexer([bounds_lcc[3], bounds_lcc[1]], method="nearest")
nx = len(ds_coords.x) # full x coordinate array
ny = len(ds_coords.y) # full y coordinate array
x_start, x_stop = sorted(x_idx)
y_start, y_stop = sorted(y_idx)
x_len = x_stop - x_start + 1
y_len = y_stop - y_start + 1
# Chunking when opening a dataset allows to split the download into further multiple subsets, and Xarray has internal logic
# that can reassemble each piece.
Nt = 3 # each remote file will be split into `Nt` downloads.
Chunks = {
'time': round(len(ds_coords.time) / Nt),
"x": (x_start, x_len, nx - x_start - x_len),
"y": (y_start, y_len, ny - y_start - y_len),
}
# ============ Now we apply Hint 1 and Hint 2
ds = xr.merge([ds_coords] + [xr.open_dataset(url, engine='pydap',session=session, chunks=Chunks) for url in urls])
dsStream data in parallel¶
Should take only a couple of minutes.
f_name = "_".join(Path(hyrax[0]).stem.split("_")[:-3])
ds.isel(y=slice(y_idx[0], y_idx[1]), x=slice(x_idx[0], x_idx[1])).to_netcdf(f"{f_name}_subset.nc4")
sub_ds = xr.open_dataset(f"{f_name}_subset.nc4")
sub_dsprint(f"From the {round(ds.nbytes/1e9, 1)} Gbs of remote data, we only downloaded {round(sub_ds.nbytes/1e9,1)} Gbs")From the 645.4 Gbs of remote data, we only downloaded 17.8 Gbs
- Thornton, M. M., Shrestha, R., Wei, Y., Thornton, P. E., & Kao, S.-C. (2022). Daymet: Daily Surface Weather Data on a 1-km Grid for North America, Version 4 R1. ORNL Distributed Active Archive Center. 10.3334/ORNLDAAC/2129