NASA Ames Global Hyperspectral Synthetic Data (AGHSD)


INTRODUCTION:
-----------------------------
This is the Ames Global Hyperspectral Synthetic Data (AGHSD, pronounced like "august") public data set. The data represents Top-of-Atmosphere
(TOA) reflectance in the spectral range from 380 nm to 2500 nm with 10 nm bandwidths and ~1km spatial resolution, generated based on
global MODIS observations and endmembers from an AVIRIS [1] hyperspectral library with a linear unmixing algorithm.

This data set consists of synthetic (or simulated) daily global TOA hyperspectral reflectance for
the year 2019. The data are provided on geographic grids from 60degN to 60degS and 180degW to 180degE at a 0.01x0.01
degree spacing. Each pixel value includes 213 spectral bands from 380 nm to 2500 nm at a 10 nm
spectral resolution. The synthetic data set is generated based on the MODIS TOA calibrated radiance product
(MOD02) and a hyperspectral library sampled from AVIRIS-NG data with a linear unmixing algorithm. The goal
of this data set is to provide a practical analog for the data format and volume of the future
Surface Biology and Geology (SBG) [2] missions.

The AGHSD data is exported on the NAS dataportal [3] and is currently (2021) in the NEX [4,5] datapool [6].


DIRECTORY & FILE NAMING CONVENTIONS & STRUCTURE
------------------------------------------------
The data directory and file naming convention is the following:

<version>/h<hid>v<vid>/<year>/sbg_mdl_h<hid>v<vid>_YYYYDDD_toa.hdf5

The "parts" are defined as:

<version> := The data version code, is currently version1.0
<hid> := The horizontal tile ID.
<vid> := The vertical tile ID.
<year> := The year of the synthetic data
<DDD> := The day of the year of the synthetic data, e.g. 001, 365, ...

All files are in HDF5 [7] file format.

The file index_v10_md5.txt contains a full listing of md5 digests for each
hdf5 data file. One my use the program md5sum [8] to detect any 
file corruption after data download, e.g. md5sum -c index_v10_md5.txt


BULK DOWNLOADS
---------------
As noted above, the archive contains a file with the MD5 digests, while this file can be used to
check for data integrity after data download, it can also help one download an entire tile 
directory, or more. From a standard UNIX/Linux machine terminal, the following chain of common commands 
below will download the tile directory h15v16 using 2 processes (2 simultaneous download streams):

$ curl https://data.nas.nasa.gov/aghsd/download_data.php?file=/aghsddata/index_v10_md5.txt | grep h15v16  | \
    cut -d ' ' -f 3 | xargs -P 2 -I % wget --content-disposition "https://data.nas.nasa.gov/aghsd/download_data.php?file=/aghsddata/%"

The above command uses both curl [9] and wget [10] along with common UNIX/Linux commands found on the majority of systems.


COMMUNITY AND SUPPORT
-----------------------------
For specific support questions and issues with the data products email support@nas.nasa.gov and include "AGHSD" in the subject line.


REFERENCES
-----------------------------
1. https://aviris.jpl.nasa.gov/data/get_aviris_data.html
2. https://science.nasa.gov/earth-science/decadal-sbg
3. https://data.nas.nasa.gov/
4. https://www.nasa.gov/nex/
5. https://www.nasa.gov/nex/access
6. File system at NAS /nex/datapool/AGHSD/
7. https://portal.hdfgroup.org/display/HDF5/HDF5
8. https://www.gnu.org/software/coreutils/manual/html_node/md5sum-invocation.html
9. https://curl.se/
10. https://www.gnu.org/software/wget/

All reference links retrieved in 2021.

DISCLAIMER
-----------------------------
The data are considered provisional and subject to change. The data are provided as is without any warranty of any
kind, either express or implied, arising by law or otherwise, including but not limited to warranties of completeness,
non-infringement, accuracy, merchantability, or fitness for a particular purpose. The user assumes all risk associated
with the use of, or inability to use, the data.