Decoding the Metadata: Inside the HDF5 - Tanager Hyperspectral Data Analysis Course

Welcome to the final lesson of Module 2!

In our previous lessons, you learned how to navigate an HDF5 file, extract massive 3D data cubes, and read basic file attributes. However, to truly complete the picture, there is one last, crucial piece of information hidden inside the file.

Take a look at the very bottom of the directory structure below. Inside the HDFEOS INFORMATION/ folder, there is a dataset named StructMetadata.0. Both Basic (SWATHS) and Ortho (GRIDS) products store this in the same location, though the metadata content differs to match each product’s geometry.

📂 HDFEOS/
    📂 HDFEOS/ADDITIONAL/
        📂 HDFEOS/ADDITIONAL/FILE_ATTRIBUTES/
    📂 HDFEOS/SWATHS/
        📂 HDFEOS/SWATHS/HYP/
            📂 HDFEOS/SWATHS/HYP/Data Fields/
                📄 HDFEOS/SWATHS/HYP/Data Fields/aerosol_optical_depth | (501, 607) | float32
                📄 HDFEOS/SWATHS/HYP/Data Fields/beta_cirrus_mask | (501, 607) | uint8
                📄 HDFEOS/SWATHS/HYP/Data Fields/beta_cloud_mask | (501, 607) | uint8
                📄 HDFEOS/SWATHS/HYP/Data Fields/column_water_vapour | (501, 607) | float32
                📄 HDFEOS/SWATHS/HYP/Data Fields/nodata_pixels | (501, 607) | uint8
                📄 HDFEOS/SWATHS/HYP/Data Fields/sensor_azimuth | (501, 607) | float32
                📄 HDFEOS/SWATHS/HYP/Data Fields/sensor_to_ground_path_length | (501, 607) | float32
                📄 HDFEOS/SWATHS/HYP/Data Fields/sensor_zenith | (501, 607) | float32
                📄 HDFEOS/SWATHS/HYP/Data Fields/sun_azimuth | (501, 607) | float32
                📄 HDFEOS/SWATHS/HYP/Data Fields/sun_zenith | (501, 607) | float32
                📄 HDFEOS/SWATHS/HYP/Data Fields/surface_reflectance | (426, 501, 607) | float32
                📄 HDFEOS/SWATHS/HYP/Data Fields/surface_reflectance_uncertainty | (426, 501, 607) | float32
            📂 HDFEOS/SWATHS/HYP/Geolocation Fields/
                📄 HDFEOS/SWATHS/HYP/Geolocation Fields/Latitude | (501, 607) | float64
                📄 HDFEOS/SWATHS/HYP/Geolocation Fields/Longitude | (501, 607) | float64
                📄 HDFEOS/SWATHS/HYP/Geolocation Fields/Time | (501,) | float64
📂 HDFEOS INFORMATION/
    📄 HDFEOS INFORMATION/StructMetadata.0 | () | |S32000

Notice its shape is () and its data type is |S32000. This tells us that this specific item is not a standard numerical array or list like our Surface Reflectance data. Instead, it is a single, massive string of text (up to 32,000 bytes long) containing the structural blueprint of the entire file.

Because it is packaged so differently, we cannot read it using the same array indexing we used previously. This structured metadata isn’t really intended for “human consumption” but rather for existing software to find critical metadata about the file automatically. But we’ll look anyway! In this short lesson, we are going to use a special approach to crack open this dataset by decoding the text.

Setup & Target the Metadata¶

We will use the same scene from the previous lesson: Basic and Ortho surface reflectance products for the Ringkøbing-Skjern, Denmark scene. If you haven’t downloaded them yet, run the cell below. Otherwise, ensure your .h5 files are in your working directory (or update the paths accordingly).

Once we open a file, we navigate directly to that dataset at the bottom of our directory tree. We do this by passing its exact folder path, 'HDFEOS INFORMATION/StructMetadata.0', into our file object.

Let’s run the code and simply print() what we find at that location.

🧰 Your Toolkit so far¶

For this lesson, your toolkit has been reduced; we only need the downloader function for this lesson.

# Import necessary libraries

import h5py

# ---- setup -------------------------------------------------------------

def download_tanager_data(url, file_path):
    import os
    import urllib.request

    def _progress(count, block_size, total_size):
        mb_done = count * block_size / 1_000_000
        mb_total = total_size / 1_000_000
        print(f"\rDownloading... {mb_done:.1f} / {mb_total:.1f} MB", end="", flush=True)

    if not os.path.exists(file_path):
        urllib.request.urlretrieve(url, file_path, reporthook=_progress)
        print(f"\nDownload complete: {file_path}")
    else:
        print(f"File already exists, skipping download: {file_path}")
    return file_path

# Download the Basic and Ortho SR assets for this scene using your Toolkit's
# download_tanager_data (defined in the Toolkit cell above).
basic_sr_url = "https://storage.googleapis.com/open-cogs/planet-stac/tanager1-release2-core-imagery/basic_sr_hdf5/20250510_112042_16_4001_basic_sr_hdf5.h5"
ortho_sr_url = "https://storage.googleapis.com/open-cogs/planet-stac/tanager1-release2-core-imagery/ortho_sr_hdf5/20250510_112042_16_4001_ortho_sr_hdf5.h5"

# These filenames are reused throughout the lesson:
# Note: You can change the file name to any name you want.
basic_sr_path = "20250510_112042_16_4001_basic_sr_hdf5.h5"
ortho_sr_path = "20250510_112042_16_4001_ortho_sr_hdf5.h5"

download_tanager_data(basic_sr_url, basic_sr_path)
download_tanager_data(ortho_sr_url, ortho_sr_path)

print("✅ Assets ready")

File already exists, skipping download: 20250510_112042_16_4001_basic_sr_hdf5.h5
File already exists, skipping download: 20250510_112042_16_4001_ortho_sr_hdf5.h5
✅ Assets ready

with h5py.File(basic_sr_path, "r") as f:
    # This just prints the object reference
    print(f["HDFEOS INFORMATION/StructMetadata.0"])

<HDF5 dataset "StructMetadata.0": shape (), type "|S32000">

The output tells us this dataset contains a byte string up to 32,000 characters long (|S32000).

To actually read the text, we need to extract the data using [()] (which tells h5py to load it into memory) and then .decode('utf-8') to convert the raw computer bytes into human-readable text.

32,000 characters is a massive wall of text. Let’s print it anyway and see what the structural metadata looks like!

Inspecting the Metadata Structures¶

With the metadata located, let’s read and interpret the two structural blocks that describe each product’s geometry: the Ortho GridStructure and the Basic SwathStructure.

Ortho Product (GridStructure)¶

Let’s read the StructMetadata from the Ortho product first.

with h5py.File(ortho_sr_path, "r") as f:
    metadata_dataset = f["HDFEOS INFORMATION/StructMetadata.0"]
    raw_metadata = metadata_dataset[()].decode("utf-8")
    print("\n--- Ortho Product (GridStructure) ---\n" + "="*40)
    print(raw_metadata)


--- Ortho Product (GridStructure) ---
========================================
GROUP=SwathStructure
END_GROUP=SwathStructure
GROUP=GridStructure
	GROUP=GRID_1
		GridName="HYP"
		Band=426
		XDim=847
		YDim=724
		UpperLeftPointMtrs=(437820.00,6244950.00)
		LowerRightMtrs=(463230.00,6223230.00)
		Projection=HE5_GCTP_UTM
		ZoneCode=32
		SphereCode=12
		CompressionType=HE5_HDFE_COMP_DEFLATE
		DeflateLevel=4
		PixelRegistration=HE5_HDFE_CORNER
		GridOrigin=HE5_HDFE_GD_UL
		GROUP=Dimension
			OBJECT=Dimension_1
				DimensionName="Band"
				Size=426
			END_OBJECT=Dimension_1
			OBJECT=Dimension_2
				DimensionName="YDim"
				Size=724
			END_OBJECT=Dimension_2
			OBJECT=Dimension_3
				DimensionName="XDim"
				Size=847
			END_OBJECT=Dimension_3
		END_GROUP=Dimension
		GROUP=DataField
			OBJECT=DataField_1
				DataFieldName="surface_reflectance"
				DataType=H5T_NATIVE_FLOAT
				DimList=("Band","YDim","XDim")
				MaxdimList=("Band","YDim","XDim")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=4
			END_OBJECT=DataField_1
			OBJECT=DataField_2
				DataFieldName="sensor_zenith"
				DataType=H5T_NATIVE_FLOAT
				DimList=("YDim","XDim")
				MaxdimList=("YDim","XDim")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=4
			END_OBJECT=DataField_2
			OBJECT=DataField_3
				DataFieldName="sensor_azimuth"
				DataType=H5T_NATIVE_FLOAT
				DimList=("YDim","XDim")
				MaxdimList=("YDim","XDim")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=4
			END_OBJECT=DataField_3
			OBJECT=DataField_4
				DataFieldName="sensor_to_ground_path_length"
				DataType=H5T_NATIVE_FLOAT
				DimList=("YDim","XDim")
				MaxdimList=("YDim","XDim")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=4
			END_OBJECT=DataField_4
			OBJECT=DataField_5
				DataFieldName="sun_zenith"
				DataType=H5T_NATIVE_FLOAT
				DimList=("YDim","XDim")
				MaxdimList=("YDim","XDim")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=4
			END_OBJECT=DataField_5
			OBJECT=DataField_6
				DataFieldName="sun_azimuth"
				DataType=H5T_NATIVE_FLOAT
				DimList=("YDim","XDim")
				MaxdimList=("YDim","XDim")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=4
			END_OBJECT=DataField_6
			OBJECT=DataField_7
				DataFieldName="beta_cloud_mask"
				DataType=H5T_NATIVE_UINT
				DimList=("YDim","XDim")
				MaxdimList=("YDim","XDim")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=4
			END_OBJECT=DataField_7
			OBJECT=DataField_8
				DataFieldName="beta_cirrus_mask"
				DataType=H5T_NATIVE_UINT
				DimList=("YDim","XDim")
				MaxdimList=("YDim","XDim")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=4
			END_OBJECT=DataField_8
			OBJECT=DataField_9
				DataFieldName="nodata_pixels"
				DataType=H5T_NATIVE_UINT
				DimList=("YDim","XDim")
				MaxdimList=("YDim","XDim")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=4
			END_OBJECT=DataField_9
			OBJECT=DataField_10
				DataFieldName="time"
				DataType=H5T_NATIVE_DOUBLE
				DimList=("YDim","XDim")
				MaxdimList=("YDim","XDim")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=4
			END_OBJECT=DataField_10
			OBJECT=DataField_11
				DataFieldName="aerosol_optical_depth"
				DataType=H5T_NATIVE_FLOAT
				DimList=("YDim","XDim")
				MaxdimList=("YDim","XDim")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=4
			END_OBJECT=DataField_11
			OBJECT=DataField_12
				DataFieldName="column_water_vapour"
				DataType=H5T_NATIVE_FLOAT
				DimList=("YDim","XDim")
				MaxdimList=("YDim","XDim")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=4
			END_OBJECT=DataField_12
			OBJECT=DataField_13
				DataFieldName="surface_reflectance_uncertainty"
				DataType=H5T_NATIVE_FLOAT
				DimList=("Band","YDim","XDim")
				MaxdimList=("Band","YDim","XDim")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=4
			END_OBJECT=DataField_13
		END_GROUP=DataField
	END_GROUP=GRID_1
END_GROUP=GridStructure
GROUP=PointStructure
END_GROUP=PointStructure
GROUP=ZaStructure
END_GROUP=ZaStructure
END

Success! You just printed the HDF-EOS5 Structural Metadata.

While it might look like a block of old computer code, this text is essentially the “blueprint” of our satellite image.

If we read through this block carefully, it reveals three critical pieces of information about our data:

1. The Shape of the Cube (Dimensions)

Look at the GROUP=Dimension section. It tells us exactly how big our 3D data cube is:

Band = 426 (We have 426 spectral wavelengths)
YDim = 724 (The image is 724 pixels tall)
XDim = 847 (The image is 847 pixels wide)

2. The Geography (Spatial Extent)

How does software know where to place this image on a map? Look near the top of the GRID_1 group:

Projection: HE5_GCTP_UTM (Universal Transverse Mercator)
Zone: ZoneCode=32 (UTM Zone 32)
Datum: SphereCode=12 (WGS84 ellipsoid)
Bounding Box: It gives the exact metric coordinates for the UpperLeftPointMtrs and LowerRightMtrs.

With just these few lines, any GIS software knows exactly where this image belongs on the Earth!

3. The Hidden Datasets (Data Fields)

Finally, look at the GROUP=DataField section. This is a list of every single dataset packaged inside this HDF5 file.

You will see our main surface_reflectance array, but look closely at what else is included:

DataField_7: beta_cloud_mask
DataField_8: beta_cirrus_mask
DataField_9: nodata_pixels

Let’s do the same thing with the Basic data product.

with h5py.File(basic_sr_path, "r") as f:
    # 1. Access the dataset
    metadata_dataset = f["HDFEOS INFORMATION/StructMetadata.0"]
    
    # 2. Open the envelope [()] and decode the bytes into a string
    raw_metadata = metadata_dataset[()].decode("utf-8")
    
    # 3. Print the entire metadata
    print("\n--- Basic Product (SwathStructure) ---\n" + "="*40)
    print(raw_metadata)


--- Basic Product (SwathStructure) ---
========================================
GROUP=SwathStructure
	GROUP=SWATH_1
		SwathName="HYP"
		GROUP=Dimension
			OBJECT=Dimension_1
				DimensionName="Band"
				Size=426
			END_OBJECT=Dimension_1
			OBJECT=Dimension_2
				DimensionName="AlongTrack"
				Size=501
			END_OBJECT=Dimension_2
			OBJECT=Dimension_3
				DimensionName="CrossTrack"
				Size=607
			END_OBJECT=Dimension_3
		END_GROUP=Dimension
		GROUP=DimensionMap
		END_GROUP=DimensionMap
		GROUP=IndexDimensionMap
		END_GROUP=IndexDimensionMap
		GROUP=GeoField
			OBJECT=GeoField_1
				GeoFieldName="Latitude"
				DataType=H5T_NATIVE_DOUBLE
				DimList=("AlongTrack","CrossTrack")
				MaxdimList=("AlongTrack","CrossTrack")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=4
			END_OBJECT=GeoField_1
			OBJECT=GeoField_2
				GeoFieldName="Longitude"
				DataType=H5T_NATIVE_DOUBLE
				DimList=("AlongTrack","CrossTrack")
				MaxdimList=("AlongTrack","CrossTrack")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=4
			END_OBJECT=GeoField_2
			OBJECT=GeoField_3
				GeoFieldName="Time"
				DataType=H5T_NATIVE_DOUBLE
				DimList=("AlongTrack")
				MaxdimList=("AlongTrack")
			END_OBJECT=GeoField_3
		END_GROUP=GeoField
		GROUP=DataField
			OBJECT=DataField_1
				DataFieldName="surface_reflectance"
				DataType=H5T_NATIVE_FLOAT
				DimList=("Band","AlongTrack","CrossTrack")
				MaxdimList=("Band","AlongTrack","CrossTrack")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=4
			END_OBJECT=DataField_1
			OBJECT=DataField_2
				DataFieldName="sensor_zenith"
				DataType=H5T_NATIVE_FLOAT
				DimList=("AlongTrack","CrossTrack")
				MaxdimList=("AlongTrack","CrossTrack")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=4
			END_OBJECT=DataField_2
			OBJECT=DataField_3
				DataFieldName="sensor_azimuth"
				DataType=H5T_NATIVE_FLOAT
				DimList=("AlongTrack","CrossTrack")
				MaxdimList=("AlongTrack","CrossTrack")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=4
			END_OBJECT=DataField_3
			OBJECT=DataField_4
				DataFieldName="sensor_to_ground_path_length"
				DataType=H5T_NATIVE_FLOAT
				DimList=("AlongTrack","CrossTrack")
				MaxdimList=("AlongTrack","CrossTrack")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=4
			END_OBJECT=DataField_4
			OBJECT=DataField_5
				DataFieldName="sun_zenith"
				DataType=H5T_NATIVE_FLOAT
				DimList=("AlongTrack","CrossTrack")
				MaxdimList=("AlongTrack","CrossTrack")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=4
			END_OBJECT=DataField_5
			OBJECT=DataField_6
				DataFieldName="sun_azimuth"
				DataType=H5T_NATIVE_FLOAT
				DimList=("AlongTrack","CrossTrack")
				MaxdimList=("AlongTrack","CrossTrack")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=4
			END_OBJECT=DataField_6
			OBJECT=DataField_7
				DataFieldName="beta_cloud_mask"
				DataType=H5T_NATIVE_UINT
				DimList=("AlongTrack","CrossTrack")
				MaxdimList=("AlongTrack","CrossTrack")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=4
			END_OBJECT=DataField_7
			OBJECT=DataField_8
				DataFieldName="beta_cirrus_mask"
				DataType=H5T_NATIVE_UINT
				DimList=("AlongTrack","CrossTrack")
				MaxdimList=("AlongTrack","CrossTrack")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=4
			END_OBJECT=DataField_8
			OBJECT=DataField_9
				DataFieldName="nodata_pixels"
				DataType=H5T_NATIVE_UINT
				DimList=("AlongTrack","CrossTrack")
				MaxdimList=("AlongTrack","CrossTrack")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=4
			END_OBJECT=DataField_9
			OBJECT=DataField_10
				DataFieldName="aerosol_optical_depth"
				DataType=H5T_NATIVE_FLOAT
				DimList=("AlongTrack","CrossTrack")
				MaxdimList=("AlongTrack","CrossTrack")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=4
			END_OBJECT=DataField_10
			OBJECT=DataField_11
				DataFieldName="column_water_vapour"
				DataType=H5T_NATIVE_FLOAT
				DimList=("AlongTrack","CrossTrack")
				MaxdimList=("AlongTrack","CrossTrack")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=4
			END_OBJECT=DataField_11
			OBJECT=DataField_12
				DataFieldName="surface_reflectance_uncertainty"
				DataType=H5T_NATIVE_FLOAT
				DimList=("Band","AlongTrack","CrossTrack")
				MaxdimList=("Band","AlongTrack","CrossTrack")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=4
			END_OBJECT=DataField_12
		END_GROUP=DataField
		GROUP=ProfileField
		END_GROUP=ProfileField
		GROUP=MergedFields
		END_GROUP=MergedFields
	END_GROUP=SWATH_1
END_GROUP=SwathStructure
GROUP=GridStructure
END_GROUP=GridStructure
GROUP=PointStructure
END_GROUP=PointStructure
GROUP=ZaStructure
END_GROUP=ZaStructure
END

Interpreting the Basic Product (SwathStructure)¶

The Basic product uses SwathStructure because it preserves the raw sensor geometry. Key elements:

1. Dimensions — In GROUP=Dimension:

Band = 426 (spectral wavelengths)
AlongTrack = 501 (pixels along the satellite track)
CrossTrack = 607 (pixels across the track)

2. Geolocation — Instead of UTM coordinates, Basic uses GROUP=GeoField:

Latitude and Longitude (2D arrays per pixel)
Time (1D along the track)

3. Data Fields — Same science and quality layers as Ortho, but dimensioned by AlongTrack and CrossTrack.

Summary & Next Steps¶

You now know how to extract the physical data and read the geographic metadata.

In Module 3: Data Cleaning & Preprocessing, we are going to extract the cloud_mask and nodata layers you just discovered and use them to build an automated cleaning pipeline.

See you there!