Preprocessing Hyperspectral Data with TanagerSpec¶

Before you can confidently analyze hyperspectral scenes, it is essential to properly clean and prepare your data. The Tanager-1 HDF5-EOS format includes several built-in quality masks—such as cirrus, cloud, and no-data masks—to help you easily filter out invalid pixels across your scene.

TanagerSpec simplifies this data preparation by offering a streamlined suite of preprocessing tools designed specifically for hyperspectral workflows.

What You Will Learn¶

In this lesson, you will learn how to leverage these integrated features to ensure your data is highly reliable and ready for downstream analysis. Specifically, we will cover:

• Wavelength Exclusion (Dropping Bands): Removing specific noisy channels, such as dense atmospheric water absorption bands.
• Quality Masking: Filtering out invalid pixels, cloud cover, or sensor artifacts using the built-in masks.
• Reflectance Clipping: Constraining surface reflectance values to valid physical ranges.
• Denoising: Smoothing out spectral noise to retrieve a cleaner signal.

Let's begin by initializing our workspace and loading our scene!

Workspace Setup & TanagerSpec Initialization¶

Since each lesson operates as a standalone notebook, we first need to re-establish our directory structure and ensure our Tanager hyperspectral scene is available.

Note

If you just completed Lesson 1 and are running this notebook locally, you likely already have the scene downloaded in your data/ folder. However, if you are running this notebook on Google Colab, the environment is temporary, and you will need to download the scene again. To save time and bandwidth where possible, the code below checks if the cloudy_ortho_sr_scene.h5 file already exists. If it does, TanagerSpec will instantly connect to it; if not, it will automatically download the file for you from the Open STAC catalog.

In [ ]:

## Unhide cell for package install if needed

# %pip install tanagerspec

## Unhide cell for package install if needed # %pip install tanagerspec

Note

We changed the previous scene from a clean scene to a contaminated scene with clouds, which helps illustrate the potential of TanagerSpec tools for working with cloudy scenes.

In [ ]:

# Standard library imports
from datetime import datetime               # For timestamping output folders
from pathlib import Path                    # For handling file and directory paths

# TanagerSpec package imports
from tanagerspec import download_scene      # Utility function to download a Tanager scene
from tanagerspec import TanagerSpec         # Main class for interacting with Tanager hyperspectral data

# third party imports
import matplotlib.pyplot as plt
import numpy as np

# 1. Setup Data and Output Directories
DATA_DIR = Path("data")
DATA_DIR.mkdir(parents=True, exist_ok=True)

timestamp = datetime.now().strftime("%Y%m%d_%H")
OUTPUT_DIR = Path(f"outputs_{timestamp}")
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)

# Figure directory
FIGURE_DIR = OUTPUT_DIR / "figures"
FIGURE_DIR.mkdir(parents=True, exist_ok=True)

# 2. Define Scene Parameters
URL = 'https://storage.googleapis.com/open-cogs/planet-stac/tanager1-release2-core-imagery/ortho_sr_hdf5/20250929_085922_06_4001_ortho_sr_hdf5.h5'
FILE_NAME = "cloudy_ortho_sr_scene.h5"
target_path = DATA_DIR / FILE_NAME

# 3. Smart Download: Check if file exists before downloading
if target_path.exists():
    print(f"Scene already exists locally at: {target_path}. Skipping download.")
    TANAGER_SCENE_PATH = target_path
else:
    print("Local scene not found. Downloading from STAC...")
    TANAGER_SCENE_PATH = download_scene(
        URL,          # URL of the HDF5 file to download 
        target_path   # Destination path to save the file
    )

# 4. Verify Final Path
if TANAGER_SCENE_PATH is not None and Path(TANAGER_SCENE_PATH).exists():
    print(f"Ready for preprocessing with scene: {TANAGER_SCENE_PATH}")
else:
    print("Error: Scene file is missing or download failed.")

# 5. Initialize the TanagerSpec object using the verified scene path
tanager_ortho_sr = TanagerSpec.from_file(TANAGER_SCENE_PATH)

print("TanagerSpec object successfully initialized!")

# Standard library imports from datetime import datetime # For timestamping output folders from pathlib import Path # For handling file and directory paths # TanagerSpec package imports from tanagerspec import download_scene # Utility function to download a Tanager scene from tanagerspec import TanagerSpec # Main class for interacting with Tanager hyperspectral data # third party imports import matplotlib.pyplot as plt import numpy as np # 1. Setup Data and Output Directories DATA_DIR = Path("data") DATA_DIR.mkdir(parents=True, exist_ok=True) timestamp = datetime.now().strftime("%Y%m%d_%H") OUTPUT_DIR = Path(f"outputs_{timestamp}") OUTPUT_DIR.mkdir(parents=True, exist_ok=True) # Figure directory FIGURE_DIR = OUTPUT_DIR / "figures" FIGURE_DIR.mkdir(parents=True, exist_ok=True) # 2. Define Scene Parameters URL = 'https://storage.googleapis.com/open-cogs/planet-stac/tanager1-release2-core-imagery/ortho_sr_hdf5/20250929_085922_06_4001_ortho_sr_hdf5.h5' FILE_NAME = "cloudy_ortho_sr_scene.h5" target_path = DATA_DIR / FILE_NAME # 3. Smart Download: Check if file exists before downloading if target_path.exists(): print(f"Scene already exists locally at: {target_path}. Skipping download.") TANAGER_SCENE_PATH = target_path else: print("Local scene not found. Downloading from STAC...") TANAGER_SCENE_PATH = download_scene( URL, # URL of the HDF5 file to download target_path # Destination path to save the file ) # 4. Verify Final Path if TANAGER_SCENE_PATH is not None and Path(TANAGER_SCENE_PATH).exists(): print(f"Ready for preprocessing with scene: {TANAGER_SCENE_PATH}") else: print("Error: Scene file is missing or download failed.") # 5. Initialize the TanagerSpec object using the verified scene path tanager_ortho_sr = TanagerSpec.from_file(TANAGER_SCENE_PATH) print("TanagerSpec object successfully initialized!")

Establishing a Baseline with .info()¶

Before we apply any preprocessing methods, let's run .info() to capture the current baseline state of our hyperspectral scene.

By taking a quick "before" snapshot, we can clearly see the impact of our data cleaning steps. Pay close attention to the visualization and the Bands Dropped status at the bottom. As we begin excluding noisy wavelengths in the next steps, these values will dynamically update to reflect our changes.

In [ ]:

# Check the baseline state before preprocessing
tanager_ortho_sr.info(
    # save_png=FIGURE_DIR / "ortho_sr_info.png" # Uncomment to save the plot as a PNG
)

# Check the baseline state before preprocessing tanager_ortho_sr.info( # save_png=FIGURE_DIR / "ortho_sr_info.png" # Uncomment to save the plot as a PNG )

All methods below follow the same call pattern introduced in Lesson 15 — How TanagerSpec Method Calls Are Structured (object.method(...) or object.accessor.method(...)).

Excluding Specific Wavelengths with .drop_bands()¶

Depending on your specific research question or application, you might only be interested in focusing on targeted regions of the electromagnetic spectrum. The .drop_bands() method is a highly flexible utility that lets you exclude any range of wavelengths that are not relevant to your current analysis.

You can filter out these unwanted intervals by passing a list of wavelength ranges (in nanometers) defined by their start and end points: [(wavelength_start, wavelength_end), ...].

How It Works Behind the Scenes¶

When you drop bands, TanagerSpec does not delete the data from your underlying array. Instead, it updates an internal mask (good_wavelengths), marking those specific channels as invalid. This preserves the original data matrix shape while ensuring that downstream plotting, masking, and analysis algorithms automatically ignore the excluded wavelengths.

Note

A quick note on the ranges used below: they are illustrative chosen to show the syntax, not as a recommended recipe. Which bands you drop depends entirely on your science question. In practice, the most common candidates for exclusion are the atmospheric water-absorption bands (around 1350–1450 nm and 1800–1950 nm), where atmospheric water vapour swallows most of the signal and the reflectance retrieval is unreliable. You might also drop the noisy detector edges at the extreme short- and long-wavelength ends of the sensor's range.

Let's drop a couple of custom wavelength intervals to focus our analysis:

In [ ]:

# Exclude specific wavelength intervals (in nanometers) that are not of interest
tanager_ortho_sr.drop_bands(
    [
        (400, 500),    # Example: Excluding the short-visible / blue edge region
        (1000, 1250),  # Example: Excluding a targeted transition region in the NIR
    ]
)

print("Specified wavelength bands have been successfully excluded from active analysis.")

# Exclude specific wavelength intervals (in nanometers) that are not of interest tanager_ortho_sr.drop_bands( [ (400, 500), # Example: Excluding the short-visible / blue edge region (1000, 1250), # Example: Excluding a targeted transition region in the NIR ] ) print("Specified wavelength bands have been successfully excluded from active analysis.")

Verifying Excluded Bands with .info()¶

Now that we have applied our custom wavelength filters, let's run .info() a second time to see how our dataset's diagnostic profile has changed.

By comparing this output to our baseline snapshot, you will immediately notice a few key updates:

• Bands Dropped Status: The indicator at the bottom has switched from False to True.
• Excluded Ranges Listed: The summary log now explicitly tracks the exact cumulative wavelength intervals we removed from active analysis.

In [ ]:

tanager_ortho_sr.info(
    # save_png=FIGURE_DIR / "ortho_sr_info.png" # Uncomment to save the plot as a PNG
)

tanager_ortho_sr.info( # save_png=FIGURE_DIR / "ortho_sr_info.png" # Uncomment to save the plot as a PNG )

Note

For the remainder of the lesson, we need to re-initialize the TanagerSpec to restore the data to its original state.

Applying Quality Masks and Reflectance Clipping with .preprocess()¶

Even though our data has already been processed to surface reflectance, a major challenge in remote sensing is dealing with environmental contamination like cloud cover and invalid pixels. Fortunately, Tanager-1 data comes equipped with built-in quality masks to help manage this.

The .preprocess() method provides a streamlined, one-step command to clean your spatial pixels before running advanced analyses or visualizations:

• Quality Masking (masking=True): Automatically reads the scene's built-in quality classification layers to identify and mask out contaminated pixels, including unmapped nodata regions, thick cloud cover, and cirrus clouds.
• Reflectance Clipping (clipping=True): Constrains all surface reflectance values to the valid physical range of 0.0 to 1.0. Surface reflectance is a ratio of reflected to incoming light, so it cannot physically fall outside this range—but the atmospheric-correction step that produces it can leave small artifacts slightly below 0 or above 1 (for example over deep shadow or bright specular surfaces). Clipping removes those out-of-range outliers so they don't skew your statistics or wash out the contrast in your colour plots.

Let's apply both steps to our scene!

In [ ]:

# Apply quality masking and physical reflectance clipping
preprocessed_cube = tanager_ortho_sr.preprocess(
    masking=True,  # Mask out nodata, cloud, and cirrus pixels
    clipping=True  # Clip surface reflectance values to the 0.0 - 1.0 range
)

print("Preprocessing complete: Quality masks applied and reflectance values clipped.")

# Apply quality masking and physical reflectance clipping preprocessed_cube = tanager_ortho_sr.preprocess( masking=True, # Mask out nodata, cloud, and cirrus pixels clipping=True # Clip surface reflectance values to the 0.0 - 1.0 range ) print("Preprocessing complete: Quality masks applied and reflectance values clipped.")

Let's compare before and after preprocess to see how it works and how invalid pixels are masked.

In [ ]:

# Reload the original Tanager SpecCube from file to ensure a fresh start for comparison
tanager_ortho_sr = TanagerSpec.from_file(TANAGER_SCENE_PATH)

# Set the visualization preset; here, we use 'true_color' for natural color rendering
preset = "true_color"

# Generate the RGB image before preprocessing (i.e., before masking/clipping)
rgb_before = tanager_ortho_sr.plot.rgb(preset=preset, plot=False)

# Apply preprocessing: masks out cloud/nodata/cirrus and clips reflectance to [0, 1]
tanager_ortho_sr.preprocess(masking=True, clipping=True)

# Generate the RGB image after preprocessing
rgb_after = tanager_ortho_sr.plot.rgb(preset=preset, plot=False)

def show_image(ax, img, title):
    # Create a mask for any pixels containing NaN in any channel (invalid/masked out)
    nan_mask = np.isnan(img).any(axis=2)

    # Show the original RGB image, replacing NaNs with black (0,0,0)
    ax.imshow(np.nan_to_num(img, nan=0))

    # Create an overlay image: where nan_mask is True, show a yellow transparent mask
    overlay = np.zeros((*nan_mask.shape, 4))  # RGBA last dim
    overlay[nan_mask] = [1, 1, 0, 0.7]  # Yellow with alpha=0.7 (highlights masked pixels)
    ax.imshow(overlay)

    # Set figure aesthetics
    ax.set_title(title)
    ax.axis("off")

# Create a figure with 2 subplots (side-by-side) to compare before/after
fig, axes = plt.subplots(1, 2, figsize=(14, 6))

# Show 'before' image in left panel, 'after' in right panel
show_image(axes[0], rgb_before, f"Before masking")
show_image(axes[1], rgb_after, "After masking")

# Make layout tight and display
plt.tight_layout()
plt.show()

# Reload the original Tanager SpecCube from file to ensure a fresh start for comparison tanager_ortho_sr = TanagerSpec.from_file(TANAGER_SCENE_PATH) # Set the visualization preset; here, we use 'true_color' for natural color rendering preset = "true_color" # Generate the RGB image before preprocessing (i.e., before masking/clipping) rgb_before = tanager_ortho_sr.plot.rgb(preset=preset, plot=False) # Apply preprocessing: masks out cloud/nodata/cirrus and clips reflectance to [0, 1] tanager_ortho_sr.preprocess(masking=True, clipping=True) # Generate the RGB image after preprocessing rgb_after = tanager_ortho_sr.plot.rgb(preset=preset, plot=False) def show_image(ax, img, title): # Create a mask for any pixels containing NaN in any channel (invalid/masked out) nan_mask = np.isnan(img).any(axis=2) # Show the original RGB image, replacing NaNs with black (0,0,0) ax.imshow(np.nan_to_num(img, nan=0)) # Create an overlay image: where nan_mask is True, show a yellow transparent mask overlay = np.zeros((*nan_mask.shape, 4)) # RGBA last dim overlay[nan_mask] = [1, 1, 0, 0.7] # Yellow with alpha=0.7 (highlights masked pixels) ax.imshow(overlay) # Set figure aesthetics ax.set_title(title) ax.axis("off") # Create a figure with 2 subplots (side-by-side) to compare before/after fig, axes = plt.subplots(1, 2, figsize=(14, 6)) # Show 'before' image in left panel, 'after' in right panel show_image(axes[0], rgb_before, f"Before masking") show_image(axes[1], rgb_after, "After masking") # Make layout tight and display plt.tight_layout() plt.show()

Note

As you noticed that there are remaining clouds in the scene after running preprocessing to mask the invalid pixels, it is because the current cloud mask is in beta.

Reducing Spectral Noise with .denoise() (PCA Reconstruction)¶

Even after excluding unneeded bands and applying spatial masks, hyperspectral data can still contain random, uncorrelated sensor noise. The .denoise() method applies Principal Component Analysis (PCA) reconstruction to smooth out this noise while preserving your core spectral signatures.

How PCA Denoising Works¶

The algorithm projects your high-dimensional hyperspectral cube into a lower-dimensional subspace defined by its most dominant spectral features, and then reconstructs the full dataset. Because random noise does not correlate strongly across bands, it is effectively stripped out during the reconstruction.

You control the aggressiveness of this filter using the n_components parameter:

• Increase n_components if your spectral curves look overly smoothed or if you are worried about losing subtle absorption features.
• Decrease n_components if noticeable random noise still remains in the signal.

💡 Best Practice: It is highly recommended to pair this step with visualization plotting so you can directly inspect the "before and after" spectral curves of specific pixels to verify your component selection.

Determining the Optimal Number of Components¶

Before applying the .denoise() method, you need to decide how many principal components to retain. A practical approach is to first extract the components using TanagerSpec's dimensionality reduction module, visualize them, and then select the optimal number based on where the signal drops off into random noise.

You can extract the initial components for evaluation like this:

pca_components = tanager_ortho_sr.analysis.dim_reduction(method="PCA", n_components=6)

By plotting this pca_components array using Matplotlib, you can visually analyze the spatial structure within each individual component. Once you identify the threshold where the components stop showing meaningful structure and begin showing primarily noise (often around the 3rd to 5th component for many scenes), you can set your n_components for the final denoising step.

Let's run PCA with 6 components and analyze them to determine the best number of components for denoising reconstruction!

In [ ]:

# Reload data for a fresh start
tanager_ortho_sr = TanagerSpec.from_file(TANAGER_SCENE_PATH)

# Extract the first 6 PCA components
pca_components = tanager_ortho_sr.analysis.dim_reduction(method="PCA", n_components=6, plot=False, save_png=False)

fig, axes = plt.subplots(2, 3, figsize=(20, 14))
fig.suptitle("Spatial Maps of the First 6 PCA Components", fontsize=22, y=1.02)

for i in range(6):
    ax = axes.flat[i]
    component = pca_components[:, :, i]
    vmin, vmax = np.percentile(component, (2, 98))
    im = ax.imshow(component, cmap="viridis", vmin=vmin, vmax=vmax)
    ax.set_title(f"PCA Component {i+1}", fontsize=15)
    ax.axis("off")
    plt.colorbar(im, ax=ax, fraction=0.04, pad=0.01)

plt.tight_layout()
plt.show()

# Reload data for a fresh start tanager_ortho_sr = TanagerSpec.from_file(TANAGER_SCENE_PATH) # Extract the first 6 PCA components pca_components = tanager_ortho_sr.analysis.dim_reduction(method="PCA", n_components=6, plot=False, save_png=False) fig, axes = plt.subplots(2, 3, figsize=(20, 14)) fig.suptitle("Spatial Maps of the First 6 PCA Components", fontsize=22, y=1.02) for i in range(6): ax = axes.flat[i] component = pca_components[:, :, i] vmin, vmax = np.percentile(component, (2, 98)) im = ax.imshow(component, cmap="viridis", vmin=vmin, vmax=vmax) ax.set_title(f"PCA Component {i+1}", fontsize=15) ax.axis("off") plt.colorbar(im, ax=ax, fraction=0.04, pad=0.01) plt.tight_layout() plt.show()

The results indicate that a 3-component PCA reconstruction is optimal. When a 4th component is included, noise, specifically striping artifacts, begins to appear.

Keep in mind that the optimal number of components for denoising is not fixed across all scenes. Each scene must be treated as a unique case, requiring careful analysis to determine the appropriate number of components for denoising and reconstruction.

Very Crucial

In this lesson, we used a scene with cloud cover. Because the cloud masking tool is still in beta, it did not filter out all the clouds, leaving behind contaminated pixels. This residual cloud cover negatively impacts the PCA's performance, causing noise to transfer more easily into the primary principal components.

In [ ]:

# Apply PCA-based denoising to smooth out spectral noise

n_components = 3  # Number of principal components to retain for reconstruction

denoised_cube = tanager_ortho_sr.denoise(
    n_components=n_components
    )

print(f"Denoising complete: Data reconstructed using {n_components} PCA components.")

# Apply PCA-based denoising to smooth out spectral noise n_components = 3 # Number of principal components to retain for reconstruction denoised_cube = tanager_ortho_sr.denoise( n_components=n_components ) print(f"Denoising complete: Data reconstructed using {n_components} PCA components.")

RGB before vs after denoise¶

To see how denoising improves quality and reduces noise in the hyperspectral cube, run the comparison below.

In [ ]:

# reinitialize the object
tanager_ortho_sr = TanagerSpec.from_file(TANAGER_SCENE_PATH)

preset = "true_color"

preprocessed_cube = tanager_ortho_sr.preprocess(
    masking=True,  # Mask out nodata, cloud, and cirrus pixels
    clipping=True  # Clip surface reflectance values to the 0.0 - 1.0 range
)

rgb_before = tanager_ortho_sr.plot.rgb(preset=preset, plot=False)

tanager_ortho_sr.denoise(n_components=3)

rgb_after = tanager_ortho_sr.plot.rgb(preset=preset, plot=False)

_ , axes = plt.subplots(1, 2, figsize=(14, 6), sharex=True, sharey=True)

axes[0].imshow(rgb_before)
axes[0].set_title(f"Before denoise — {preset}")
axes[0].axis("off")
axes[1].imshow(rgb_after)
axes[1].set_title("After denoise (PCA, same preset & stretch)")
axes[1].axis("off")

plt.tight_layout()
plt.show()

# reinitialize the object tanager_ortho_sr = TanagerSpec.from_file(TANAGER_SCENE_PATH) preset = "true_color" preprocessed_cube = tanager_ortho_sr.preprocess( masking=True, # Mask out nodata, cloud, and cirrus pixels clipping=True # Clip surface reflectance values to the 0.0 - 1.0 range ) rgb_before = tanager_ortho_sr.plot.rgb(preset=preset, plot=False) tanager_ortho_sr.denoise(n_components=3) rgb_after = tanager_ortho_sr.plot.rgb(preset=preset, plot=False) _ , axes = plt.subplots(1, 2, figsize=(14, 6), sharex=True, sharey=True) axes[0].imshow(rgb_before) axes[0].set_title(f"Before denoise — {preset}") axes[0].axis("off") axes[1].imshow(rgb_after) axes[1].set_title("After denoise (PCA, same preset & stretch)") axes[1].axis("off") plt.tight_layout() plt.show()

Sensitivity of PCA

Try rerunning the comparison above without applying the preprocessing steps to see the difference! This demonstrates how contaminated cloud pixels, which are treated as outliers, can significantly impact the PCA results.

⚠️ Critical Note on Memory Management:

To maximize efficiency and save RAM, both .preprocess() and .denoise() update the main hyperspectral data cube in-place. Because hyperspectral arrays are massive, TanagerSpec is intentionally designed to avoid keeping duplicate full-size cubes in memory simultaneously.

This means your underlying pixel values are directly modified. Once you execute this cell, the rest of the notebook will operate on the newly preprocessed and denoised cube rather than the raw data. If you ever need to start over or return to the original, unedited pixel values, you will need to reinitialize your TanagerSpec object from the source file.

Note

To prevent accidental over-smoothing, calling .denoise() on an already denoised object acts as a safe "no-op"—it will skip processing and display a log warning instead.

What's Next?¶

Congratulations! You have successfully learned how to clean and preprocess your Tanager hyperspectral scene inside Python.

However, depending on your workflow, you might prefer to take your raw data straight into external Geographic Information System (GIS) or remote sensing software (like QGIS, ArcGIS, or ENVI) before applying any modifications.

In the next lesson, Exporting and File Conversion, we will learn how to use built-in TanagerSpec methods to easily convert the original HDF5-EOS scene as-is into highly interoperable formats like GeoTIFF and ENVI-BIL.

Note

This conversion exports the complete dataset exactly as downloaded, without applying the band dropping, masking, or denoising steps we covered today, giving you the untouched baseline data for your external software.

See you in the next lesson!