Running Inference¶

Run predictions on videos and label files.

predict is the unified inference command

sleap-nn predict is the canonical command for inference — it runs both trained checkpoints and exported ONNX/TensorRT models (auto-detected), and supports tracking. sleap-nn track is the older legacy pipeline (still supported), and sleap-nn infer is a deprecated alias for predict that emits a DeprecationWarning. Prefer predict for all new workflows.

Using uv workflow

If using uvx, no installation needed
If using uv sync, prefix commands with uv run:
```
uv run sleap-nn predict ...
```

Quick Start¶

TL;DR - Just want predictions?

# Single model (single-instance or bottom-up)
sleap-nn predict -i video.mp4 -m models/my_model/

# Top-down (two models required)
sleap-nn predict -i video.mp4 -m models/centroid/ -m models/centered_instance/

Output: video.mp4.predictions.slp

If results aren't great, try these common fixes:

Too many false detections? Add --filter_min_instance_score 0.3
Duplicate detections on same animal? Add --filter_overlapping
Missing detections? Lower --peak_threshold 0.1
Known number of animals? Add --max_instances N
Want tracking IDs? Add --tracking

Basic Inference¶

sleap-nn predict --data_path video.mp4 --model_paths models/my_model/

Output: video.mp4.predictions.slp

See the CLI Reference for all available parameters.

Running Exported Models (ONNX/TensorRT)¶

sleap-nn predict runs both trained checkpoints and exported ONNX/TensorRT models through the same command. If --model_paths/-m points to an export directory (one containing model.onnx or model.trt, written by sleap-nn export), predict auto-detects it and routes to the exported-model runtime instead of loading a checkpoint:

# Auto-detected export directory
sleap-nn predict -i video.mp4 -m exports/my_model/ -o predictions.slp

# Pick the runtime explicitly
sleap-nn predict -i video.mp4 -m exports/my_model/ --runtime tensorrt

Runtime	Behavior
`auto` (default)	Prefer TensorRT, fall back to ONNX
`onnx`	Force the ONNX runtime
`tensorrt`	Force the TensorRT runtime

--runtime is ignored for trained checkpoints. Most other options (device, batch size, filters, tracking) apply to exported models too. For exporting models and runtime details, see the Export guide.

Viewing Results¶

After inference, you'll have a .slp file containing predictions. To view them:

Option 1: Open in SLEAP GUI

sleap predictions.slp

This opens the predictions in the SLEAP labeling interface where you can visualize skeletons overlaid on video frames.

Option 2: Quick inspection with Python

import sleap_io as sio

labels = sio.load_slp("video.mp4.predictions.slp")
print(f"Frames with predictions: {len(labels)}")
print(f"Total instances: {sum(len(lf.instances) for lf in labels)}")

# Check a specific frame
lf = labels[0]
print(f"Frame {lf.frame_idx}: {len(lf.instances)} instances")
for inst in lf.instances:
    print(f"  Score: {inst.score:.3f}, Points: {inst.numpy().shape}")

Option 3: Export to analysis formats

# Export to Analysis HDF5 for downstream analysis
labels.export("predictions.analysis.h5")

# Or convert to CSV/DataFrames
# See sleap-io documentation for more export options

Essential Parameters¶

Parameter	Description	Values	Default
`--data_path` / `-i`	Video or labels file	`PATH`	Required
`--model_paths` / `-m`	Model dir, or its `best.ckpt` / `training_config.yaml` (repeat for top-down)	`PATH`	Required*
`--output_path` / `-o`	Output file path	`PATH`	`<input>.predictions.slp`
`--device` / `-d`	Compute device	`auto`, `cuda`, `cuda:0`, `cpu`, `mps`	`auto`
`--batch_size` / `-b`	Frames per batch	`INT`	`4`
`--max_instances` / `-n`	Max instances per frame (forward pass only)	`INT`	`None`
`--tracking` / `-t`	Enable tracking	Flag	`False`
`--peak_threshold`	Min confidence for peaks	`FLOAT`	`0.2`

*Not required for track-only mode.

Each --model_paths entry may be a model directory, or a path to that model's best.ckpt or training_config.yaml/.json file — all three resolve to the model directory and load best.ckpt. (Pointing at a different checkpoint such as last.ckpt still loads best.ckpt and warns; use --backbone_ckpt_path / --head_ckpt_path to load a specific checkpoint.)

Device selection

The --device parameter accepts:

auto - Automatically detect best available device
cuda - Use default CUDA GPU
cuda:0, cuda:1, etc. - Use a specific GPU by index
cpu - Use CPU
mps - Use Apple Metal (macOS)

Max instances

The --max_instances / -n parameter is only applied during the model forward pass. When running in track-only mode (without model inference), this parameter has no effect since no peak detection is performed.

For all parameters including image pre-processing and data selection options, see the CLI Reference.

Model Types¶

Single Instance¶

For videos with exactly one animal:

sleap-nn predict -i video.mp4 -m models/single_instance/

How parameters apply:

--peak_threshold: Keypoints below threshold become NaN (instance still exists, but with missing keypoints)
--max_instances: Not applicable (always outputs one instance)
Post-processing filters: Generally not needed, but can filter out frames with poor detection

Bottom-Up¶

For multi-animal videos using part affinity fields:

sleap-nn predict -i video.mp4 -m models/bottomup/

How parameters apply:

--peak_threshold: Peaks below threshold are not detected and won't be used for PAF grouping. Lower values detect more peaks but may create spurious instances.
--max_instances: Limits instances after PAF grouping. Keeps the top N instances by instance score.
Post-processing filters: Particularly useful for bottom-up models to clean up the output:
- Use overlap filter (--filter_overlapping) to remove duplicate detections
- Use node count filter to remove partial instances from grouping errors

When max_instances applies in bottom-up

In bottom-up models, --max_instances is applied after PAF grouping assembles instances (not during peak detection). All peaks are detected, PAF grouping creates instances, then the top N instances by score are kept.

Top-Down¶

For multi-animal videos using centroid + centered instance approach:

sleap-nn predict -i video.mp4 \
    -m models/centroid/ \
    -m models/centered_instance/

How parameters apply:

--peak_threshold: Applies to both the centroid model and the centered instance model, but behaves differently:
- Centroid model: Centroids below threshold are removed entirely (no instance created)
- Centered instance model: Keypoints below threshold become NaN (instance still exists, but with missing keypoints)
--max_instances: Limits the number of centroids (and thus instances). Takes the top N centroids by confidence score.
Post-processing filters: Apply to assembled instances after centered instance prediction

Controlling instance count in top-down models

You have two ways to control how many instances are detected:

--max_instances N: Hard limit on centroids. Only the top N highest-confidence centroids generate instances.
--peak_threshold: Soft limit. Centroids below this confidence are not detected.

Use --max_instances when you know exactly how many animals are in the video. Use --peak_threshold when the number varies.

Avoiding NaN keypoints in top-down predictions

When --peak_threshold is too high, good instances may have some keypoints set to NaN (below threshold).

Solution: Keep --peak_threshold low enough to detect all keypoints, then use --filter_min_instance_score to remove low-quality instances entirely. This preserves complete skeletons for good instances.

# Instead of raising peak_threshold (which creates NaN keypoints)
sleap-nn predict -i video.mp4 -m models/centroid/ -m models/ci/ \
    --peak_threshold 0.1 \
    --filter_min_instance_score 0.3

Bottom-Up Segmentation¶

Bottom-up segmentation models predict a per-instance mask (plus instance-center heatmap/offsets). The mask-specific prediction knobs are:

--fg_threshold (default 0.5): foreground probability threshold for binarizing the segmentation map.
--min_mask_area (default 0): drop masks smaller than this many original-image pixels (over-segmentation suppression). The filter is measured at output-stride resolution and converted from original-pixel units, so the threshold means the same thing regardless of stride.
--center_nms_kernel (default 3): odd window for instance-center peak NMS; larger merges nearby duplicate centers.
--mask_cleanup / --no-mask_cleanup (default off): keep only each mask's largest connected component and fill interior holes (despeckle).
--mask_cleanup_radius (default 0): when --mask_cleanup is set, additionally apply a morphological open→close with an elliptical kernel of this radius (output-stride pixels) before keep-largest-CC. 0 keeps the keep-largest+fill behavior. Useful for boundary noise; for disconnected speckle/fragments, keep-largest-CC alone is usually sufficient.

Mask encoding (output-stride by default). Predicted masks are stored at the model's output-stride resolution, carrying a sio scale/offset that maps mask coordinates to image pixels (image_coord = mask_coord / scale + offset). This is ~stride× smaller on disk and lossless at model resolution — the stored mask is the model's native output. Decode a mask to the original image grid with sio's mask.resampled(*mask.image_extent).data (eval, the training data loader, and tracking do this automatically). Note image_extent can differ from the true frame size by ±1 px (the mask resolution is round(orig * scale)), so clamp to the real frame size when indexing an image.

--full_res_masks (default off): encode masks at full original resolution instead (legacy behavior). Only needed for external consumers that read mask.data assuming original resolution.

Polygon output (interop). For polygon-based interop you can emit a Douglas-Peucker-simplified sio.PredictedROI alongside (or instead of) the RLE mask:

--mask_output (default mask): mask (RLE masks only), polygon (simplified ROIs only), or both (exact mask + simplified ROI). The stored mask is always exact — polygon simplification applies only to the emitted ROI, so eval/tracking are unaffected.
--polygon_epsilon (default 0.01): simplification tolerance as a fraction of each contour's perimeter (larger = coarser).

Polygon output is CPU-heavy on noisy masks — pair it with --mask_cleanup

Building a polygon walks the mask's boundary, so its cost scales with the number of RLE runs. A clean single-blob mask polygonizes in ~10 ms; a fragmented/speckled mask (thousands of runs) can be ~10× slower per mask and dominate inference time. --mask_output polygon/both is therefore best paired with --mask_cleanup (keep-largest-CC removes the speckle, so each instance is a single component). The default --mask_output mask does no polygon work and is cheaper than the pre-#618 behavior (masks are encoded at output-stride resolution, not upsampled to full resolution first). Memory for the default path is unaffected.

# Smaller .slp via stride encoding (default) + despeckle + a polygon ROI for interop
sleap-nn predict -i video.mp4 -m models/bottomup_segmentation/ \
    --mask_cleanup --mask_cleanup_radius 2 \
    --mask_output both --polygon_epsilon 0.02

Filtering Instances¶

Post-processing filters (node count, confidence score, overlap NMS) remove low-quality or duplicate predictions before tracking. See the dedicated guide:

Post-Processing Filters

Processing Order¶

Deep dive

This section explains the internal pipeline in detail. You don't need to read this to use inference - the Quick Start and Troubleshooting sections cover most use cases. Read on if you want to understand exactly how parameters interact.

When running inference, operations are applied in a specific order. Understanding this order helps you choose the right parameters to control your predictions.

Overview¶

1. Model Forward Pass
   ├── peak_threshold: Filter low-confidence peaks
   └── max_instances: Limit instances by score
       ├── Top-down: limits centroids before instance prediction
       └── Bottom-up: limits instances after PAF grouping

2. Post-processing Filters (in order)
   ├── Node count filter
   ├── Confidence score filter
   └── Overlap filter (NMS)

3. Tracking
   └── Instance identity assignment across frames

Step 1: Model Forward Pass¶

The first filtering happens during the model's peak detection phase.

--peak_threshold (default: 0.2)

Filters out peaks with confidence values below this threshold
Applied during peak detection, before instances are assembled
Effect depends on what's being detected:
- Centroids (top-down): Below-threshold centroids are removed — no instance is created
- Keypoints: Below-threshold keypoints become NaN — instance exists but with missing points

--max_instances (default: unlimited)

Limits the number of instances per frame based on confidence score
Keeps the top N highest-scoring instances, discards the rest
Top-down: Limits centroids before centered instance model runs
Bottom-up: Limits instances after PAF grouping assembles them
Single instance: Not applicable (always outputs one instance)

Step 2: Post-processing Filters¶

After instances are assembled, post-processing filters are applied in this order:

Node Count Filter → removes instances with too few keypoints
Confidence Score Filter → removes low-confidence instances
Overlap Filter (NMS) → removes duplicate detections

See Post-Processing Filters for detailed parameter documentation and examples.

Step 3: Tracking¶

After filtering, tracking assigns consistent identities across frames. See Tracking for details.

Why filtering happens before tracking

Filtering is applied before tracking to prevent spurious track creation. If low-quality instances were tracked first and then filtered out, their track IDs would be lost, causing track switches in subsequent frames.

In track-only mode (no model paths), filtering is still applied before tracking on existing predictions.

Python API¶

The unified entry point is sleap_nn.inference.predict — one call handles checkpoints and exported models and returns a sio.Labels:

from sleap_nn.inference import predict

# Trained checkpoint(s)
labels = predict(
    "video.mp4",
    model_paths=["models/my_model/"],
    output_path="predictions.slp",   # also writes the .slp
)

# Exported ONNX/TensorRT directory
labels = predict("video.mp4", export_dir="exports/my_model/", runtime="auto")

For the reusable Predictor class, streaming, raw Outputs, filtering, and tracking from Python, see the Inference API guide.

Legacy run_inference

from sleap_nn.legacy_predict import run_inference is the older legacy-pipeline entry point (it backs sleap-nn track). Prefer sleap_nn.inference.predict for new code.

Provenance Metadata¶

Output files include metadata about how predictions were generated:

import sleap_io as sio

labels = sio.load_slp("predictions.slp")
provenance = labels.provenance

print(f"sleap-nn version: {provenance.get('sleap_nn_version')}")
print(f"Model type: {provenance.get('model_type')}")
print(f"Runtime: {provenance.get('runtime_sec')}s")

Recorded Information¶

Category	Fields
Timestamps	`timestamp_start`, `timestamp_end`, `runtime_sec`
Versions	`sleap_nn_version`, `sleap_io_version`, `torch_version`
Model	`model_paths`, `model_type`, `head_type`
Input	`source_path`, `source_video_paths`
Config	`peak_threshold`, `batch_size`, `max_instances`
System	`device`, `python_version`, `cuda_version`, `gpu_names`

Legacy SLEAP Model Support¶

Run inference with SLEAP <=v1.4 models (UNet only):

sleap-nn predict -i video.mp4 -m /path/to/sleap_model/

The directory should contain: - best_model.h5 - training_config.json

UNet only

Only UNet backbone models from SLEAP ≤1.4 are supported.

Troubleshooting¶

No Predictions / Empty Output¶

Getting zero predictions on all frames

Possible causes and solutions:

Peak threshold too high - The model is detecting peaks but they're being filtered out
```
sleap-nn predict -i video.mp4 -m models/ --peak_threshold 0.05
```
Wrong model type - Using a single-instance model on multi-animal video (or vice versa)
Check your training config to confirm model type matches your data
Preprocessing mismatch - Video has different properties than training data
Check if training used grayscale vs RGB: --ensure_grayscale or --ensure_rgb
Check if training used different input scaling: --input_scale 0.5
Model didn't train properly - Check training loss curves and validation metrics

Getting predictions on some frames but not others

This is usually normal - frames without confident detections won't have predictions.

To get predictions on more frames:

sleap-nn predict -i video.mp4 -m models/ --peak_threshold 0.1

To keep empty frames in output (useful for analysis):

# Empty frames are kept by default; use --no_empty_frames to remove them

Too Many Predictions¶

Getting many false positive detections

Solution 1: Filter by instance confidence score

sleap-nn predict -i video.mp4 -m models/ --filter_min_instance_score 0.3

Solution 2: Filter by number of visible keypoints

sleap-nn predict -i video.mp4 -m models/ --filter_min_visible_node_fraction 0.5

Solution 3: If you know the exact number of animals

sleap-nn predict -i video.mp4 -m models/ --max_instances 3

Getting duplicate detections on the same animal

Enable overlap filtering with NMS:

# Basic overlap filter
sleap-nn predict -i video.mp4 -m models/ --filter_overlapping

# More aggressive filtering
sleap-nn predict -i video.mp4 -m models/ \
    --filter_overlapping \
    --filter_overlapping_threshold 0.5

# Pose-aware filtering (better for overlapping animals)
sleap-nn predict -i video.mp4 -m models/ \
    --filter_overlapping \
    --filter_overlapping_method oks

Too Few Predictions¶

Missing animals that are clearly visible

Solution 1: Lower the peak threshold

sleap-nn predict -i video.mp4 -m models/ --peak_threshold 0.1

Solution 2: For top-down models, check if centroids are being detected - The centroid model might be missing animals - Try lowering peak threshold significantly: --peak_threshold 0.05

Predictions have missing keypoints (NaN values)

This happens when individual keypoint confidence is below the peak threshold.

Solution: Lower peak threshold and filter by instance score instead

sleap-nn predict -i video.mp4 -m models/ \
    --peak_threshold 0.1 \
    --filter_min_instance_score 0.3

This detects more keypoints while still removing low-quality instances.

Wrong Predictions¶

Predictions are in the wrong location / shifted

Possible causes:

Input scaling mismatch - Training used different scale than inference

# Check training config for scale value, then match it
sleap-nn predict -i video.mp4 -m models/ --input_scale 0.5

Crop size mismatch (top-down only)

# Check training config for crop_hw value
sleap-nn predict -i video.mp4 -m models/ --crop_size 256

Skeletons are jumbled / keypoints assigned to wrong body parts

This usually indicates a model training issue rather than inference settings.

Check if training data had consistent labeling
Verify skeleton definition matches between training and inference
Consider retraining with more data or data augmentation

Performance Issues¶

Out of GPU memory (CUDA OOM)

Solution 1: Reduce batch size

sleap-nn predict -i video.mp4 -m models/ --batch_size 2

Solution 2: Use CPU (slower but no memory limit)

sleap-nn predict -i video.mp4 -m models/ --device cpu

Solution 3: Process fewer frames at once

sleap-nn predict -i video.mp4 -m models/ --frames 0-1000

Inference is very slow

Check GPU is being used:

sleap-nn system  # Verify CUDA is available
sleap-nn predict -i video.mp4 -m models/ --device cuda

Increase batch size (if memory allows):

sleap-nn predict -i video.mp4 -m models/ --batch_size 16

Tune opt-in flags — see the Inference Performance guide for FP16 / torch.compile / paf_workers recommendations backed by benchmark data on each model type.

Running Inference¶

Quick Start¶

Basic Inference¶

Running Exported Models (ONNX/TensorRT)¶

Viewing Results¶

Essential Parameters¶

Model Types¶

Single Instance¶

Bottom-Up¶

Top-Down¶

Bottom-Up Segmentation¶

Filtering Instances¶

Processing Order¶

Overview¶

Step 1: Model Forward Pass¶

Step 2: Post-processing Filters¶

Step 3: Tracking¶

Python API¶

Provenance Metadata¶

Recorded Information¶

Legacy SLEAP Model Support¶

Troubleshooting¶

No Predictions / Empty Output¶

Too Many Predictions¶

Too Few Predictions¶

Wrong Predictions¶

Performance Issues¶

Next Steps¶