Skip to content

Evaluation Metrics

Here, we explain the various evaluation metrics we output at the end of running inference with a trained SLEAP model. We report 6 broad categories of metrics:

Distance Metrics

This metric computes the Euclidean distance between pairs of predicted and ground-truth (gt) instances. For each instance pair, we calculate the L2 norm of the difference between the predicted and corresponding ground-truth keypoints. The following statistics are reported:

  • Avg_dist: Mean Euclidean distance across all (pred, gt) pairs.

  • Dist@k: Percentile-based distance metrics which includes the distance at the 50th, 75th, 90th, 95th, and 99th percentiles (denoted as p50, p75, p90, p95, p99).

These metrics provide insight into the distribution of how far off the predictions are from the ground-truth key-points.

Object-keypoint similarity (OKS)

This returns the mean OKS score between every pair of ground truth and predicted instance, ranging from 0 to 1.0 and 1.0 indicating a perfect match. OKS provides a measure of similarity between ground-truth and predicted pose by taking into account the instance size (scale) and node visibility.

OKS is computed by measuring the Euclidean distance between each predicted keypoint and its corresponding ground-truth keypoint. This distance is then normalized based on the scale of the object (bounding box area for the instance) and standard-deviation that defines the spread in the localization accuracy of each node. For each node, keypoint similarity is computed by taking the negative exponent of the normalized distance. Mean OKS is the average of keypoint similarities across all visible nodes.

The implementation is based off of the descriptions in: Ronch & Perona. "Benchmarking and Error Diagnosis in Multi-Instance Pose Estimation." ICCV (2017).

Percentage of Correct Keypoints (PCK) metrics

This metric measures the fraction of keypoints that fall within a certain pixel distance (threshold) from the ground-truth location. This is useful to evaluate how precise the predicted points are. The following are generated using PCK metric:

  • PCKs: PCK on each predicted instances for each node at different thresholds. (Thresholds: [1, 2, 3, ..., 10])
  • mPCK part: Mean PCK per node averaged over all predicted instances across thresholds.
  • mPCK: Mean PCK across all predicted nodes and thresholds.

Visibility metrics

This metric evaluates the visibility accuracy of the predicted nodes. It measures how well the model identifies whether the keypoint is present or missing - independent of its spatial accuracy (i.e. distance from ground-truth). This is useful for evaluating models on datasets with occlusions (NaN nodes).

The following statistics are computed across all matched instance pairs:

  • True positives (TP): Node is visible in both ground-truth and prediction.
  • False positives (FP): Node is missing in ground-truth but visible in prediction.
  • True negatives (TN): Node is missing in both ground-truth and prediction.
  • False negatives (FN): Node is visible in ground-truth but missing in prediction.
  • Precision (TP / (TP + FP)): Proportion of predicted visible nodes that are correct.
  • Recall (TP / (TP + FN)): Proportion of actual visible nodes that were correctly predicted.

VOC metrics

The following VOC-style metrics are generated using either OKS or PCK as the matching scores and a set of thresholds where a predicted instance is considered as a True Positive if its match score is greater than the threshold else it is counted as a False Positive.

  • Average Precision (AP): Average of best precisions over fixed set of recall thresholds, at each match score threshold.
  • Average Recall (AR): Maximum recall achieved at each match score threshold.
  • Mean Average Precision (mAP): Mean of average precisions across match thresholds.
  • Mean Average Recall (mAR): Mean of average recalls across match thresholds.

To learn how to generate these metrics using sleap-nn, see the Evaluation guide.

Centroid Metrics

Centroid models predict a single point (centroid) per instance rather than full pose skeletons. These models use specialized distance-based metrics that are more appropriate for point detection tasks than OKS/PCK metrics.

Prediction Matching

Predictions are matched to ground truth using the Hungarian algorithm (optimal bipartite matching) to minimize total distance. A match_threshold parameter (default: 50 pixels) determines the maximum distance for a valid match:

  • True Positive (TP): Prediction matched to GT within threshold
  • False Positive (FP): Prediction with no GT match within threshold
  • False Negative (FN): GT with no prediction match within threshold

Distance Metrics

For matched prediction-GT pairs, the following distance statistics are computed:

Metric Description
centroid_dist_avg Mean Euclidean distance (pixels)
centroid_dist_median Median distance
centroid_dist_p90 90th percentile distance
centroid_dist_p95 95th percentile distance
centroid_dist_max Maximum distance

Detection Metrics

These metrics evaluate how well the model detects instances:

Metric Formula Description
centroid_precision TP / (TP + FP) Proportion of predictions that are correct
centroid_recall TP / (TP + FN) Proportion of GT instances that were detected
centroid_f1 2 * P * R / (P + R) Harmonic mean of precision and recall

Count Metrics

Metric Description
centroid_n_tp Number of true positives
centroid_n_fp Number of false positives
centroid_n_fn Number of false negatives

When to Use

Centroid metrics are automatically used during epoch-end evaluation when training centroid models. They can also be used for post-training evaluation of centroid model predictions.

Adjusting match_threshold

The match_threshold parameter should be set based on your expected centroid accuracy:

  • Smaller threshold (e.g., 20px): Stricter matching, may undercount true positives
  • Larger threshold (e.g., 100px): More lenient, may incorrectly match distant predictions

For most applications, the default of 50 pixels works well.