evaluation

sleap_nn.evaluation

This module is to compute evaluation metrics for trained models.

Classes:

Name	Description
Evaluator	Compute the standard evaluation metrics with the predicted and the ground-truth Labels.
MatchInstance	Class to have a new structure for sio.Instance object.

Functions:

Name	Description
compute_dists	Compute Euclidean distances between matched pairs of instances.
compute_gt_centroids	Compute ground-truth centroids mirroring generate_centroids in numpy.
compute_instance_area	Compute the area of the bounding box of a set of keypoints.
compute_oks	Compute the object keypoints similarity between sets of points.
find_frame_pairs	Find corresponding frames across two sets of labels.
get_instances	Get a list of instances of type MatchInstance from the Labeled Frame.
load_metrics	Load metrics from a model folder or metrics file.
mask_cldice	Centerline Dice (clDice) between two binary masks.
match_centroids	Match predicted centroids to ground truth using Hungarian algorithm.
match_frame_pairs	Match all ground truth and predicted instances within each pair of frames.
match_instances	Match pairs of instances between ground truth and predictions in a frame.
match_masks	Match predicted masks to ground-truth masks by IoU (Hungarian).
run_evaluation	Evaluate SLEAP-NN model predictions against ground truth labels.

Evaluator

Compute the standard evaluation metrics with the predicted and the ground-truth Labels.

This class is used to calculate the common metrics for pose estimation models which includes voc metrics (with oks and pck), mOKS, distance metrics, pck metrics and visibility metrics.

Parameters:

Name	Type	Description	Default
ground_truth_instances	Labels	The sio.Labels dataset object with ground truth labels.	required
predicted_instances	Labels	The sio.Labels dataset object with predicted labels.	required
oks_stddev	float	The standard deviation to use for calculating object keypoint similarity; see compute_oks function for details.	0.025
oks_scale	Optional[float]	The scale to use for calculating object keypoint similarity; see compute_oks function for details.	None
match_threshold	float	The threshold to use when determining which instances match between ground truth and predicted frames. For match_method="oks" this is an OKS threshold; for match_method="centroid" this is a PIXEL distance threshold.	0
user_labels_only	bool	If False, predicted instances in the ground truth frame may be considered for matching.	True
match_method	str	Either "oks" (default, full-skeleton OKS matching) or "centroid" (single-point distance matching for centroid-only / single-node predictions).	'oks'
anchor_ind	Optional[int]	For match_method="centroid", the index of the GT skeleton node used to compute each ground-truth centroid (see :func:compute_gt_centroids and #586). None falls back to the NaN-ignoring mean of visible nodes.	None

Methods:

Name	Description
__init__	Initialize the Evaluator class with ground-truth and predicted labels.
detection_metrics	Compute detection metrics (precision/recall/F1) over TP/FP/FN counts.
distance_metrics	Compute the Euclidean distance error at different percentiles using the pairwise distances.
evaluate	Return the evaluation metrics.
mOKS	Return the meanOKS value.
mask_metrics	Compute mask-IoU summary statistics for match_method="mask".
mask_voc_metrics	COCO-style score-ranked mask Average Precision / Recall.
pck_metrics	Compute PCK across a range of thresholds using the pair-wise distances.
semantic_metrics	Aggregate whole-frame foreground metrics for match_method="semantic".
visibility_metrics	Compute node visibility metrics for the matched pair of instances.
voc_metrics	Compute VOC metrics for a matched pairs of instances positive pairs and false negatives.

Source code in sleap_nn/evaluation.py

class Evaluator:
    """Compute the standard evaluation metrics with the predicted and the ground-truth Labels.

    This class is used to calculate the common metrics for pose estimation models which
    includes voc metrics (with oks and pck), mOKS, distance metrics, pck metrics and
    visibility metrics.

    Args:
        ground_truth_instances: The `sio.Labels` dataset object with ground truth labels.
        predicted_instances: The `sio.Labels` dataset object with predicted labels.
        oks_stddev: The standard deviation to use for calculating object
            keypoint similarity; see `compute_oks` function for details.
        oks_scale: The scale to use for calculating object
            keypoint similarity; see `compute_oks` function for details.
        match_threshold: The threshold to use when determining which instances
            match between ground truth and predicted frames. For
            ``match_method="oks"`` this is an OKS threshold; for
            ``match_method="centroid"`` this is a PIXEL distance threshold.
        user_labels_only: If False, predicted instances in the ground truth frame may be
            considered for matching.
        match_method: Either ``"oks"`` (default, full-skeleton OKS matching) or
            ``"centroid"`` (single-point distance matching for centroid-only /
            single-node predictions).
        anchor_ind: For ``match_method="centroid"``, the index of the GT
            skeleton node used to compute each ground-truth centroid (see
            :func:`compute_gt_centroids` and #586). ``None`` falls back to the
            NaN-ignoring mean of visible nodes.

    """

    def __init__(
        self,
        ground_truth_instances: sio.Labels,
        predicted_instances: sio.Labels,
        oks_stddev: float = 0.025,
        oks_scale: Optional[float] = None,
        match_threshold: float = 0,
        user_labels_only: bool = True,
        match_method: str = "oks",
        anchor_ind: Optional[int] = None,
        exclude_predicted_instance_masks: bool = False,
    ):
        """Initialize the Evaluator class with ground-truth and predicted labels.

        ``exclude_predicted_instance_masks`` (``match_method="mask"`` only) drops
        masks linked to a ``PredictedInstance`` from the ground-truth labels, so a
        labels file that carries stray predicted instances (each of which gets a
        mask when masks are built from poses) does not treat them as ground truth.
        It is kept separate from ``user_labels_only`` (which controls the frame-pair
        filter) because mask mode disables that frame filter -- see
        :func:`run_evaluation`.
        """
        self.ground_truth_instances = ground_truth_instances
        self.predicted_instances = predicted_instances
        self.match_threshold = match_threshold
        self.oks_stddev = oks_stddev
        self.oks_scale = oks_scale
        self.user_labels_only = user_labels_only
        self.match_method = match_method
        self.anchor_ind = anchor_ind
        self.exclude_predicted_instance_masks = exclude_predicted_instance_masks
        # Populated only in centroid / mask mode.
        self.false_positives = []
        # Matched-pair IoUs, populated only in mask mode.
        self.mask_ious = np.array([])
        # Per-frame mask records + matched TP mask pairs, populated only in mask
        # mode (feed mask_voc_metrics / boundary-IoU / fragmentation / per-size).
        self._mask_frames = []
        self._matched_mask_pairs = []
        # Per-frame (iou, cldice, boundary_iou) triples, populated only in
        # match_method="semantic" (whole-frame foreground, no matching).
        self._semantic_rows = []

        self._process_frames()

    def _process_frames(self):
        self.frame_pairs = find_frame_pairs(
            self.ground_truth_instances, self.predicted_instances, self.user_labels_only
        )
        if not self.frame_pairs:
            message = "Empty Frame Pairs. No match found for the video frames"
            logger.error(message)
            raise Exception(message)

        if self.match_method == "centroid":
            self._process_frames_centroid()
            return

        if self.match_method == "mask":
            self._process_frames_mask()
            return

        if self.match_method == "semantic":
            self._process_frames_semantic()
            return

        self.positive_pairs, self.false_negatives = match_frame_pairs(
            self.frame_pairs,
            stddev=self.oks_stddev,
            scale=self.oks_scale,
            threshold=self.match_threshold,
        )

        self.dists_dict = compute_dists(self.positive_pairs)

    def _process_frames_centroid(self):
        """Match predicted vs GT centroids by pixel distance (per frame).

        Each predicted instance is collapsed to its single centroid point (its
        sole visible point / node-0 for a 1-node prediction). Ground-truth
        centroids are computed via :func:`compute_gt_centroids` to exactly
        mirror the centroid target used during training (#586). Matching uses
        :func:`match_centroids` with ``self.match_threshold`` as a PIXEL
        distance. Populates ``positive_pairs`` as ``(gt_inst, pr_inst, dist)``
        3-tuples, ``false_negatives`` (unmatched GT), and ``false_positives``
        (unmatched predictions).
        """
        self.positive_pairs = []
        self.false_negatives = []
        self.false_positives = []

        for frame_gt, frame_pr in self.frame_pairs:
            gt_match_instances = get_instances(frame_gt)
            pr_match_instances = get_instances(frame_pr)

            # Collapse each predicted instance to its single centroid point.
            pred_centroids = np.array(
                [
                    self._collapse_pred_centroid(m.instance.numpy())
                    for m in pr_match_instances
                ]
            ).reshape(-1, 2)

            # GT centroids mirror generate_centroids exactly (#586).
            gt_centroids = np.array(
                [
                    compute_gt_centroids(m.instance.numpy(), self.anchor_ind)
                    for m in gt_match_instances
                ]
            ).reshape(-1, 2)

            # Drop NaN centroids before Hungarian matching: scipy's cdist /
            # linear_sum_assignment reject NaN, and a fully-occluded (all-NaN)
            # GT instance is common in real labels. Index maps translate the
            # filtered match indices back to the original instance lists so
            # FN/FP/positive-pair attribution stays correct. (A NaN-row GT is
            # counted as an automatic false negative — matching the legacy
            # CentroidEvaluationCallback; a NaN-row prediction is not a real
            # detection and is simply excluded.)
            gt_valid = ~np.isnan(gt_centroids).any(axis=1)
            pred_valid = ~np.isnan(pred_centroids).any(axis=1)
            gt_map = np.flatnonzero(gt_valid)
            pred_map = np.flatnonzero(pred_valid)

            matched_pred, matched_gt, unmatched_pred, unmatched_gt = match_centroids(
                pred_centroids[pred_valid],
                gt_centroids[gt_valid],
                max_distance=self.match_threshold,
            )

            for p_local, g_local in zip(matched_pred, matched_gt):
                p_idx = int(pred_map[int(p_local)])
                g_idx = int(gt_map[int(g_local)])
                dist = float(
                    np.linalg.norm(pred_centroids[p_idx] - gt_centroids[g_idx])
                )
                self.positive_pairs.append(
                    (gt_match_instances[g_idx], pr_match_instances[p_idx], dist)
                )

            for g_local in unmatched_gt:
                self.false_negatives.append(
                    gt_match_instances[int(gt_map[int(g_local)])]
                )
            # Fully-occluded (all-NaN) GT instances -> automatic false negatives.
            for g_idx in np.flatnonzero(~gt_valid):
                self.false_negatives.append(gt_match_instances[int(g_idx)])

            for p_local in unmatched_pred:
                self.false_positives.append(
                    pr_match_instances[int(pred_map[int(p_local)])]
                )

        # Build the dists dict directly from matched-pair centroid distances so
        # distance_metrics() works uniformly across match methods.
        dists = np.array([dist for _, _, dist in self.positive_pairs])
        self.dists_dict = {
            "dists": dists,
            "frame_idxs": [gt.frame_idx for gt, _, _ in self.positive_pairs],
            "video_paths": [gt.video_path for gt, _, _ in self.positive_pairs],
        }

    def _process_frames_mask(self):
        """Match predicted vs GT segmentation masks by IoU (per frame).

        Pulls per-instance boolean masks from ``LabeledFrame.masks`` on each
        paired frame and matches them with :func:`match_masks` using
        ``self.match_threshold`` as the IoU threshold. Populates
        ``positive_pairs`` as ``(frame_gt, frame_pr, iou)`` 3-tuples (the frame
        objects are stored only as tokens; detection counting uses the list
        lengths, and per-pair IoUs feed :meth:`mask_metrics`), plus
        ``false_negatives`` (unmatched GT masks) and ``false_positives``
        (unmatched predicted masks). No keypoint distances exist for masks, so
        ``dists_dict`` is left empty (``distance_metrics`` reports NaN; IoU is
        reported via :meth:`mask_metrics`).
        """
        self.positive_pairs = []
        self.false_negatives = []
        self.false_positives = []
        ious: List[float] = []
        # Per-frame decoded masks + scores + IoU/intersection matrices, reused by
        # mask_voc_metrics (score-ranked COCO AP) and the fragmentation/per-size
        # breakdowns without re-decoding RLE masks.
        self._mask_frames = []
        # Matched (pred_mask, gt_mask) TP pairs (aligned to ``self.mask_ious``),
        # used for boundary-IoU scoring.
        self._matched_mask_pairs = []

        for frame_gt, frame_pr in self.frame_pairs:
            # Ground-truth masks drop any PredictedInstance-linked masks when the
            # caller asked for user-only labels; predicted-side masks are the
            # model's output and are always kept in full.
            gt_masks = _frame_masks(
                frame_gt,
                drop_predicted_instances=self.exclude_predicted_instance_masks,
            )
            pr_masks = _frame_masks(frame_pr)
            pr_scores = _frame_pred_scores(frame_pr)
            iou_mat, inter_mat = _mask_pair_stats(pr_masks, gt_masks)
            self._mask_frames.append(
                {
                    "pred_masks": pr_masks,
                    "pred_scores": pr_scores,
                    "gt_masks": gt_masks,
                    "iou": iou_mat,
                    "inter": inter_mat,
                    "gt_areas": np.array([int(m.sum()) for m in gt_masks], dtype=float),
                    "pred_areas": np.array(
                        [int(m.sum()) for m in pr_masks], dtype=float
                    ),
                }
            )

            matched_pred, matched_gt, unmatched_pred, unmatched_gt, pair_ious = (
                match_masks(pr_masks, gt_masks, min_iou=self.match_threshold)
            )

            for iou in pair_ious:
                self.positive_pairs.append((frame_gt, frame_pr, float(iou)))
                ious.append(float(iou))
            for p_idx, g_idx in zip(matched_pred, matched_gt):
                self._matched_mask_pairs.append(
                    (pr_masks[int(p_idx)], gt_masks[int(g_idx)])
                )
            for _ in unmatched_gt:
                self.false_negatives.append(frame_gt)
            for _ in unmatched_pred:
                self.false_positives.append(frame_pr)

        self.mask_ious = np.asarray(ious, dtype=float)
        self.dists_dict = {"dists": np.array([]), "frame_idxs": [], "video_paths": []}

    def _process_frames_semantic(self):
        """Whole-frame foreground evaluation (no instance matching).

        For semantic (binary foreground/background) segmentation there is a single
        foreground mask per frame and no instance grouping, so there is nothing to
        match. Each paired frame's predicted and ground-truth masks are unioned
        into one foreground mask (:func:`_union_frame_fg`) and scored directly with
        :func:`_mask_iou`, :func:`mask_cldice`, and :func:`_boundary_iou`. Frames
        whose GROUND-TRUTH foreground is empty are skipped (there is no foreground
        to score).

        Populates ``self._semantic_rows`` as ``(iou, cldice, boundary_iou)``
        triples (consumed by :meth:`semantic_metrics`). The matching-based
        attributes (``positive_pairs`` / ``false_negatives`` / ``false_positives``
        / ``dists_dict``) are left empty so the shared plumbing degrades gracefully
        (semantic mode reports only ``semantic_metrics``).
        """
        self.positive_pairs = []
        self.false_negatives = []
        self.false_positives = []
        self._semantic_rows = []

        for frame_gt, frame_pr in self.frame_pairs:
            gt_fg = _union_frame_fg(frame_gt)
            if not gt_fg.any():
                # No ground-truth foreground: nothing to score on this frame.
                continue
            pr_fg = _union_frame_fg(frame_pr)
            iou = _mask_iou(pr_fg, gt_fg)
            cldice = mask_cldice(pr_fg, gt_fg)
            biou = _boundary_iou(pr_fg, gt_fg)
            self._semantic_rows.append((iou, cldice, biou))

        self.dists_dict = {"dists": np.array([]), "frame_idxs": [], "video_paths": []}

    @staticmethod
    def _collapse_pred_centroid(points: np.ndarray) -> np.ndarray:
        """Collapse a predicted instance to its single centroid point.

        For a 1-node ('centroid') prediction this is node-0. For predictions
        with multiple nodes (e.g. a single-instance model used as a detector)
        we take the single visible point, falling back to node-0.
        """
        points = np.asarray(points, dtype=np.float64).reshape(-1, 2)
        visible = ~np.isnan(points).any(axis=-1)
        if visible.any():
            return points[np.argmax(visible)]
        return points[0]

    def voc_metrics(
        self,
        match_score_by="oks",
        match_score_thresholds: np.ndarray = np.linspace(
            0.5, 0.95, 10
        ),  # 0.5:0.05:0.95
        recall_thresholds: np.ndarray = np.linspace(0, 1, 101),  # 0.0:0.01:1.00
    ):
        """Compute VOC metrics for a matched pairs of instances positive pairs and false negatives.

        Args:
            match_score_by: The score to be used for computing the metrics. "ock" or "pck"
            match_score_thresholds: Score thresholds at which to consider matches as a true
                positive match.
            recall_thresholds: Recall thresholds at which to evaluate Average Precision.

        Returns:
            A dictionary of VOC metrics.
        """
        if match_score_by == "oks":
            match_scores = np.array([oks for _, _, oks in self.positive_pairs])
            name = "oks_voc"
        elif match_score_by == "pck":
            name = "pck_voc"
            if not self.positive_pairs:
                # Guard the empty-match case: the (n_pairs, n_nodes, n_thresholds)
                # ``pcks`` array is empty along the pairs axis, so reducing it with
                # nested .mean() calls would hit "Mean of empty slice".
                match_scores = np.array([])
            else:
                pck_metrics = self.pck_metrics()
                match_scores = pck_metrics["pcks"].mean(axis=-1).mean(axis=-1)
        else:
            message = "Invalid Option for match_score_by. Choose either `oks` or `pck`"
            logger.error(message)
            raise Exception(message)

        detection_scores = np.array(
            [pp[1].instance.score for pp in self.positive_pairs]
        )

        inds = np.argsort(-detection_scores, kind="mergesort")
        detection_scores = detection_scores[inds]
        match_scores = match_scores[inds]

        precisions = []
        recalls = []

        npig = len(self.positive_pairs) + len(
            self.false_negatives
        )  # total number of GT instances

        for match_score_threshold in match_score_thresholds:
            tp = np.cumsum(match_scores >= match_score_threshold)
            fp = np.cumsum(match_scores < match_score_threshold)

            if tp.size == 0:
                return {
                    name + ".match_score_thresholds": 0,
                    name + ".recall_thresholds": 0,
                    name + ".match_scores": 0,
                    name + ".precisions": 0,
                    name + ".recalls": 0,
                    name + ".AP": 0,
                    name + ".AR": 0,
                    name + ".mAP": 0,
                    name + ".mAR": 0,
                }

            rc = tp / npig
            pr = tp / (fp + tp + np.spacing(1))

            recall = rc[-1]  # best recall at this OKS threshold

            # Ensure strictly decreasing precisions.
            for i in range(len(pr) - 1, 0, -1):
                if pr[i] > pr[i - 1]:
                    pr[i - 1] = pr[i]

            # Find best precision at each recall threshold.
            rc_inds = np.searchsorted(rc, recall_thresholds, side="left")
            precision = np.zeros(rc_inds.shape)
            is_valid_rc_ind = rc_inds < len(pr)
            precision[is_valid_rc_ind] = pr[rc_inds[is_valid_rc_ind]]

            precisions.append(precision)
            recalls.append(recall)

        precisions = np.array(precisions)
        recalls = np.array(recalls)

        AP = precisions.mean(
            axis=1
        )  # AP = average precision over fixed set of recall thresholds
        AR = recalls  # AR = max recall given a fixed number of detections per image

        mAP = precisions.mean()  # mAP = mean over all OKS thresholds
        mAR = recalls.mean()  # mAR = mean over all OKS thresholds

        return {
            name + ".match_score_thresholds": match_score_thresholds,
            name + ".recall_thresholds": recall_thresholds,
            name + ".match_scores": match_scores,
            name + ".precisions": precisions,
            name + ".recalls": recalls,
            name + ".AP": AP,
            name + ".AR": AR,
            name + ".mAP": mAP,
            name + ".mAR": mAR,
        }

    def mOKS(self):
        """Return the meanOKS value."""
        pair_oks = np.array([oks for _, _, oks in self.positive_pairs])
        return {"mOKS": float(pair_oks.mean()) if pair_oks.size else np.nan}

    def distance_metrics(self):
        """Compute the Euclidean distance error at different percentiles using the pairwise distances.

        Returns:
            A dictionary of distance metrics.
        """
        dists = self.dists_dict["dists"]
        results = {
            "frame_idxs": self.dists_dict["frame_idxs"],
            "video_paths": self.dists_dict["video_paths"],
            "dists": dists,
            # Guard the empty / all-NaN matched set (zero true positives in a
            # split) so np.nanmean doesn't emit a "Mean of empty slice" warning.
            "avg": (
                float(np.nanmean(dists))
                if np.asarray(dists).size and not np.all(np.isnan(dists))
                else np.nan
            ),
            "p50": np.nan,
            "p75": np.nan,
            "p90": np.nan,
            "p95": np.nan,
            "p99": np.nan,
        }

        is_non_nan = ~np.isnan(dists)
        if np.any(is_non_nan):
            non_nans = dists[is_non_nan]
            for ptile in (50, 75, 90, 95, 99):
                results[f"p{ptile}"] = np.percentile(non_nans, ptile)

        return results

    def detection_metrics(self) -> dict:
        """Compute detection metrics (precision/recall/F1) over TP/FP/FN counts.

        Used by both ``match_method="centroid"`` and ``match_method="mask"``
        (it only reads the matched/unmatched list lengths and ``dists_dict``).
        Mirrors ``CentroidEvaluationCallback._compute_metrics``. For centroid
        mode the localization-error percentiles are computed over the Euclidean
        distances of matched centroid pairs; for mask mode ``dists_dict`` is
        empty so those percentiles are NaN (per-pair IoU is reported separately
        via :meth:`mask_metrics`). Not used for ``match_method="oks"`` (which
        reports OKS-based VOC metrics instead).

        Returns:
            A dict with ``precision``, ``recall``, ``f1``, ``n_tp``, ``n_fp``,
            ``n_fn`` and localization-error percentiles ``avg``/``p50``/``p75``/
            ``p90``/``p95``/``p99`` (NaN when there are no matched pairs).
        """
        n_tp = len(self.positive_pairs)
        n_fp = len(self.false_positives)
        n_fn = len(self.false_negatives)

        precision = n_tp / (n_tp + n_fp) if (n_tp + n_fp) > 0 else 0.0
        recall = n_tp / (n_tp + n_fn) if (n_tp + n_fn) > 0 else 0.0
        f1 = (
            2 * precision * recall / (precision + recall)
            if (precision + recall) > 0
            else 0.0
        )

        dists = self.dists_dict["dists"]
        results = {
            "precision": precision,
            "recall": recall,
            "f1": f1,
            "n_tp": n_tp,
            "n_fp": n_fp,
            "n_fn": n_fn,
            "avg": np.nan,
            "p50": np.nan,
            "p75": np.nan,
            "p90": np.nan,
            "p95": np.nan,
            "p99": np.nan,
        }

        is_non_nan = ~np.isnan(dists) if len(dists) else np.array([], dtype=bool)
        if np.any(is_non_nan):
            non_nans = dists[is_non_nan]
            results["avg"] = float(np.mean(non_nans))
            for ptile in (50, 75, 90, 95, 99):
                results[f"p{ptile}"] = float(np.percentile(non_nans, ptile))

        return results

    def mask_metrics(self) -> dict:
        """Compute mask-IoU summary statistics for ``match_method="mask"``.

        Reports complementary IoU summaries, panoptic-quality, boundary-IoU,
        fragmentation, and per-object-size breakdowns:

        * ``mean_iou`` (and ``min``/``max``/percentiles) over the matched (TP)
          pairs only — COCO-style segmentation quality, blind to misses.
        * ``mean_iou_all_gt`` — IoU averaged over *all* ground-truth masks,
          where an unmatched GT (a miss) contributes ``0``. This penalizes
          recall and complements the TP-only mean.
        * Panoptic Quality ``pq = sq * rq`` with ``sq = mean_iou`` (segmentation
          quality) and ``rq = TP / (TP + 0.5*FP + 0.5*FN)`` (recognition
          quality, == detection F1). See Kirillov et al., "Panoptic
          Segmentation" (2019).
        * ``mean_boundary_iou`` — boundary IoU over the matched pairs (Cheng et
          al., 2021), more sensitive to contour error than mask IoU.
        * ``mean_cldice`` — centerline Dice over the matched pairs (Shit et al.,
          CVPR 2021), connectivity-aware and nearly width-insensitive; a fairer
          score than IoU for thin/tubular structures. NaN if scikit-image is
          unavailable.
        * ``oversegmentation`` / ``undersegmentation`` — fragmentation counts:
          GT masks split across >=2 predictions, and predictions spanning >=2
          GT masks (each with >=10% area overlap). The headline over-/under-
          segmentation failure mode is invisible to the 1-to-1 match.
        * ``per_size`` — COCO small/medium/large breakdown of GT count, TP
          count, and TP-only mean IoU (buckets sum to the GT total).

        Returns:
            A dict with ``mean_iou``, ``min``, ``max``, percentiles ``p25``/
            ``p50``/``p75``, ``mean_iou_all_gt``, ``pq``/``sq``/``rq``,
            ``mean_boundary_iou``, ``oversegmentation``/``undersegmentation``,
            ``per_size``, the TP count ``n_matched`` (plus ``n_fp``/``n_fn``),
            and the raw ``ious`` array. Quantities are NaN when undefined.
        """
        ious = np.asarray(self.mask_ious, dtype=float)
        n_tp = len(self.positive_pairs)
        n_fp = len(self.false_positives)
        n_fn = len(self.false_negatives)
        over, under = self._fragmentation_counts()
        results = {
            "mean_iou": np.nan,
            "min": np.nan,
            "max": np.nan,
            "p25": np.nan,
            "p50": np.nan,
            "p75": np.nan,
            "mean_iou_all_gt": np.nan,
            "pq": np.nan,
            "sq": np.nan,
            "rq": np.nan,
            "mean_boundary_iou": np.nan,
            "mean_cldice": np.nan,
            "oversegmentation": over,
            "undersegmentation": under,
            "per_size": self._mask_per_size_stats(),
            "n_matched": int(ious.size),
            "n_fp": n_fp,
            "n_fn": n_fn,
            "ious": ious,
        }
        if ious.size:
            results["mean_iou"] = float(np.mean(ious))
            results["min"] = float(np.min(ious))
            results["max"] = float(np.max(ious))
            for ptile in (25, 50, 75):
                results[f"p{ptile}"] = float(np.percentile(ious, ptile))

        if self._matched_mask_pairs:
            boundary_ious = np.array(
                [_boundary_iou(p, g) for p, g in self._matched_mask_pairs],
                dtype=float,
            )
            results["mean_boundary_iou"] = float(np.mean(boundary_ious))
            # Centerline Dice (clDice): connectivity/width-tolerant, fairer than
            # IoU for thin structures. NaN entries (scikit-image missing) drop out.
            cldices = np.array(
                [mask_cldice(p, g) for p, g in self._matched_mask_pairs],
                dtype=float,
            )
            cldices = cldices[~np.isnan(cldices)]
            if cldices.size:
                results["mean_cldice"] = float(np.mean(cldices))

        iou_sum = float(np.sum(ious)) if ious.size else 0.0
        # Miss-penalizing mean: averaged over every GT mask (TP + FN).
        n_gt = n_tp + n_fn
        if n_gt > 0:
            results["mean_iou_all_gt"] = iou_sum / n_gt
        # Panoptic quality: SQ = TP-only mean IoU, RQ = detection F1, PQ = SQ*RQ
        # = iou_sum / (TP + 0.5*FP + 0.5*FN).
        pq_denom = n_tp + 0.5 * n_fp + 0.5 * n_fn
        if pq_denom > 0:
            results["sq"] = results["mean_iou"]
            results["rq"] = n_tp / pq_denom
            results["pq"] = iou_sum / pq_denom
        return results

    def semantic_metrics(self) -> dict:
        """Aggregate whole-frame foreground metrics for ``match_method="semantic"``.

        Averages the per-frame foreground IoU, centerline Dice (clDice), and
        boundary IoU computed by :meth:`_process_frames_semantic` over all frames
        with non-empty ground-truth foreground. clDice entries that are NaN
        (scikit-image unavailable) are dropped from the clDice mean; if every entry
        is NaN the reported ``mean_cldice`` is NaN.

        Returns:
            A dict with ``mean_iou``, ``mean_cldice``, ``mean_boundary_iou``, the
            per-frame ``ious`` / ``cldices`` / ``boundary_ious`` arrays, and
            ``n_frames`` (frames scored). Means are NaN when no frame was scored.
        """
        rows = np.asarray(self._semantic_rows, dtype=float).reshape(-1, 3)
        ious = rows[:, 0]
        cldices = rows[:, 1]
        bious = rows[:, 2]
        cld_valid = cldices[~np.isnan(cldices)]
        return {
            "mean_iou": float(np.mean(ious)) if ious.size else float("nan"),
            "mean_cldice": (
                float(np.mean(cld_valid)) if cld_valid.size else float("nan")
            ),
            "mean_boundary_iou": (
                float(np.mean(bious)) if bious.size else float("nan")
            ),
            "ious": ious,
            "cldices": cldices,
            "boundary_ious": bious,
            "n_frames": int(ious.size),
        }

    def _fragmentation_counts(self, overlap_frac: float = 0.1) -> Tuple[int, int]:
        """Count over-/under-segmented instances across all mask frames.

        A prediction "covers" a GT mask when their intersection is at least
        ``overlap_frac`` of the GT area. Over-segmentation counts GT masks
        covered by >=2 predictions (one animal split into fragments);
        under-segmentation counts predictions covering >=2 GT masks (one mask
        merging neighbors). Both directly surface the failure mode the 1-to-1
        Hungarian match hides (extra fragments otherwise just become FPs).
        """
        over = under = 0
        for f in self._mask_frames:
            inter = f["inter"]
            gt_areas = f["gt_areas"]
            n_pred, n_gt = inter.shape
            if n_pred == 0 or n_gt == 0:
                continue
            # Fraction of each GT (cols) covered by each prediction (rows).
            cov_gt = inter / np.maximum(gt_areas[None, :], 1.0)
            covers = cov_gt >= overlap_frac
            over += int(np.count_nonzero(covers.sum(axis=0) >= 2))  # GT split
            under += int(np.count_nonzero(covers.sum(axis=1) >= 2))  # pred merged
        return over, under

    def _per_size_breakdown(
        self,
        gt_areas_all: np.ndarray,
        tp_iou: np.ndarray,
        tp_gt_area: np.ndarray,
        edges: np.ndarray,
    ) -> dict:
        """small/medium/large GT count, TP count and TP mean IoU under ``edges``.

        ``n_gt`` over the three buckets sums to the total GT count (every GT area
        falls in exactly one half-open bucket).
        """
        out = {"edges": [float(e) for e in edges]}
        for idx, bucket in enumerate(_SIZE_KEYS):
            in_gt = _size_mask(gt_areas_all, idx, edges)
            in_tp = (
                _size_mask(tp_gt_area, idx, edges)
                if tp_gt_area.size
                else np.array([], dtype=bool)
            )
            out[bucket] = {
                "n_gt": int(np.count_nonzero(in_gt)),
                "n_tp": int(np.count_nonzero(in_tp)),
                "mean_iou": (
                    float(np.mean(tp_iou[in_tp])) if np.any(in_tp) else np.nan
                ),
            }
        return out

    def _mask_per_size_stats(self) -> dict:
        """Per-object-size GT/TP/IoU breakdown under both bucketing schemes.

        GT objects are bucketed by mask area (``mask.sum()``). The primary
        scheme (top-level ``small``/``medium``/``large`` keys) uses
        dataset-relative percentile edges (terciles by default) so the buckets
        adapt to the actual mask scale; the COCO fixed-cutoff scheme (small <
        32^2 <= medium < 96^2 <= large) is reported additionally under
        ``"coco"`` for cross-dataset comparability.
        """
        gt_areas_all = np.array(
            [a for f in self._mask_frames for a in f["gt_areas"]], dtype=float
        )
        tp_iou = np.asarray(self.mask_ious, dtype=float)
        tp_gt_area = np.array(
            [int(g.sum()) for _, g in self._matched_mask_pairs], dtype=float
        )
        pct_edges = _percentile_size_edges(gt_areas_all)
        out = self._per_size_breakdown(gt_areas_all, tp_iou, tp_gt_area, pct_edges)
        out["scheme"] = "percentile"
        out["coco"] = self._per_size_breakdown(
            gt_areas_all, tp_iou, tp_gt_area, COCO_SIZE_EDGES
        )
        return out

    def _match_masks_coco(
        self, iou_threshold: float
    ) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
        """Greedy score-ranked pred->GT matching at one IoU threshold (COCO).

        Per frame, predictions are considered in descending score order; each
        claims the highest-IoU not-yet-claimed GT whose IoU >= ``iou_threshold``
        (a TP), else it is a FP. Mirrors ``pycocotools`` matching.

        Returns:
            ``(scores, matched, matched_gt_area, pred_area)`` flat arrays over
            every prediction across all frames (aligned). ``matched`` is the
            TP flag; ``matched_gt_area`` is the area of the claimed GT (NaN for
            a FP); ``pred_area`` is the prediction's own area.
        """
        scores, matched, matched_gt_area, pred_area = [], [], [], []
        for f in self._mask_frames:
            iou = f["iou"]
            pred_scores = f["pred_scores"]
            gt_areas = f["gt_areas"]
            pred_areas = f["pred_areas"]
            n_pred, n_gt = iou.shape
            order = (
                np.argsort(-pred_scores, kind="mergesort")
                if n_pred
                else np.array([], dtype=int)
            )
            gt_taken = np.zeros(n_gt, dtype=bool)
            for p in order:
                scores.append(float(pred_scores[p]))
                pred_area.append(float(pred_areas[p]))
                if n_gt == 0:
                    matched.append(False)
                    matched_gt_area.append(np.nan)
                    continue
                row = iou[p].copy()
                row[gt_taken] = -1.0
                g = int(np.argmax(row))
                if row[g] >= iou_threshold:
                    gt_taken[g] = True
                    matched.append(True)
                    matched_gt_area.append(float(gt_areas[g]))
                else:
                    matched.append(False)
                    matched_gt_area.append(np.nan)
        return (
            np.array(scores, dtype=float),
            np.array(matched, dtype=bool),
            np.array(matched_gt_area, dtype=float),
            np.array(pred_area, dtype=float),
        )

    def mask_voc_metrics(
        self,
        iou_thresholds: np.ndarray = MASK_IOU_THRESHOLDS,
        recall_thresholds: np.ndarray = np.linspace(0, 1, 101),
        size_percentiles: Tuple[float, float] = DEFAULT_SIZE_PERCENTILES,
    ) -> dict:
        """COCO-style score-ranked mask Average Precision / Recall.

        Re-matches predictions to GT independently at each IoU threshold
        (:meth:`_match_masks_coco`), score-ranks the resulting TP/FP flags, and
        integrates the precision-recall curve (101-point interpolation, mirrors
        :meth:`voc_metrics`). Reports overall AP@[.5:.95]/AP50/AP75/AR plus a
        per-object-size AP breakdown under two bucketing schemes (GT outside a
        bucket is ignored, as in ``pycocotools`` ``areaRng``): the primary
        (default) buckets use dataset-relative percentile edges (terciles), and
        the COCO fixed-cutoff buckets are reported additionally under the
        ``mask_voc.coco.`` prefix — analogous to the dual OKS/PCK VOC.

        Args:
            iou_thresholds: IoU thresholds to average AP over.
            recall_thresholds: Recall grid for 101-point interpolation.
            size_percentiles: Two percentiles of the GT area distribution
                delimiting the primary small/medium/large buckets.

        Returns:
            A dict keyed under ``"mask_voc."``: ``AP`` (per-threshold array),
            ``mAP``, ``AP50``, ``AP75``, ``AR``, ``recalls``, ``iou_thresholds``,
            ``n_gt``; primary per-size ``AP_small``/``AP_medium``/``AP_large``,
            ``n_gt_small``/``..._medium``/``..._large``, ``size_scheme`` and
            ``size_edges``; and COCO per-size ``coco.AP_*``/``coco.n_gt_*``/
            ``coco.size_edges``. AP values are NaN when the relevant GT set is
            empty.
        """
        iou_thresholds = np.asarray(iou_thresholds, dtype=float)
        recall_thresholds = np.asarray(recall_thresholds, dtype=float)
        gt_areas_all = np.array(
            [a for f in self._mask_frames for a in f["gt_areas"]], dtype=float
        )
        npig = int(gt_areas_all.size)

        # Primary (percentile, dataset-relative) + additional (COCO) edges.
        schemes = {
            "percentile": _percentile_size_edges(gt_areas_all, size_percentiles),
            "coco": COCO_SIZE_EDGES,
        }
        n_gt_size = {
            name: [
                int(np.count_nonzero(_size_mask(gt_areas_all, i, edges)))
                for i in range(len(_SIZE_KEYS))
            ]
            for name, edges in schemes.items()
        }

        ap_overall = np.full(iou_thresholds.size, np.nan)
        recall_overall = np.full(iou_thresholds.size, np.nan)
        ap_size = {
            name: [np.full(iou_thresholds.size, np.nan) for _ in _SIZE_KEYS]
            for name in schemes
        }

        for ti, thr in enumerate(iou_thresholds):
            scores, matched, matched_gt_area, pred_area = self._match_masks_coco(
                float(thr)
            )
            ap_overall[ti], recall_overall[ti] = _ap_from_pr(
                scores, matched, npig, recall_thresholds
            )
            for name, edges in schemes.items():
                for i in range(len(_SIZE_KEYS)):
                    # COCO areaRng: keep TPs whose matched GT is in-bucket and
                    # FPs whose own area is in-bucket; ignore everything else.
                    keep_tp = matched & _size_mask(matched_gt_area, i, edges)
                    keep_fp = (~matched) & _size_mask(pred_area, i, edges)
                    keep = keep_tp | keep_fp
                    ap_size[name][i][ti], _ = _ap_from_pr(
                        scores[keep],
                        keep_tp[keep],
                        n_gt_size[name][i],
                        recall_thresholds,
                    )

        def _nanmean(arr: np.ndarray) -> float:
            return float(np.nanmean(arr)) if np.any(~np.isnan(arr)) else np.nan

        def _at(target: float) -> float:
            return float(ap_overall[int(np.argmin(np.abs(iou_thresholds - target)))])

        results = {
            "mask_voc.iou_thresholds": iou_thresholds,
            "mask_voc.AP": ap_overall,
            "mask_voc.recalls": recall_overall,
            "mask_voc.mAP": _nanmean(ap_overall),
            "mask_voc.AR": _nanmean(recall_overall),
            "mask_voc.AP50": _at(0.5),
            "mask_voc.AP75": _at(0.75),
            "mask_voc.n_gt": npig,
            "mask_voc.size_scheme": "percentile",
            "mask_voc.size_edges": [float(e) for e in schemes["percentile"]],
            "mask_voc.coco.size_edges": [float(e) for e in schemes["coco"]],
        }
        # Primary (percentile) per-size keys are unprefixed; COCO is additional.
        for name, prefix in (("percentile", "mask_voc."), ("coco", "mask_voc.coco.")):
            for i, bucket in enumerate(_SIZE_KEYS):
                results[f"{prefix}AP_{bucket}"] = _nanmean(ap_size[name][i])
                results[f"{prefix}n_gt_{bucket}"] = n_gt_size[name][i]
        return results

    def pck_metrics(self, thresholds: np.ndarray = np.linspace(1, 10, 10)):
        """Compute PCK across a range of thresholds using the pair-wise distances.

        Args:
            thresholds: A list of distance thresholds in pixels.

        Returns:
            A dictionary of PCK metrics evaluated at each threshold.
        """
        dists = self.dists_dict["dists"]
        dists = np.copy(dists)
        dists[np.isnan(dists)] = np.inf
        pcks = np.expand_dims(dists, -1) < np.reshape(thresholds, (1, 1, -1))

        # Guard the empty-match case (0 positive pairs for the whole split) so
        # the nested .mean() reductions below don't hit "Mean of empty slice".
        if dists.size == 0:
            mPCK_parts = np.array([])
            mPCK = np.nan
            pck5 = np.nan
            pck10 = np.nan
        else:
            mPCK_parts = pcks.mean(axis=0).mean(axis=-1)
            mPCK = float(mPCK_parts.mean())

            # Precompute PCK at common thresholds
            idx_5 = np.argmin(np.abs(thresholds - 5))
            idx_10 = np.argmin(np.abs(thresholds - 10))
            pck5 = float(pcks[:, :, idx_5].mean())
            pck10 = float(pcks[:, :, idx_10].mean())

        return {
            "thresholds": thresholds,
            "pcks": pcks,
            "mPCK_parts": mPCK_parts,
            "mPCK": mPCK,
            "PCK@5": pck5,
            "PCK@10": pck10,
        }

    def visibility_metrics(self):
        """Compute node visibility metrics for the matched pair of instances.

        Returns:
            A dictionary of visibility metrics, including the confusion matrix.
        """
        vis_tp = 0
        vis_fn = 0
        vis_fp = 0
        vis_tn = 0

        for instance_gt, instance_pr, _ in self.positive_pairs:
            missing_nodes_gt = np.isnan(instance_gt.instance.numpy()).any(axis=-1)
            missing_nodes_pr = np.isnan(instance_pr.instance.numpy()).any(axis=-1)

            vis_tn += ((missing_nodes_gt) & (missing_nodes_pr)).sum()
            vis_fn += ((~missing_nodes_gt) & (missing_nodes_pr)).sum()
            vis_fp += ((missing_nodes_gt) & (~missing_nodes_pr)).sum()
            vis_tp += ((~missing_nodes_gt) & (~missing_nodes_pr)).sum()

        return {
            "tp": vis_tp,
            "fp": vis_fp,
            "tn": vis_tn,
            "fn": vis_fn,
            "precision": vis_tp / (vis_tp + vis_fp) if (vis_tp + vis_fp) else np.nan,
            "recall": vis_tp / (vis_tp + vis_fn) if (vis_tp + vis_fn) else np.nan,
        }

    def evaluate(self):
        """Return the evaluation metrics."""
        if self.match_method == "centroid":
            # Single-node / centroid-only: OKS/PCK/mOKS/visibility are
            # degenerate for one node, so we only report detection +
            # distance metrics. We intentionally do NOT compute OKS for a
            # single node (no magic OKS-scale constant) — the OKS path stays
            # only for match_method="oks".
            return {
                "detection_metrics": self.detection_metrics(),
                "distance_metrics": self.distance_metrics(),
            }

        if self.match_method == "mask":
            # Instance segmentation: detection (precision/recall/F1 over
            # IoU-matched masks) + mask-IoU quality + COCO-style score-ranked
            # mask AP/AR. OKS/PCK/visibility are keypoint-only and not computed.
            return {
                "detection_metrics": self.detection_metrics(),
                "mask_metrics": self.mask_metrics(),
                "mask_voc_metrics": self.mask_voc_metrics(),
            }

        if self.match_method == "semantic":
            # Whole-frame binary foreground segmentation: no instances to match, so
            # report only matching-free foreground IoU / clDice / boundary-IoU.
            return {"semantic_metrics": self.semantic_metrics()}

        if not self.positive_pairs:
            # 0 matched instances for the whole split (e.g. a collapsed model
            # predicting nothing, or predictions that never clear the OKS
            # threshold) -- every metric below is undefined by construction.
            # The individual methods already guard their own NaN/empty-array
            # math, so this is just one clear line instead of relying on the
            # reader to infer "collapsed model" from a wall of NaNs.
            logger.info(
                "0 matched instances: metrics undefined (model predicted "
                "nothing usable, or training likely collapsed)."
            )

        metrics = {}
        metrics["voc_metrics"] = self.voc_metrics(match_score_by="oks")
        metrics["voc_metrics"].update(self.voc_metrics(match_score_by="pck"))
        metrics["mOKS"] = self.mOKS()
        metrics["distance_metrics"] = self.distance_metrics()
        metrics["pck_metrics"] = self.pck_metrics()
        metrics["visibility_metrics"] = self.visibility_metrics()

        return metrics

__init__(ground_truth_instances, predicted_instances, oks_stddev=0.025, oks_scale=None, match_threshold=0, user_labels_only=True, match_method='oks', anchor_ind=None, exclude_predicted_instance_masks=False)

Initialize the Evaluator class with ground-truth and predicted labels.

exclude_predicted_instance_masks (match_method="mask" only) drops masks linked to a PredictedInstance from the ground-truth labels, so a labels file that carries stray predicted instances (each of which gets a mask when masks are built from poses) does not treat them as ground truth. It is kept separate from user_labels_only (which controls the frame-pair filter) because mask mode disables that frame filter -- see :func:run_evaluation.

Source code in sleap_nn/evaluation.py

def __init__(
    self,
    ground_truth_instances: sio.Labels,
    predicted_instances: sio.Labels,
    oks_stddev: float = 0.025,
    oks_scale: Optional[float] = None,
    match_threshold: float = 0,
    user_labels_only: bool = True,
    match_method: str = "oks",
    anchor_ind: Optional[int] = None,
    exclude_predicted_instance_masks: bool = False,
):
    """Initialize the Evaluator class with ground-truth and predicted labels.

    ``exclude_predicted_instance_masks`` (``match_method="mask"`` only) drops
    masks linked to a ``PredictedInstance`` from the ground-truth labels, so a
    labels file that carries stray predicted instances (each of which gets a
    mask when masks are built from poses) does not treat them as ground truth.
    It is kept separate from ``user_labels_only`` (which controls the frame-pair
    filter) because mask mode disables that frame filter -- see
    :func:`run_evaluation`.
    """
    self.ground_truth_instances = ground_truth_instances
    self.predicted_instances = predicted_instances
    self.match_threshold = match_threshold
    self.oks_stddev = oks_stddev
    self.oks_scale = oks_scale
    self.user_labels_only = user_labels_only
    self.match_method = match_method
    self.anchor_ind = anchor_ind
    self.exclude_predicted_instance_masks = exclude_predicted_instance_masks
    # Populated only in centroid / mask mode.
    self.false_positives = []
    # Matched-pair IoUs, populated only in mask mode.
    self.mask_ious = np.array([])
    # Per-frame mask records + matched TP mask pairs, populated only in mask
    # mode (feed mask_voc_metrics / boundary-IoU / fragmentation / per-size).
    self._mask_frames = []
    self._matched_mask_pairs = []
    # Per-frame (iou, cldice, boundary_iou) triples, populated only in
    # match_method="semantic" (whole-frame foreground, no matching).
    self._semantic_rows = []

    self._process_frames()

detection_metrics()

Compute detection metrics (precision/recall/F1) over TP/FP/FN counts.

Used by both match_method="centroid" and match_method="mask" (it only reads the matched/unmatched list lengths and dists_dict). Mirrors CentroidEvaluationCallback._compute_metrics. For centroid mode the localization-error percentiles are computed over the Euclidean distances of matched centroid pairs; for mask mode dists_dict is empty so those percentiles are NaN (per-pair IoU is reported separately via :meth:mask_metrics). Not used for match_method="oks" (which reports OKS-based VOC metrics instead).

Returns:

Type	Description
dict	A dict with precision, recall, f1, n_tp, n_fp, n_fn and localization-error percentiles avg/p50/p75/ p90/p95/p99 (NaN when there are no matched pairs).

Source code in sleap_nn/evaluation.py

def detection_metrics(self) -> dict:
    """Compute detection metrics (precision/recall/F1) over TP/FP/FN counts.

    Used by both ``match_method="centroid"`` and ``match_method="mask"``
    (it only reads the matched/unmatched list lengths and ``dists_dict``).
    Mirrors ``CentroidEvaluationCallback._compute_metrics``. For centroid
    mode the localization-error percentiles are computed over the Euclidean
    distances of matched centroid pairs; for mask mode ``dists_dict`` is
    empty so those percentiles are NaN (per-pair IoU is reported separately
    via :meth:`mask_metrics`). Not used for ``match_method="oks"`` (which
    reports OKS-based VOC metrics instead).

    Returns:
        A dict with ``precision``, ``recall``, ``f1``, ``n_tp``, ``n_fp``,
        ``n_fn`` and localization-error percentiles ``avg``/``p50``/``p75``/
        ``p90``/``p95``/``p99`` (NaN when there are no matched pairs).
    """
    n_tp = len(self.positive_pairs)
    n_fp = len(self.false_positives)
    n_fn = len(self.false_negatives)

    precision = n_tp / (n_tp + n_fp) if (n_tp + n_fp) > 0 else 0.0
    recall = n_tp / (n_tp + n_fn) if (n_tp + n_fn) > 0 else 0.0
    f1 = (
        2 * precision * recall / (precision + recall)
        if (precision + recall) > 0
        else 0.0
    )

    dists = self.dists_dict["dists"]
    results = {
        "precision": precision,
        "recall": recall,
        "f1": f1,
        "n_tp": n_tp,
        "n_fp": n_fp,
        "n_fn": n_fn,
        "avg": np.nan,
        "p50": np.nan,
        "p75": np.nan,
        "p90": np.nan,
        "p95": np.nan,
        "p99": np.nan,
    }

    is_non_nan = ~np.isnan(dists) if len(dists) else np.array([], dtype=bool)
    if np.any(is_non_nan):
        non_nans = dists[is_non_nan]
        results["avg"] = float(np.mean(non_nans))
        for ptile in (50, 75, 90, 95, 99):
            results[f"p{ptile}"] = float(np.percentile(non_nans, ptile))

    return results

distance_metrics()

Compute the Euclidean distance error at different percentiles using the pairwise distances.

Returns:

Type	Description
	A dictionary of distance metrics.

Source code in sleap_nn/evaluation.py

def distance_metrics(self):
    """Compute the Euclidean distance error at different percentiles using the pairwise distances.

    Returns:
        A dictionary of distance metrics.
    """
    dists = self.dists_dict["dists"]
    results = {
        "frame_idxs": self.dists_dict["frame_idxs"],
        "video_paths": self.dists_dict["video_paths"],
        "dists": dists,
        # Guard the empty / all-NaN matched set (zero true positives in a
        # split) so np.nanmean doesn't emit a "Mean of empty slice" warning.
        "avg": (
            float(np.nanmean(dists))
            if np.asarray(dists).size and not np.all(np.isnan(dists))
            else np.nan
        ),
        "p50": np.nan,
        "p75": np.nan,
        "p90": np.nan,
        "p95": np.nan,
        "p99": np.nan,
    }

    is_non_nan = ~np.isnan(dists)
    if np.any(is_non_nan):
        non_nans = dists[is_non_nan]
        for ptile in (50, 75, 90, 95, 99):
            results[f"p{ptile}"] = np.percentile(non_nans, ptile)

    return results

evaluate()

Return the evaluation metrics.

Source code in sleap_nn/evaluation.py

def evaluate(self):
    """Return the evaluation metrics."""
    if self.match_method == "centroid":
        # Single-node / centroid-only: OKS/PCK/mOKS/visibility are
        # degenerate for one node, so we only report detection +
        # distance metrics. We intentionally do NOT compute OKS for a
        # single node (no magic OKS-scale constant) — the OKS path stays
        # only for match_method="oks".
        return {
            "detection_metrics": self.detection_metrics(),
            "distance_metrics": self.distance_metrics(),
        }

    if self.match_method == "mask":
        # Instance segmentation: detection (precision/recall/F1 over
        # IoU-matched masks) + mask-IoU quality + COCO-style score-ranked
        # mask AP/AR. OKS/PCK/visibility are keypoint-only and not computed.
        return {
            "detection_metrics": self.detection_metrics(),
            "mask_metrics": self.mask_metrics(),
            "mask_voc_metrics": self.mask_voc_metrics(),
        }

    if self.match_method == "semantic":
        # Whole-frame binary foreground segmentation: no instances to match, so
        # report only matching-free foreground IoU / clDice / boundary-IoU.
        return {"semantic_metrics": self.semantic_metrics()}

    if not self.positive_pairs:
        # 0 matched instances for the whole split (e.g. a collapsed model
        # predicting nothing, or predictions that never clear the OKS
        # threshold) -- every metric below is undefined by construction.
        # The individual methods already guard their own NaN/empty-array
        # math, so this is just one clear line instead of relying on the
        # reader to infer "collapsed model" from a wall of NaNs.
        logger.info(
            "0 matched instances: metrics undefined (model predicted "
            "nothing usable, or training likely collapsed)."
        )

    metrics = {}
    metrics["voc_metrics"] = self.voc_metrics(match_score_by="oks")
    metrics["voc_metrics"].update(self.voc_metrics(match_score_by="pck"))
    metrics["mOKS"] = self.mOKS()
    metrics["distance_metrics"] = self.distance_metrics()
    metrics["pck_metrics"] = self.pck_metrics()
    metrics["visibility_metrics"] = self.visibility_metrics()

    return metrics

mOKS()

Return the meanOKS value.

Source code in sleap_nn/evaluation.py

def mOKS(self):
    """Return the meanOKS value."""
    pair_oks = np.array([oks for _, _, oks in self.positive_pairs])
    return {"mOKS": float(pair_oks.mean()) if pair_oks.size else np.nan}

mask_metrics()

Compute mask-IoU summary statistics for match_method="mask".

Reports complementary IoU summaries, panoptic-quality, boundary-IoU, fragmentation, and per-object-size breakdowns:

• mean_iou (and min/max/percentiles) over the matched (TP) pairs only — COCO-style segmentation quality, blind to misses.
• mean_iou_all_gt — IoU averaged over all ground-truth masks, where an unmatched GT (a miss) contributes 0. This penalizes recall and complements the TP-only mean.
• Panoptic Quality pq = sq * rq with sq = mean_iou (segmentation quality) and rq = TP / (TP + 0.5*FP + 0.5*FN) (recognition quality, == detection F1). See Kirillov et al., "Panoptic Segmentation" (2019).
• mean_boundary_iou — boundary IoU over the matched pairs (Cheng et al., 2021), more sensitive to contour error than mask IoU.
• mean_cldice — centerline Dice over the matched pairs (Shit et al., CVPR 2021), connectivity-aware and nearly width-insensitive; a fairer score than IoU for thin/tubular structures. NaN if scikit-image is unavailable.
• oversegmentation / undersegmentation — fragmentation counts: GT masks split across >=2 predictions, and predictions spanning >=2 GT masks (each with >=10% area overlap). The headline over-/under- segmentation failure mode is invisible to the 1-to-1 match.
• per_size — COCO small/medium/large breakdown of GT count, TP count, and TP-only mean IoU (buckets sum to the GT total).

Returns:

Type	Description
dict	A dict with mean_iou, min, max, percentiles p25/ p50/p75, mean_iou_all_gt, pq/sq/rq, mean_boundary_iou, oversegmentation/undersegmentation, per_size, the TP count n_matched (plus n_fp/n_fn), and the raw ious array. Quantities are NaN when undefined.

Source code in sleap_nn/evaluation.py

def mask_metrics(self) -> dict:
    """Compute mask-IoU summary statistics for ``match_method="mask"``.

    Reports complementary IoU summaries, panoptic-quality, boundary-IoU,
    fragmentation, and per-object-size breakdowns:

    * ``mean_iou`` (and ``min``/``max``/percentiles) over the matched (TP)
      pairs only — COCO-style segmentation quality, blind to misses.
    * ``mean_iou_all_gt`` — IoU averaged over *all* ground-truth masks,
      where an unmatched GT (a miss) contributes ``0``. This penalizes
      recall and complements the TP-only mean.
    * Panoptic Quality ``pq = sq * rq`` with ``sq = mean_iou`` (segmentation
      quality) and ``rq = TP / (TP + 0.5*FP + 0.5*FN)`` (recognition
      quality, == detection F1). See Kirillov et al., "Panoptic
      Segmentation" (2019).
    * ``mean_boundary_iou`` — boundary IoU over the matched pairs (Cheng et
      al., 2021), more sensitive to contour error than mask IoU.
    * ``mean_cldice`` — centerline Dice over the matched pairs (Shit et al.,
      CVPR 2021), connectivity-aware and nearly width-insensitive; a fairer
      score than IoU for thin/tubular structures. NaN if scikit-image is
      unavailable.
    * ``oversegmentation`` / ``undersegmentation`` — fragmentation counts:
      GT masks split across >=2 predictions, and predictions spanning >=2
      GT masks (each with >=10% area overlap). The headline over-/under-
      segmentation failure mode is invisible to the 1-to-1 match.
    * ``per_size`` — COCO small/medium/large breakdown of GT count, TP
      count, and TP-only mean IoU (buckets sum to the GT total).

    Returns:
        A dict with ``mean_iou``, ``min``, ``max``, percentiles ``p25``/
        ``p50``/``p75``, ``mean_iou_all_gt``, ``pq``/``sq``/``rq``,
        ``mean_boundary_iou``, ``oversegmentation``/``undersegmentation``,
        ``per_size``, the TP count ``n_matched`` (plus ``n_fp``/``n_fn``),
        and the raw ``ious`` array. Quantities are NaN when undefined.
    """
    ious = np.asarray(self.mask_ious, dtype=float)
    n_tp = len(self.positive_pairs)
    n_fp = len(self.false_positives)
    n_fn = len(self.false_negatives)
    over, under = self._fragmentation_counts()
    results = {
        "mean_iou": np.nan,
        "min": np.nan,
        "max": np.nan,
        "p25": np.nan,
        "p50": np.nan,
        "p75": np.nan,
        "mean_iou_all_gt": np.nan,
        "pq": np.nan,
        "sq": np.nan,
        "rq": np.nan,
        "mean_boundary_iou": np.nan,
        "mean_cldice": np.nan,
        "oversegmentation": over,
        "undersegmentation": under,
        "per_size": self._mask_per_size_stats(),
        "n_matched": int(ious.size),
        "n_fp": n_fp,
        "n_fn": n_fn,
        "ious": ious,
    }
    if ious.size:
        results["mean_iou"] = float(np.mean(ious))
        results["min"] = float(np.min(ious))
        results["max"] = float(np.max(ious))
        for ptile in (25, 50, 75):
            results[f"p{ptile}"] = float(np.percentile(ious, ptile))

    if self._matched_mask_pairs:
        boundary_ious = np.array(
            [_boundary_iou(p, g) for p, g in self._matched_mask_pairs],
            dtype=float,
        )
        results["mean_boundary_iou"] = float(np.mean(boundary_ious))
        # Centerline Dice (clDice): connectivity/width-tolerant, fairer than
        # IoU for thin structures. NaN entries (scikit-image missing) drop out.
        cldices = np.array(
            [mask_cldice(p, g) for p, g in self._matched_mask_pairs],
            dtype=float,
        )
        cldices = cldices[~np.isnan(cldices)]
        if cldices.size:
            results["mean_cldice"] = float(np.mean(cldices))

    iou_sum = float(np.sum(ious)) if ious.size else 0.0
    # Miss-penalizing mean: averaged over every GT mask (TP + FN).
    n_gt = n_tp + n_fn
    if n_gt > 0:
        results["mean_iou_all_gt"] = iou_sum / n_gt
    # Panoptic quality: SQ = TP-only mean IoU, RQ = detection F1, PQ = SQ*RQ
    # = iou_sum / (TP + 0.5*FP + 0.5*FN).
    pq_denom = n_tp + 0.5 * n_fp + 0.5 * n_fn
    if pq_denom > 0:
        results["sq"] = results["mean_iou"]
        results["rq"] = n_tp / pq_denom
        results["pq"] = iou_sum / pq_denom
    return results

mask_voc_metrics(iou_thresholds=MASK_IOU_THRESHOLDS, recall_thresholds=np.linspace(0, 1, 101), size_percentiles=DEFAULT_SIZE_PERCENTILES)

COCO-style score-ranked mask Average Precision / Recall.

Re-matches predictions to GT independently at each IoU threshold (:meth:_match_masks_coco), score-ranks the resulting TP/FP flags, and integrates the precision-recall curve (101-point interpolation, mirrors :meth:voc_metrics). Reports overall AP@[.5:.95]/AP50/AP75/AR plus a per-object-size AP breakdown under two bucketing schemes (GT outside a bucket is ignored, as in pycocotools areaRng): the primary (default) buckets use dataset-relative percentile edges (terciles), and the COCO fixed-cutoff buckets are reported additionally under the mask_voc.coco. prefix — analogous to the dual OKS/PCK VOC.

Parameters:

Name	Type	Description	Default
iou_thresholds	ndarray	IoU thresholds to average AP over.	MASK_IOU_THRESHOLDS
recall_thresholds	ndarray	Recall grid for 101-point interpolation.	linspace(0, 1, 101)
size_percentiles	Tuple[float, float]	Two percentiles of the GT area distribution delimiting the primary small/medium/large buckets.	DEFAULT_SIZE_PERCENTILES

Returns:

Type	Description
dict	A dict keyed under "mask_voc.": AP (per-threshold array), mAP, AP50, AP75, AR, recalls, iou_thresholds, n_gt; primary per-size AP_small/AP_medium/AP_large, n_gt_small/..._medium/..._large, size_scheme and size_edges; and COCO per-size coco.AP_*/coco.n_gt_*/ coco.size_edges. AP values are NaN when the relevant GT set is empty.

Source code in sleap_nn/evaluation.py

def mask_voc_metrics(
    self,
    iou_thresholds: np.ndarray = MASK_IOU_THRESHOLDS,
    recall_thresholds: np.ndarray = np.linspace(0, 1, 101),
    size_percentiles: Tuple[float, float] = DEFAULT_SIZE_PERCENTILES,
) -> dict:
    """COCO-style score-ranked mask Average Precision / Recall.

    Re-matches predictions to GT independently at each IoU threshold
    (:meth:`_match_masks_coco`), score-ranks the resulting TP/FP flags, and
    integrates the precision-recall curve (101-point interpolation, mirrors
    :meth:`voc_metrics`). Reports overall AP@[.5:.95]/AP50/AP75/AR plus a
    per-object-size AP breakdown under two bucketing schemes (GT outside a
    bucket is ignored, as in ``pycocotools`` ``areaRng``): the primary
    (default) buckets use dataset-relative percentile edges (terciles), and
    the COCO fixed-cutoff buckets are reported additionally under the
    ``mask_voc.coco.`` prefix — analogous to the dual OKS/PCK VOC.

    Args:
        iou_thresholds: IoU thresholds to average AP over.
        recall_thresholds: Recall grid for 101-point interpolation.
        size_percentiles: Two percentiles of the GT area distribution
            delimiting the primary small/medium/large buckets.

    Returns:
        A dict keyed under ``"mask_voc."``: ``AP`` (per-threshold array),
        ``mAP``, ``AP50``, ``AP75``, ``AR``, ``recalls``, ``iou_thresholds``,
        ``n_gt``; primary per-size ``AP_small``/``AP_medium``/``AP_large``,
        ``n_gt_small``/``..._medium``/``..._large``, ``size_scheme`` and
        ``size_edges``; and COCO per-size ``coco.AP_*``/``coco.n_gt_*``/
        ``coco.size_edges``. AP values are NaN when the relevant GT set is
        empty.
    """
    iou_thresholds = np.asarray(iou_thresholds, dtype=float)
    recall_thresholds = np.asarray(recall_thresholds, dtype=float)
    gt_areas_all = np.array(
        [a for f in self._mask_frames for a in f["gt_areas"]], dtype=float
    )
    npig = int(gt_areas_all.size)

    # Primary (percentile, dataset-relative) + additional (COCO) edges.
    schemes = {
        "percentile": _percentile_size_edges(gt_areas_all, size_percentiles),
        "coco": COCO_SIZE_EDGES,
    }
    n_gt_size = {
        name: [
            int(np.count_nonzero(_size_mask(gt_areas_all, i, edges)))
            for i in range(len(_SIZE_KEYS))
        ]
        for name, edges in schemes.items()
    }

    ap_overall = np.full(iou_thresholds.size, np.nan)
    recall_overall = np.full(iou_thresholds.size, np.nan)
    ap_size = {
        name: [np.full(iou_thresholds.size, np.nan) for _ in _SIZE_KEYS]
        for name in schemes
    }

    for ti, thr in enumerate(iou_thresholds):
        scores, matched, matched_gt_area, pred_area = self._match_masks_coco(
            float(thr)
        )
        ap_overall[ti], recall_overall[ti] = _ap_from_pr(
            scores, matched, npig, recall_thresholds
        )
        for name, edges in schemes.items():
            for i in range(len(_SIZE_KEYS)):
                # COCO areaRng: keep TPs whose matched GT is in-bucket and
                # FPs whose own area is in-bucket; ignore everything else.
                keep_tp = matched & _size_mask(matched_gt_area, i, edges)
                keep_fp = (~matched) & _size_mask(pred_area, i, edges)
                keep = keep_tp | keep_fp
                ap_size[name][i][ti], _ = _ap_from_pr(
                    scores[keep],
                    keep_tp[keep],
                    n_gt_size[name][i],
                    recall_thresholds,
                )

    def _nanmean(arr: np.ndarray) -> float:
        return float(np.nanmean(arr)) if np.any(~np.isnan(arr)) else np.nan

    def _at(target: float) -> float:
        return float(ap_overall[int(np.argmin(np.abs(iou_thresholds - target)))])

    results = {
        "mask_voc.iou_thresholds": iou_thresholds,
        "mask_voc.AP": ap_overall,
        "mask_voc.recalls": recall_overall,
        "mask_voc.mAP": _nanmean(ap_overall),
        "mask_voc.AR": _nanmean(recall_overall),
        "mask_voc.AP50": _at(0.5),
        "mask_voc.AP75": _at(0.75),
        "mask_voc.n_gt": npig,
        "mask_voc.size_scheme": "percentile",
        "mask_voc.size_edges": [float(e) for e in schemes["percentile"]],
        "mask_voc.coco.size_edges": [float(e) for e in schemes["coco"]],
    }
    # Primary (percentile) per-size keys are unprefixed; COCO is additional.
    for name, prefix in (("percentile", "mask_voc."), ("coco", "mask_voc.coco.")):
        for i, bucket in enumerate(_SIZE_KEYS):
            results[f"{prefix}AP_{bucket}"] = _nanmean(ap_size[name][i])
            results[f"{prefix}n_gt_{bucket}"] = n_gt_size[name][i]
    return results

pck_metrics(thresholds=np.linspace(1, 10, 10))

Compute PCK across a range of thresholds using the pair-wise distances.

Parameters:

Name	Type	Description	Default
thresholds	ndarray	A list of distance thresholds in pixels.	linspace(1, 10, 10)

Returns:

Type	Description
	A dictionary of PCK metrics evaluated at each threshold.

Source code in sleap_nn/evaluation.py

def pck_metrics(self, thresholds: np.ndarray = np.linspace(1, 10, 10)):
    """Compute PCK across a range of thresholds using the pair-wise distances.

    Args:
        thresholds: A list of distance thresholds in pixels.

    Returns:
        A dictionary of PCK metrics evaluated at each threshold.
    """
    dists = self.dists_dict["dists"]
    dists = np.copy(dists)
    dists[np.isnan(dists)] = np.inf
    pcks = np.expand_dims(dists, -1) < np.reshape(thresholds, (1, 1, -1))

    # Guard the empty-match case (0 positive pairs for the whole split) so
    # the nested .mean() reductions below don't hit "Mean of empty slice".
    if dists.size == 0:
        mPCK_parts = np.array([])
        mPCK = np.nan
        pck5 = np.nan
        pck10 = np.nan
    else:
        mPCK_parts = pcks.mean(axis=0).mean(axis=-1)
        mPCK = float(mPCK_parts.mean())

        # Precompute PCK at common thresholds
        idx_5 = np.argmin(np.abs(thresholds - 5))
        idx_10 = np.argmin(np.abs(thresholds - 10))
        pck5 = float(pcks[:, :, idx_5].mean())
        pck10 = float(pcks[:, :, idx_10].mean())

    return {
        "thresholds": thresholds,
        "pcks": pcks,
        "mPCK_parts": mPCK_parts,
        "mPCK": mPCK,
        "PCK@5": pck5,
        "PCK@10": pck10,
    }

semantic_metrics()

Aggregate whole-frame foreground metrics for match_method="semantic".

Averages the per-frame foreground IoU, centerline Dice (clDice), and boundary IoU computed by :meth:_process_frames_semantic over all frames with non-empty ground-truth foreground. clDice entries that are NaN (scikit-image unavailable) are dropped from the clDice mean; if every entry is NaN the reported mean_cldice is NaN.

Returns:

Type	Description
dict	A dict with mean_iou, mean_cldice, mean_boundary_iou, the per-frame ious / cldices / boundary_ious arrays, and n_frames (frames scored). Means are NaN when no frame was scored.

Source code in sleap_nn/evaluation.py

def semantic_metrics(self) -> dict:
    """Aggregate whole-frame foreground metrics for ``match_method="semantic"``.

    Averages the per-frame foreground IoU, centerline Dice (clDice), and
    boundary IoU computed by :meth:`_process_frames_semantic` over all frames
    with non-empty ground-truth foreground. clDice entries that are NaN
    (scikit-image unavailable) are dropped from the clDice mean; if every entry
    is NaN the reported ``mean_cldice`` is NaN.

    Returns:
        A dict with ``mean_iou``, ``mean_cldice``, ``mean_boundary_iou``, the
        per-frame ``ious`` / ``cldices`` / ``boundary_ious`` arrays, and
        ``n_frames`` (frames scored). Means are NaN when no frame was scored.
    """
    rows = np.asarray(self._semantic_rows, dtype=float).reshape(-1, 3)
    ious = rows[:, 0]
    cldices = rows[:, 1]
    bious = rows[:, 2]
    cld_valid = cldices[~np.isnan(cldices)]
    return {
        "mean_iou": float(np.mean(ious)) if ious.size else float("nan"),
        "mean_cldice": (
            float(np.mean(cld_valid)) if cld_valid.size else float("nan")
        ),
        "mean_boundary_iou": (
            float(np.mean(bious)) if bious.size else float("nan")
        ),
        "ious": ious,
        "cldices": cldices,
        "boundary_ious": bious,
        "n_frames": int(ious.size),
    }

visibility_metrics()

Compute node visibility metrics for the matched pair of instances.

Returns:

Type	Description
	A dictionary of visibility metrics, including the confusion matrix.

Source code in sleap_nn/evaluation.py

def visibility_metrics(self):
    """Compute node visibility metrics for the matched pair of instances.

    Returns:
        A dictionary of visibility metrics, including the confusion matrix.
    """
    vis_tp = 0
    vis_fn = 0
    vis_fp = 0
    vis_tn = 0

    for instance_gt, instance_pr, _ in self.positive_pairs:
        missing_nodes_gt = np.isnan(instance_gt.instance.numpy()).any(axis=-1)
        missing_nodes_pr = np.isnan(instance_pr.instance.numpy()).any(axis=-1)

        vis_tn += ((missing_nodes_gt) & (missing_nodes_pr)).sum()
        vis_fn += ((~missing_nodes_gt) & (missing_nodes_pr)).sum()
        vis_fp += ((missing_nodes_gt) & (~missing_nodes_pr)).sum()
        vis_tp += ((~missing_nodes_gt) & (~missing_nodes_pr)).sum()

    return {
        "tp": vis_tp,
        "fp": vis_fp,
        "tn": vis_tn,
        "fn": vis_fn,
        "precision": vis_tp / (vis_tp + vis_fp) if (vis_tp + vis_fp) else np.nan,
        "recall": vis_tp / (vis_tp + vis_fn) if (vis_tp + vis_fn) else np.nan,
    }

voc_metrics(match_score_by='oks', match_score_thresholds=np.linspace(0.5, 0.95, 10), recall_thresholds=np.linspace(0, 1, 101))

Compute VOC metrics for a matched pairs of instances positive pairs and false negatives.

Parameters:

Name	Type	Description	Default
match_score_by		The score to be used for computing the metrics. "ock" or "pck"	'oks'
match_score_thresholds	ndarray	Score thresholds at which to consider matches as a true positive match.	linspace(0.5, 0.95, 10)
recall_thresholds	ndarray	Recall thresholds at which to evaluate Average Precision.	linspace(0, 1, 101)

Returns:

Type	Description
	A dictionary of VOC metrics.

Source code in sleap_nn/evaluation.py

def voc_metrics(
    self,
    match_score_by="oks",
    match_score_thresholds: np.ndarray = np.linspace(
        0.5, 0.95, 10
    ),  # 0.5:0.05:0.95
    recall_thresholds: np.ndarray = np.linspace(0, 1, 101),  # 0.0:0.01:1.00
):
    """Compute VOC metrics for a matched pairs of instances positive pairs and false negatives.

    Args:
        match_score_by: The score to be used for computing the metrics. "ock" or "pck"
        match_score_thresholds: Score thresholds at which to consider matches as a true
            positive match.
        recall_thresholds: Recall thresholds at which to evaluate Average Precision.

    Returns:
        A dictionary of VOC metrics.
    """
    if match_score_by == "oks":
        match_scores = np.array([oks for _, _, oks in self.positive_pairs])
        name = "oks_voc"
    elif match_score_by == "pck":
        name = "pck_voc"
        if not self.positive_pairs:
            # Guard the empty-match case: the (n_pairs, n_nodes, n_thresholds)
            # ``pcks`` array is empty along the pairs axis, so reducing it with
            # nested .mean() calls would hit "Mean of empty slice".
            match_scores = np.array([])
        else:
            pck_metrics = self.pck_metrics()
            match_scores = pck_metrics["pcks"].mean(axis=-1).mean(axis=-1)
    else:
        message = "Invalid Option for match_score_by. Choose either `oks` or `pck`"
        logger.error(message)
        raise Exception(message)

    detection_scores = np.array(
        [pp[1].instance.score for pp in self.positive_pairs]
    )

    inds = np.argsort(-detection_scores, kind="mergesort")
    detection_scores = detection_scores[inds]
    match_scores = match_scores[inds]

    precisions = []
    recalls = []

    npig = len(self.positive_pairs) + len(
        self.false_negatives
    )  # total number of GT instances

    for match_score_threshold in match_score_thresholds:
        tp = np.cumsum(match_scores >= match_score_threshold)
        fp = np.cumsum(match_scores < match_score_threshold)

        if tp.size == 0:
            return {
                name + ".match_score_thresholds": 0,
                name + ".recall_thresholds": 0,
                name + ".match_scores": 0,
                name + ".precisions": 0,
                name + ".recalls": 0,
                name + ".AP": 0,
                name + ".AR": 0,
                name + ".mAP": 0,
                name + ".mAR": 0,
            }

        rc = tp / npig
        pr = tp / (fp + tp + np.spacing(1))

        recall = rc[-1]  # best recall at this OKS threshold

        # Ensure strictly decreasing precisions.
        for i in range(len(pr) - 1, 0, -1):
            if pr[i] > pr[i - 1]:
                pr[i - 1] = pr[i]

        # Find best precision at each recall threshold.
        rc_inds = np.searchsorted(rc, recall_thresholds, side="left")
        precision = np.zeros(rc_inds.shape)
        is_valid_rc_ind = rc_inds < len(pr)
        precision[is_valid_rc_ind] = pr[rc_inds[is_valid_rc_ind]]

        precisions.append(precision)
        recalls.append(recall)

    precisions = np.array(precisions)
    recalls = np.array(recalls)

    AP = precisions.mean(
        axis=1
    )  # AP = average precision over fixed set of recall thresholds
    AR = recalls  # AR = max recall given a fixed number of detections per image

    mAP = precisions.mean()  # mAP = mean over all OKS thresholds
    mAR = recalls.mean()  # mAR = mean over all OKS thresholds

    return {
        name + ".match_score_thresholds": match_score_thresholds,
        name + ".recall_thresholds": recall_thresholds,
        name + ".match_scores": match_scores,
        name + ".precisions": precisions,
        name + ".recalls": recalls,
        name + ".AP": AP,
        name + ".AR": AR,
        name + ".mAP": mAP,
        name + ".mAR": mAR,
    }

MatchInstance

Class to have a new structure for sio.Instance object.

Source code in sleap_nn/evaluation.py

@attrs.define(auto_attribs=True, slots=True)
class MatchInstance:
    """Class to have a new structure for sio.Instance object."""

    instance: sio.Instance
    frame_idx: int
    video_path: str

compute_dists(positive_pairs)

Compute Euclidean distances between matched pairs of instances.

Parameters:

Name	Type	Description	Default
positive_pairs	List[Tuple[Instance, PredictedInstance, Any]]	A list of tuples of the form (instance_gt, instance_pr, _) containing the matched pair of instances.	required

Returns:

Type	Description
Dict[str, Union[ndarray, List[int], List[str]]]	A dictionary with the following keys: dists: An array of pairwise distances of shape (n_positive_pairs, n_nodes) frame_idxs: A list of frame indices corresponding to the dists video_paths: A list of video paths corresponding to the dists

Source code in sleap_nn/evaluation.py

def compute_dists(
    positive_pairs: List[Tuple[sio.Instance, sio.PredictedInstance, Any]],
) -> Dict[str, Union[np.ndarray, List[int], List[str]]]:
    """Compute Euclidean distances between matched pairs of instances.

    Args:
        positive_pairs: A list of tuples of the form `(instance_gt, instance_pr, _)`
            containing the matched pair of instances.

    Returns:
        A dictionary with the following keys:
            dists: An array of pairwise distances of shape `(n_positive_pairs, n_nodes)`
            frame_idxs: A list of frame indices corresponding to the `dists`
            video_paths: A list of video paths corresponding to the `dists`
    """
    dists = []
    frame_idxs = []
    video_paths = []
    for instance_gt, instance_pr, _ in positive_pairs:
        points_gt = instance_gt.instance.numpy()
        points_pr = instance_pr.instance.numpy()

        dists.append(np.linalg.norm(points_pr - points_gt, axis=-1))
        frame_idxs.append(instance_gt.frame_idx)
        video_paths.append(instance_gt.video_path)

    dists = np.array(dists)

    # Bundle everything into a dictionary
    dists_dict = {
        "dists": dists,
        "frame_idxs": frame_idxs,
        "video_paths": video_paths,
    }

    return dists_dict

compute_gt_centroids(instance_gt_points, anchor_ind)

Compute ground-truth centroids mirroring generate_centroids in numpy.

This is the numpy mirror of :func:sleap_nn.data.instance_centroids.generate_centroids. The centroid's MEANING is defined by that function (see also #586): when the configured anchor node is present (non-NaN) it is that node; otherwise the centroid falls back to the NaN-ignoring MEAN of visible nodes (NOT the bounding-box midpoint). The fallback is computed via np.nanmean over the node axis, and is NaN only when every node of an instance is NaN.

Parameters:

Name	Type	Description	Default
instance_gt_points	ndarray	Ground-truth keypoints of shape (n_instances, n_nodes, 2) or (n_nodes, 2). Missing/occluded nodes are NaN.	required
anchor_ind	Optional[int]	Index of the node to use as the anchor. If None, or if the anchor node is NaN for a given instance, the centroid falls back to the NaN-ignoring mean of visible nodes for that instance.	required

Returns:

Type	Description
ndarray	Centroids of shape (n_instances, 2) (or (2,) for a single instance input), reducing the node axis.

Source code in sleap_nn/evaluation.py

def compute_gt_centroids(
    instance_gt_points: np.ndarray, anchor_ind: Optional[int]
) -> np.ndarray:
    """Compute ground-truth centroids mirroring ``generate_centroids`` in numpy.

    This is the numpy mirror of
    :func:`sleap_nn.data.instance_centroids.generate_centroids`. The centroid's
    MEANING is defined by that function (see also #586): when the configured
    anchor node is present (non-NaN) it is that node; otherwise the centroid
    falls back to the NaN-ignoring MEAN of visible nodes (NOT the bounding-box
    midpoint). The fallback is computed via ``np.nanmean`` over the node axis,
    and is NaN only when every node of an instance is NaN.

    Args:
        instance_gt_points: Ground-truth keypoints of shape ``(n_instances,
            n_nodes, 2)`` or ``(n_nodes, 2)``. Missing/occluded nodes are NaN.
        anchor_ind: Index of the node to use as the anchor. If ``None``, or if
            the anchor node is NaN for a given instance, the centroid falls back
            to the NaN-ignoring mean of visible nodes for that instance.

    Returns:
        Centroids of shape ``(n_instances, 2)`` (or ``(2,)`` for a single
        instance input), reducing the node axis.
    """
    points = np.asarray(instance_gt_points, dtype=np.float64)

    if anchor_ind is not None:
        centroids = points[..., anchor_ind, :].copy()
    else:
        centroids = np.full(points.shape[:-2] + (2,), np.nan, dtype=points.dtype)

    missing_anchors = np.isnan(centroids).any(axis=-1)
    if np.any(missing_anchors):
        # NaN-ignoring mean of visible nodes. np.nanmean over the node axis
        # yields NaN only when all nodes for that instance are NaN (matching
        # find_points_mean). Suppress the all-NaN-slice RuntimeWarning.
        import warnings

        with warnings.catch_warnings():
            warnings.simplefilter("ignore", category=RuntimeWarning)
            mean_fallback = np.nanmean(points, axis=-2)
        centroids[missing_anchors] = mean_fallback[missing_anchors]

    return centroids

compute_instance_area(points)

Compute the area of the bounding box of a set of keypoints.

Parameters:

Name	Type	Description	Default
points	ndarray	A numpy array of coordinates.	required

Returns:

Type	Description
ndarray	The area of the bounding box of the points.

Source code in sleap_nn/evaluation.py

def compute_instance_area(points: np.ndarray) -> np.ndarray:
    """Compute the area of the bounding box of a set of keypoints.

    Args:
        points: A numpy array of coordinates.

    Returns:
        The area of the bounding box of the points.
    """
    if points.ndim == 2:
        points = np.expand_dims(points, axis=0)

    min_pt = np.nanmin(points, axis=-2)
    max_pt = np.nanmax(points, axis=-2)

    return np.prod(max_pt - min_pt, axis=-1)

compute_oks(points_gt, points_pr, scale=None, stddev=0.025, use_cocoeval=True)

Compute the object keypoints similarity between sets of points.

Parameters:

Name	Type	Description	Default
points_gt	ndarray	Ground truth instances of shape (n_gt, n_nodes, n_ed), where n_nodes is the number of body parts/keypoint types, and n_ed is the number of Euclidean dimensions (typically 2 or 3). Keypoints that are missing/not visible should be represented as NaNs.	required
points_pr	ndarray	Predicted instance of shape (n_pr, n_nodes, n_ed).	required
use_cocoeval	bool	Indicates whether the OKS score is calculated like cocoeval method or not. True indicating the score is calculated using the cocoeval method (widely used and the code can be found here at https://github.com/cocodataset/cocoapi/blob/8c9bcc3cf640524c4c20a9c40e89cb6a2f2fa0e9/PythonAPI/pycocotools/cocoeval.py#L192C5-L233C20) and False indicating the score is calculated using the method exactly as given in the paper referenced in the Notes below.	True
scale	Optional[float]	Size scaling factor to use when weighing the scores, typically the area of the bounding box of the instance (in pixels). This should be of the length n_gt. If a scalar is provided, the same number is used for all ground truth instances. If set to None, the bounding box area of the ground truth instances will be calculated.	None
stddev	float	The standard deviation associated with the spread in the localization accuracy of each node/keypoint type. This should be of the length n_nodes. "Easier" keypoint types will have lower values to reflect the smaller spread expected in localizing it.	0.025

Returns:

Type	Description
ndarray	The object keypoints similarity between every pair of ground truth and predicted instance, a numpy array of of shape (n_gt, n_pr) in the range of [0, 1.0], with 1.0 denoting a perfect match.

Notes

It's important to set the stddev appropriately when accounting for the difficulty of each keypoint type. For reference, the median value for all keypoint types in COCO is 0.072. The "easiest" keypoint is the left eye, with stddev of 0.025, since it is easy to precisely locate the eyes when labeling. The "hardest" keypoint is the left hip, with stddev of 0.107, since it's hard to locate the left hip bone without external anatomical features and since it is often occluded by clothing.

The implementation here is based off of the descriptions in: Ronch & Perona. "Benchmarking and Error Diagnosis in Multi-Instance Pose Estimation." ICCV (2017).

Source code in sleap_nn/evaluation.py

def compute_oks(
    points_gt: np.ndarray,
    points_pr: np.ndarray,
    scale: Optional[float] = None,
    stddev: float = 0.025,
    use_cocoeval: bool = True,
) -> np.ndarray:
    """Compute the object keypoints similarity between sets of points.

    Args:
        points_gt: Ground truth instances of shape (n_gt, n_nodes, n_ed),
            where n_nodes is the number of body parts/keypoint types, and n_ed
            is the number of Euclidean dimensions (typically 2 or 3). Keypoints
            that are missing/not visible should be represented as NaNs.
        points_pr: Predicted instance of shape (n_pr, n_nodes, n_ed).
        use_cocoeval: Indicates whether the OKS score is calculated like cocoeval
            method or not. True indicating the score is calculated using the
            cocoeval method (widely used and the code can be found here at
            https://github.com/cocodataset/cocoapi/blob/8c9bcc3cf640524c4c20a9c40e89cb6a2f2fa0e9/PythonAPI/pycocotools/cocoeval.py#L192C5-L233C20)
            and False indicating the score is calculated using the method exactly
            as given in the paper referenced in the Notes below.
        scale: Size scaling factor to use when weighing the scores, typically
            the area of the bounding box of the instance (in pixels). This
            should be of the length n_gt. If a scalar is provided, the same
            number is used for all ground truth instances. If set to None, the
            bounding box area of the ground truth instances will be calculated.
        stddev: The standard deviation associated with the spread in the
            localization accuracy of each node/keypoint type. This should be of
            the length n_nodes. "Easier" keypoint types will have lower values
            to reflect the smaller spread expected in localizing it.

    Returns:
        The object keypoints similarity between every pair of ground truth and
        predicted instance, a numpy array of of shape (n_gt, n_pr) in the range
        of [0, 1.0], with 1.0 denoting a perfect match.

    Notes:
        It's important to set the stddev appropriately when accounting for the
        difficulty of each keypoint type. For reference, the median value for
        all keypoint types in COCO is 0.072. The "easiest" keypoint is the left
        eye, with stddev of 0.025, since it is easy to precisely locate the
        eyes when labeling. The "hardest" keypoint is the left hip, with stddev
        of 0.107, since it's hard to locate the left hip bone without external
        anatomical features and since it is often occluded by clothing.

        The implementation here is based off of the descriptions in:
        Ronch & Perona. "Benchmarking and Error Diagnosis in Multi-Instance Pose
        Estimation." ICCV (2017).
    """
    if points_gt.ndim == 2:
        points_gt = np.expand_dims(points_gt, axis=0)
    if points_pr.ndim == 2:
        points_pr = np.expand_dims(points_pr, axis=0)

    if scale is None:
        scale = compute_instance_area(points_gt)

    n_gt, n_nodes, n_ed = points_gt.shape  # n_ed = 2 or 3 (euclidean dimensions)
    n_pr = points_pr.shape[0]

    # If scalar scale was provided, use the same for each ground truth instance.
    if np.isscalar(scale):
        scale = np.full(n_gt, scale)

    # If scalar standard deviation was provided, use the same for each node.
    if np.isscalar(stddev):
        stddev = np.full(n_nodes, stddev)

    # Compute displacement between each pair.
    displacement = np.reshape(points_gt, (n_gt, 1, n_nodes, n_ed)) - np.reshape(
        points_pr, (1, n_pr, n_nodes, n_ed)
    )
    assert displacement.shape == (n_gt, n_pr, n_nodes, n_ed)

    # Convert to pairwise Euclidean distances.
    distance = (displacement**2).sum(axis=-1)  # (n_gt, n_pr, n_nodes)
    assert distance.shape == (n_gt, n_pr, n_nodes)

    # Compute the normalization factor per keypoint.
    if use_cocoeval:
        # If use_cocoeval is True, then compute normalization factor according to cocoeval.
        spread_factor = (2 * stddev) ** 2
        scale_factor = 2 * (scale + np.spacing(1))
    else:
        # If use_cocoeval is False, then compute normalization factor according to the paper.
        spread_factor = stddev**2
        scale_factor = 2 * ((scale + np.spacing(1)) ** 2)
    normalization_factor = np.reshape(spread_factor, (1, 1, n_nodes)) * np.reshape(
        scale_factor, (n_gt, 1, 1)
    )
    assert normalization_factor.shape == (n_gt, 1, n_nodes)

    # Since a "miss" is considered as KS < 0.5, we'll set the
    # distances for predicted points that are missing to inf.
    missing_pr = np.any(np.isnan(points_pr), axis=-1)  # (n_pr, n_nodes)
    assert missing_pr.shape == (n_pr, n_nodes)
    distance[:, missing_pr] = np.inf

    # Compute the keypoint similarity as per the top of Eq. 1.
    ks = np.exp(-(distance / normalization_factor))  # (n_gt, n_pr, n_nodes)
    assert ks.shape == (n_gt, n_pr, n_nodes)

    # Set the KS for missing ground truth points to 0.
    # This is equivalent to the visibility delta function of the bottom
    # of Eq. 1.
    missing_gt = np.any(np.isnan(points_gt), axis=-1)  # (n_gt, n_nodes)
    assert missing_gt.shape == (n_gt, n_nodes)
    ks[np.expand_dims(missing_gt, axis=1)] = 0

    # Compute the OKS.
    n_visible_gt = np.sum(
        (~missing_gt).astype("float32"), axis=-1, keepdims=True
    )  # (n_gt, 1)
    oks = np.sum(ks, axis=-1) / n_visible_gt
    assert oks.shape == (n_gt, n_pr)

    return oks

find_frame_pairs(labels_gt, labels_pr, user_labels_only=True)

Find corresponding frames across two sets of labels.

This function uses sleap-io's robust video matching API to handle various scenarios including embedded videos, cross-platform paths, and videos with different metadata.

Parameters:

Name	Type	Description	Default
labels_gt	Labels	A sio.Labels instance with ground truth instances.	required
labels_pr	Labels	A sio.Labels instance with predicted instances.	required
user_labels_only	bool	If False, frames with predicted instances in labels_gt will also be considered for matching.	True

Returns:

Type	Description
List[Tuple[LabeledFrame, LabeledFrame]]	A list of pairs of sio.LabeledFrames in the form (frame_gt, frame_pr).

Source code in sleap_nn/evaluation.py

def find_frame_pairs(
    labels_gt: sio.Labels, labels_pr: sio.Labels, user_labels_only: bool = True
) -> List[Tuple[sio.LabeledFrame, sio.LabeledFrame]]:
    """Find corresponding frames across two sets of labels.

    This function uses sleap-io's robust video matching API to handle various
    scenarios including embedded videos, cross-platform paths, and videos with
    different metadata.

    Args:
        labels_gt: A `sio.Labels` instance with ground truth instances.
        labels_pr: A `sio.Labels` instance with predicted instances.
        user_labels_only: If False, frames with predicted instances in `labels_gt` will
            also be considered for matching.

    Returns:
        A list of pairs of `sio.LabeledFrame`s in the form `(frame_gt, frame_pr)`.
    """
    # Use sleap-io's robust video matching API (added in 0.6.2)
    # The match() method returns a MatchResult with video_map: {pred_video: gt_video}
    #
    # NOTE: sleap-io's AUTO matcher previously shape-rejected candidates before its
    # definitive is_same_file check, so it failed to pair an embedded-subset GT video
    # with its restored-original prediction counterpart (same file, different frame
    # count) -- e.g. post-training eval on an embedded .pkg.slp logged "Empty Frame
    # Pairs". This is resolved by the pinned sleap-io (talmolab/sleap-io#473/#476),
    # whose AUTO matcher resolves effective shape through the source_video chain, so
    # the match here works with no workaround.
    match_result = labels_gt.match(labels_pr)

    frame_pairs = []
    # Iterate over matched video pairs (pred_video -> gt_video mapping)
    for video_pr, video_gt in match_result.video_map.items():
        if video_gt is None:
            # No match found for this prediction video
            continue

        # Find labeled frames in this video.
        labeled_frames_gt = labels_gt.find(video_gt)
        if user_labels_only:
            # Build fresh LabeledFrame copies restricted to user instances,
            # rather than mutating `lf.instances` in place -- `labels_gt.find`
            # returns references into the caller's actual Labels object, so
            # mutating it here permanently discards PredictedInstances from
            # ground truth the caller may reuse afterward (e.g. a second
            # Evaluator call with user_labels_only=False on the same labels_gt).
            labeled_frames_gt = [
                attrs.evolve(lf, instances=lf.user_instances)
                for lf in labeled_frames_gt
                if len(lf.user_instances) > 0
            ]

        # Attempt to match each labeled frame in the ground truth.
        for labeled_frame_gt in labeled_frames_gt:
            labeled_frames_pr = labels_pr.find(
                video_pr, frame_idx=labeled_frame_gt.frame_idx
            )

            if not labeled_frames_pr:
                # No match
                continue
            elif len(labeled_frames_pr) == 1:
                # Match!
                frame_pairs.append((labeled_frame_gt, labeled_frames_pr[0]))

    return frame_pairs

get_instances(labeled_frame)

Get a list of instances of type MatchInstance from the Labeled Frame.

Parameters:

Name	Type	Description	Default
labeled_frame	LabeledFrame	Input Labeled frame of type sio.LabeledFrame.	required

Returns:

Type	Description
List[MatchInstance]	List of MatchInstance objects for the given labeled frame.

Source code in sleap_nn/evaluation.py

def get_instances(labeled_frame: sio.LabeledFrame) -> List[MatchInstance]:
    """Get a list of instances of type MatchInstance from the Labeled Frame.

    Args:
        labeled_frame: Input Labeled frame of type sio.LabeledFrame.

    Returns:
        List of MatchInstance objects for the given labeled frame.
    """
    instance_list = []
    frame_idx = labeled_frame.frame_idx

    # Extract video path with fallbacks for embedded videos
    video = labeled_frame.video
    video_path = None
    if video is not None:
        backend = getattr(video, "backend", None)
        if backend is not None:
            # Try source_filename first (for embedded videos with provenance)
            video_path = getattr(backend, "source_filename", None)
            if video_path is None:
                video_path = getattr(backend, "filename", None)
        # Fallback to video.filename if backend doesn't have it
        if video_path is None:
            video_path = getattr(video, "filename", None)
            # Handle list filenames (image sequences)
            if isinstance(video_path, list) and video_path:
                video_path = video_path[0]
    # Final fallback: use a unique identifier
    if video_path is None:
        video_path = f"video_{id(video)}" if video is not None else "unknown"

    for instance in labeled_frame.instances:
        match_instance = MatchInstance(
            instance=instance, frame_idx=frame_idx, video_path=video_path
        )
        instance_list.append(match_instance)
    return instance_list

load_metrics(path, split='test', dataset_idx=0)

Load metrics from a model folder or metrics file.

This function supports both the new format (single "metrics" key) and the old format (individual metric keys at top level). It also handles both old and new file naming conventions in model folders.

Parameters:

Name	Type	Description	Default
path	str	Path to a model folder or metrics file (.npz).	required
split	str	Name of the split to load. Must be "train", "val", or "test". Default: "test". If "test" is not found, falls back to "val". Ignored if path points directly to a .npz file.	'test'
dataset_idx	int	Index of the dataset (for multi-dataset training). Default: 0. Ignored if path points directly to a .npz file.	0

Returns:

Type	Description
dict	Dictionary containing metrics with keys: voc_metrics, mOKS, distance_metrics, pck_metrics, visibility_metrics.

Raises:

Type	Description
FileNotFoundError	If no metrics file is found.

Examples:

>>> # Load from model folder (tries test, falls back to val)
>>> metrics = load_metrics("/path/to/model")
>>> print(metrics["mOKS"]["mOKS"])

>>> # Load specific split and dataset
>>> metrics = load_metrics("/path/to/model", split="val", dataset_idx=1)

>>> # Load directly from npz file
>>> metrics = load_metrics("/path/to/metrics.val.0.npz")

Source code in sleap_nn/evaluation.py

def load_metrics(
    path: str,
    split: str = "test",
    dataset_idx: int = 0,
) -> dict:
    """Load metrics from a model folder or metrics file.

    This function supports both the new format (single "metrics" key) and the old
    format (individual metric keys at top level). It also handles both old and new
    file naming conventions in model folders.

    Args:
        path: Path to a model folder or metrics file (.npz).
        split: Name of the split to load. Must be "train", "val", or "test".
            Default: "test". If "test" is not found, falls back to "val".
            Ignored if path points directly to a .npz file.
        dataset_idx: Index of the dataset (for multi-dataset training).
            Default: 0. Ignored if path points directly to a .npz file.

    Returns:
        Dictionary containing metrics with keys: voc_metrics, mOKS,
        distance_metrics, pck_metrics, visibility_metrics.

    Raises:
        FileNotFoundError: If no metrics file is found.

    Examples:
        >>> # Load from model folder (tries test, falls back to val)
        >>> metrics = load_metrics("/path/to/model")
        >>> print(metrics["mOKS"]["mOKS"])

        >>> # Load specific split and dataset
        >>> metrics = load_metrics("/path/to/model", split="val", dataset_idx=1)

        >>> # Load directly from npz file
        >>> metrics = load_metrics("/path/to/metrics.val.0.npz")
    """
    path = Path(path)

    if path.suffix == ".npz":
        metrics_path = path
    else:
        metrics_path = _find_metrics_file(path, split, dataset_idx)

    if not metrics_path.exists():
        raise FileNotFoundError(f"Metrics file not found at {metrics_path}")

    return _load_npz_metrics(metrics_path)

mask_cldice(pred, gt)

Centerline Dice (clDice) between two binary masks.

Shit et al., "clDice — A Novel Topology-Preserving Loss Function for Tubular Structure Segmentation," CVPR 2021 (arXiv:2003.07311). The connectivity-aware F-score of two skeleton-overlap terms:

• Tprec = fraction of the predicted skeleton lying inside the GT mask (is my centerline drawn on a real object?),
• Tsens = fraction of the GT skeleton lying inside the predicted mask (did I cover every real object along its length?),

with clDice = 2·Tprec·Tsens / (Tprec + Tsens). Nearly width-insensitive and connectivity-sensitive, so it is a fairer quality measure than area IoU for thin/tubular structures (roots, vessels, neurites). Uses a hard morphological skeleton (exact, no k to tune).

Two empty masks return 1.0 (matching the _mask_iou "identical -> 1.0" contract). Returns nan when scikit-image is unavailable so callers can drop clDice from the summary without failing.

Source code in sleap_nn/evaluation.py

def mask_cldice(pred: np.ndarray, gt: np.ndarray) -> float:
    """Centerline Dice (clDice) between two binary masks.

    Shit et al., "clDice — A Novel Topology-Preserving Loss Function for Tubular
    Structure Segmentation," CVPR 2021 (arXiv:2003.07311). The connectivity-aware
    F-score of two skeleton-overlap terms:

    * ``Tprec`` = fraction of the *predicted* skeleton lying inside the *GT* mask
      (is my centerline drawn on a real object?),
    * ``Tsens`` = fraction of the *GT* skeleton lying inside the *predicted* mask
      (did I cover every real object along its length?),

    with ``clDice = 2·Tprec·Tsens / (Tprec + Tsens)``. Nearly width-insensitive
    and connectivity-sensitive, so it is a fairer quality measure than area IoU
    for thin/tubular structures (roots, vessels, neurites). Uses a hard
    morphological skeleton (exact, no ``k`` to tune).

    Two empty masks return ``1.0`` (matching the ``_mask_iou`` "identical -> 1.0"
    contract). Returns ``nan`` when scikit-image is unavailable so callers can
    drop clDice from the summary without failing.
    """
    a, b = _align_pair(pred, gt)
    if not a.any() and not b.any():
        return 1.0
    sk_p = _skeletonize(a)
    sk_g = _skeletonize(b)
    if sk_p is None or sk_g is None:
        return float("nan")
    sp, sg = int(sk_p.sum()), int(sk_g.sum())
    if sp == 0 or sg == 0:
        return 0.0
    tprec = int(np.logical_and(sk_p, b).sum()) / sp
    tsens = int(np.logical_and(sk_g, a).sum()) / sg
    if (tprec + tsens) == 0:
        return 0.0
    return float(2.0 * tprec * tsens / (tprec + tsens))

match_centroids(pred_centroids, gt_centroids, max_distance=50.0)

Match predicted centroids to ground truth using Hungarian algorithm.

Parameters:

Name	Type	Description	Default
pred_centroids	ndarray	Predicted centroid locations, shape (n_pred, 2).	required
gt_centroids	ndarray	Ground truth centroid locations, shape (n_gt, 2).	required
max_distance	float	Maximum distance threshold for valid matches (in pixels).	50.0

Returns:

Type	Description
tuple	Tuple of: - matched_pred_indices: Indices of matched predictions - matched_gt_indices: Indices of matched ground truth - unmatched_pred_indices: Indices of unmatched predictions (false positives) - unmatched_gt_indices: Indices of unmatched ground truth (false negatives)

Source code in sleap_nn/evaluation.py

def match_centroids(
    pred_centroids: "np.ndarray",
    gt_centroids: "np.ndarray",
    max_distance: float = 50.0,
) -> tuple:
    """Match predicted centroids to ground truth using Hungarian algorithm.

    Args:
        pred_centroids: Predicted centroid locations, shape (n_pred, 2).
        gt_centroids: Ground truth centroid locations, shape (n_gt, 2).
        max_distance: Maximum distance threshold for valid matches (in pixels).

    Returns:
        Tuple of:
            - matched_pred_indices: Indices of matched predictions
            - matched_gt_indices: Indices of matched ground truth
            - unmatched_pred_indices: Indices of unmatched predictions (false positives)
            - unmatched_gt_indices: Indices of unmatched ground truth (false negatives)
    """
    import numpy as np
    from scipy.optimize import linear_sum_assignment
    from scipy.spatial.distance import cdist

    n_pred = len(pred_centroids)
    n_gt = len(gt_centroids)

    # Handle edge cases
    if n_pred == 0 and n_gt == 0:
        return np.array([]), np.array([]), np.array([]), np.array([])
    if n_pred == 0:
        return np.array([]), np.array([]), np.array([]), np.arange(n_gt)
    if n_gt == 0:
        return np.array([]), np.array([]), np.arange(n_pred), np.array([])

    # Compute pairwise distances
    cost_matrix = cdist(pred_centroids, gt_centroids)

    # Run Hungarian algorithm for optimal matching
    pred_indices, gt_indices = linear_sum_assignment(cost_matrix)

    # Filter matches that exceed max_distance
    matched_pred = []
    matched_gt = []
    for p_idx, g_idx in zip(pred_indices, gt_indices):
        if cost_matrix[p_idx, g_idx] <= max_distance:
            matched_pred.append(p_idx)
            matched_gt.append(g_idx)

    matched_pred = np.array(matched_pred)
    matched_gt = np.array(matched_gt)

    # Find unmatched indices
    all_pred = set(range(n_pred))
    all_gt = set(range(n_gt))
    unmatched_pred = np.array(list(all_pred - set(matched_pred)))
    unmatched_gt = np.array(list(all_gt - set(matched_gt)))

    return matched_pred, matched_gt, unmatched_pred, unmatched_gt

match_frame_pairs(frame_pairs, stddev=0.025, scale=None, threshold=0)

Match all ground truth and predicted instances within each pair of frames.

This is a wrapper for match_instances() but operates on lists of frames.

Parameters:

Name	Type	Description	Default
frame_pairs	List[Tuple[LabeledFrame, LabeledFrame]]	A list of pairs of sleap.LabeledFrames in the form (frame_gt, frame_pr). These can be obtained with find_frame_pairs().	required
stddev	float	The expected spread of coordinates for OKS computation.	0.025
scale	Optional[float]	The scale for normalizing the OKS. If not set, the bounding box area will be used.	None
threshold	float	The minimum OKS between a candidate pair of instances to be considered a match.	0

Returns:

Type	Description
Tuple[List[Tuple[Instance, PredictedInstance, float]], List[Instance]]	A tuple of (positive_pairs, false_negatives). positive_pairs is a list of 3-tuples of the form (instance_gt, instance_pr, oks) containing the matched pair of instances and their OKS. false_negatives is a list of ground truth sio.Instances that could not be matched.

Source code in sleap_nn/evaluation.py

def match_frame_pairs(
    frame_pairs: List[Tuple[sio.LabeledFrame, sio.LabeledFrame]],
    stddev: float = 0.025,
    scale: Optional[float] = None,
    threshold: float = 0,
) -> Tuple[List[Tuple[sio.Instance, sio.PredictedInstance, float]], List[sio.Instance]]:
    """Match all ground truth and predicted instances within each pair of frames.

    This is a wrapper for `match_instances()` but operates on lists of frames.

    Args:
        frame_pairs: A list of pairs of `sleap.LabeledFrame`s in the form
            `(frame_gt, frame_pr)`. These can be obtained with `find_frame_pairs()`.
        stddev: The expected spread of coordinates for OKS computation.
        scale: The scale for normalizing the OKS. If not set, the bounding box area will
            be used.
        threshold: The minimum OKS between a candidate pair of instances to be
            considered a match.

    Returns:
        A tuple of (`positive_pairs`, `false_negatives`).

        `positive_pairs` is a list of 3-tuples of the form
        `(instance_gt, instance_pr, oks)` containing the matched pair of instances and
        their OKS.

        `false_negatives` is a list of ground truth `sio.Instance`s that could not be
        matched.
    """
    positive_pairs = []
    false_negatives = []
    for frame_gt, frame_pr in frame_pairs:
        positive_pairs_frame, false_negatives_frame = match_instances(
            frame_gt,
            frame_pr,
            stddev=stddev,
            scale=scale,
            threshold=threshold,
        )
        positive_pairs.extend(positive_pairs_frame)
        false_negatives.extend(false_negatives_frame)

    return positive_pairs, false_negatives

match_instances(frame_gt, frame_pr, stddev=0.025, scale=None, threshold=0)

Match pairs of instances between ground truth and predictions in a frame.

Parameters:

Name	Type	Description	Default
frame_gt	LabeledFrame	A sio.LabeledFrame with ground truth instances.	required
frame_pr	LabeledFrame	A sio.LabeledFrame with predicted instances.	required
stddev	float	The expected spread of coordinates for OKS computation.	0.025
scale	Optional[float]	The scale for normalizing the OKS. If not set, the bounding box area will be used.	None
threshold	float	The minimum OKS between a candidate pair of instances to be considered a match.	0

Returns:

Type	Description
Tuple[List[Tuple[Instance, PredictedInstance, float]], List[Instance]]	A tuple of (positive_pairs, false_negatives). positive_pairs is a list of 3-tuples of the form (instance_gt, instance_pr, oks) containing the matched pair of instances and their OKS. false_negatives is a list of ground truth sleap.Instances that could not be matched.

Notes

This function uses the approach from the PASCAL VOC scoring procedure. Briefly, predictions are sorted descending by their instance-level prediction scores and greedily matched to ground truth instances which are then removed from the pool of available instances.

Ground truth instances that remain unmatched are considered false negatives.

Source code in sleap_nn/evaluation.py

def match_instances(
    frame_gt: sio.LabeledFrame,
    frame_pr: sio.LabeledFrame,
    stddev: float = 0.025,
    scale: Optional[float] = None,
    threshold: float = 0,
) -> Tuple[List[Tuple[sio.Instance, sio.PredictedInstance, float]], List[sio.Instance]]:
    """Match pairs of instances between ground truth and predictions in a frame.

    Args:
        frame_gt: A `sio.LabeledFrame` with ground truth instances.
        frame_pr: A `sio.LabeledFrame` with predicted instances.
        stddev: The expected spread of coordinates for OKS computation.
        scale: The scale for normalizing the OKS. If not set, the bounding box area will
            be used.
        threshold: The minimum OKS between a candidate pair of instances to be
            considered a match.

    Returns:
        A tuple of (`positive_pairs`, `false_negatives`).

        `positive_pairs` is a list of 3-tuples of the form
        `(instance_gt, instance_pr, oks)` containing the matched pair of instances and
        their OKS.

        `false_negatives` is a list of ground truth `sleap.Instance`s that could not be
        matched.

    Notes:
        This function uses the approach from the PASCAL VOC scoring procedure. Briefly,
        predictions are sorted descending by their instance-level prediction scores and
        greedily matched to ground truth instances which are then removed from the pool
        of available instances.

        Ground truth instances that remain unmatched are considered false negatives.
    """
    # Sort predicted instances by score.
    frame_pr_match_instances = get_instances(frame_pr)

    scores_pr = np.array(
        [
            m.instance.score
            for m in frame_pr_match_instances
            if hasattr(m.instance, "score")
        ]
    )
    idxs_pr = np.argsort(-scores_pr, kind="mergesort")  # descending
    scores_pr = scores_pr[idxs_pr]

    available_instances_gt = get_instances(frame_gt)
    available_instances_gt_idxs = list(range(len(available_instances_gt)))

    positive_pairs = []
    for idx_pr in idxs_pr:
        # Pull out predicted instance.
        instance_pr = frame_pr_match_instances[idx_pr]

        # Convert instances to point arrays.
        points_pr = np.expand_dims(instance_pr.instance.numpy(), axis=0)
        points_gt = np.stack(
            [
                available_instances_gt[idx].instance.numpy()
                for idx in available_instances_gt_idxs
            ],
            axis=0,
        )

        # Find the best match by computing OKS.
        oks = compute_oks(points_gt, points_pr, stddev=stddev, scale=scale)
        oks = np.squeeze(oks, axis=1)
        assert oks.shape == (len(points_gt),)

        oks[oks <= threshold] = np.nan
        best_match_gt_idx = np.argsort(-oks, kind="mergesort")[0]
        best_match_oks = oks[best_match_gt_idx]
        if np.isnan(best_match_oks):
            continue

        # Remove matched ground truth instance and add as a positive pair.
        instance_gt_idx = available_instances_gt_idxs.pop(best_match_gt_idx)
        instance_gt = available_instances_gt[instance_gt_idx]
        positive_pairs.append((instance_gt, instance_pr, best_match_oks))

        # Stop matching lower scoring instances if we run out of candidates in the
        # ground truth.
        if not available_instances_gt_idxs:
            break

    # Any remaining ground truth instances are considered false negatives.
    false_negatives = [
        available_instances_gt[idx] for idx in available_instances_gt_idxs
    ]

    return positive_pairs, false_negatives

match_masks(pred_masks, gt_masks, min_iou=0.5)

Match predicted masks to ground-truth masks by IoU (Hungarian).

Parameters:

Name	Type	Description	Default
pred_masks	List[ndarray]	List of boolean arrays, one per predicted instance.	required
gt_masks	List[ndarray]	List of boolean arrays, one per ground-truth instance.	required
min_iou	float	Minimum IoU for a matched pair to count as a true positive.	0.5

Returns:

Type	Description
tuple	Tuple of: - matched_pred_indices: Indices of matched predictions. - matched_gt_indices: Indices of matched ground truth. - unmatched_pred_indices: Unmatched predictions (false positives). - unmatched_gt_indices: Unmatched ground truth (false negatives). - matched_ious: IoU of each matched pair, aligned to matched_pred_indices.

Source code in sleap_nn/evaluation.py

def match_masks(
    pred_masks: List[np.ndarray],
    gt_masks: List[np.ndarray],
    min_iou: float = 0.5,
) -> tuple:
    """Match predicted masks to ground-truth masks by IoU (Hungarian).

    Args:
        pred_masks: List of boolean arrays, one per predicted instance.
        gt_masks: List of boolean arrays, one per ground-truth instance.
        min_iou: Minimum IoU for a matched pair to count as a true positive.

    Returns:
        Tuple of:
            - matched_pred_indices: Indices of matched predictions.
            - matched_gt_indices: Indices of matched ground truth.
            - unmatched_pred_indices: Unmatched predictions (false positives).
            - unmatched_gt_indices: Unmatched ground truth (false negatives).
            - matched_ious: IoU of each matched pair, aligned to
              ``matched_pred_indices``.
    """
    from scipy.optimize import linear_sum_assignment

    n_pred = len(pred_masks)
    n_gt = len(gt_masks)
    empty = np.array([], dtype=int)
    if n_pred == 0 and n_gt == 0:
        return empty, empty, empty, empty, np.array([])
    if n_pred == 0:
        return empty, empty, empty, np.arange(n_gt), np.array([])
    if n_gt == 0:
        return empty, empty, np.arange(n_pred), empty, np.array([])

    iou = _mask_iou_matrix(pred_masks, gt_masks)  # (n_pred, n_gt)
    # Maximize total IoU -> minimize negative IoU.
    pred_indices, gt_indices = linear_sum_assignment(-iou)

    matched_pred, matched_gt, matched_ious = [], [], []
    for p_idx, g_idx in zip(pred_indices, gt_indices):
        if iou[p_idx, g_idx] >= min_iou:
            matched_pred.append(int(p_idx))
            matched_gt.append(int(g_idx))
            matched_ious.append(float(iou[p_idx, g_idx]))

    matched_pred = np.array(matched_pred, dtype=int)
    matched_gt = np.array(matched_gt, dtype=int)
    unmatched_pred = np.array(
        sorted(set(range(n_pred)) - set(matched_pred.tolist())), dtype=int
    )
    unmatched_gt = np.array(
        sorted(set(range(n_gt)) - set(matched_gt.tolist())), dtype=int
    )
    return (
        matched_pred,
        matched_gt,
        unmatched_pred,
        unmatched_gt,
        np.array(matched_ious),
    )

run_evaluation(ground_truth_path, predicted_path, oks_stddev=0.025, oks_scale=None, match_threshold=0, user_labels_only=True, save_metrics=None, match_method='oks', anchor_part=None)

Evaluate SLEAP-NN model predictions against ground truth labels.

Parameters:

Name	Type	Description	Default
ground_truth_path	str	Path to the ground-truth .slp file.	required
predicted_path	str	Path to the predicted .slp file.	required
oks_stddev	float	OKS standard deviation (OKS mode only).	0.025
oks_scale	Optional[float]	OKS scale override (OKS mode only).	None
match_threshold	float	Matching threshold. OKS threshold for OKS mode; PIXEL distance for centroid mode. In centroid mode, if the caller leaves the OKS default of 0.0 it is bumped to 50.0 px.	0
user_labels_only	bool	If False, predicted instances in the GT frame may be matched. For match_method="mask" (default True), this additionally drops masks linked to a PredictedInstance from the ground-truth labels, so stray predicted instances (each of which gets a mask when masks are built per-instance from poses) are not treated as ground truth. Pass False when the GT is intentionally built from predicted poses (pseudo-mask GT). The whole-frame match_method="semantic" union is unaffected either way.	True
save_metrics	Optional[str]	Optional .npz path to save metrics to.	None
match_method	str	"oks", "centroid", "mask", "semantic", or "auto". "mask" matches predicted vs GT segmentation masks by IoU (for bottomup_segmentation models). "semantic" unions each frame's masks into one foreground and scores IoU/clDice/boundary-IoU with NO matching (for whole-frame semantic_segmentation models). "auto" switches to centroid mode when the PREDICTION skeleton is a single-node skeleton (e.g. sio.get_centroid_skeleton()); it never auto-selects "mask" or "semantic" (pass those explicitly).	'oks'
anchor_part	Optional[str]	Name of the GT skeleton node used to compute GT centroids (centroid mode). Resolved against the GT skeleton; None (or an absent name) falls back to the mean of visible nodes (#586).	None

Returns:

Type	Description
	The metrics dict, or None if the predicted labels have zero frames or contain nothing usable (no instances for "oks"/ "centroid"/"auto", no masks for "mask"/"semantic") -- metric computation is skipped entirely in that case, and no save_metrics file is written.

Source code in sleap_nn/evaluation.py

def run_evaluation(
    ground_truth_path: str,
    predicted_path: str,
    oks_stddev: float = 0.025,
    oks_scale: Optional[float] = None,
    match_threshold: float = 0,
    user_labels_only: bool = True,
    save_metrics: Optional[str] = None,
    match_method: str = "oks",
    anchor_part: Optional[str] = None,
):
    """Evaluate SLEAP-NN model predictions against ground truth labels.

    Args:
        ground_truth_path: Path to the ground-truth ``.slp`` file.
        predicted_path: Path to the predicted ``.slp`` file.
        oks_stddev: OKS standard deviation (OKS mode only).
        oks_scale: OKS scale override (OKS mode only).
        match_threshold: Matching threshold. OKS threshold for OKS mode; PIXEL
            distance for centroid mode. In centroid mode, if the caller leaves
            the OKS default of ``0.0`` it is bumped to ``50.0`` px.
        user_labels_only: If False, predicted instances in the GT frame may be
            matched. For ``match_method="mask"`` (default True), this additionally
            drops masks linked to a ``PredictedInstance`` from the ground-truth
            labels, so stray predicted instances (each of which gets a mask when
            masks are built per-instance from poses) are not treated as ground
            truth. Pass ``False`` when the GT is intentionally built from predicted
            poses (pseudo-mask GT). The whole-frame ``match_method="semantic"`` union is
            unaffected either way.
        save_metrics: Optional ``.npz`` path to save metrics to.
        match_method: ``"oks"``, ``"centroid"``, ``"mask"``, ``"semantic"``, or
            ``"auto"``. ``"mask"`` matches predicted vs GT segmentation masks by
            IoU (for ``bottomup_segmentation`` models). ``"semantic"`` unions each
            frame's masks into one foreground and scores IoU/clDice/boundary-IoU
            with NO matching (for whole-frame ``semantic_segmentation`` models).
            ``"auto"`` switches to centroid mode when the PREDICTION skeleton is a
            single-node skeleton (e.g. ``sio.get_centroid_skeleton()``); it never
            auto-selects ``"mask"`` or ``"semantic"`` (pass those explicitly).
        anchor_part: Name of the GT skeleton node used to compute GT centroids
            (centroid mode). Resolved against the GT skeleton; ``None`` (or an
            absent name) falls back to the mean of visible nodes (#586).

    Returns:
        The metrics dict, or ``None`` if the predicted labels have zero
        frames or contain nothing usable (no instances for ``"oks"``/
        ``"centroid"``/``"auto"``, no masks for ``"mask"``/``"semantic"``) --
        metric computation is skipped entirely in that case, and no
        ``save_metrics`` file is written.
    """
    logger.info("Loading ground truth labels...")
    ground_truth_instances = sio.load_slp(ground_truth_path)
    logger.info(
        f"  Ground truth: {len(ground_truth_instances.videos)} videos, "
        f"{len(ground_truth_instances.labeled_frames)} frames"
    )

    logger.info("Loading predicted labels...")
    predicted_instances = sio.load_slp(predicted_path)
    logger.info(
        f"  Predictions: {len(predicted_instances.videos)} videos, "
        f"{len(predicted_instances.labeled_frames)} frames"
    )

    # Detect a fully collapsed prediction set up front and skip the metric
    # math entirely (#719) -- frames may still be present (both predictor
    # pipelines retain empty-detection frames by default), but nothing usable
    # was predicted in any of them, so matching would only produce an
    # all-NaN/all-zero result. ``mask``/``semantic`` predictions live on
    # ``LabeledFrame.masks``, not ``.instances``.
    if match_method in ("mask", "semantic"):
        has_predictions = any(len(lf.masks) for lf in predicted_instances)
    else:
        has_predictions = any(len(lf.instances) for lf in predicted_instances)
    if not len(predicted_instances) or not has_predictions:
        logger.info(
            "0 predicted instances: skipping metric computation (model "
            "likely predicted nothing usable, or training collapsed)."
        )
        return None

    # Auto-detect centroid mode from the PREDICTION skeleton.
    pred_skeleton = (
        predicted_instances.skeletons[0] if predicted_instances.skeletons else None
    )
    if match_method == "auto":
        if _is_single_node_skeleton(pred_skeleton):
            match_method = "centroid"
            logger.info(
                "Auto-detected centroid mode (single-node prediction skeleton)."
            )
        else:
            match_method = "oks"

    # Resolve the anchor node against the GT skeleton (mirror predictor.py).
    gt_skeleton = (
        ground_truth_instances.skeletons[0]
        if ground_truth_instances.skeletons
        else None
    )
    anchor_ind = _resolve_anchor_ind(gt_skeleton, anchor_part)

    # In centroid mode, default the (pixel) match threshold to 50.0 if the
    # caller left the OKS default of 0.0.
    if match_method == "centroid" and match_threshold == 0:
        match_threshold = 50.0

    # In mask mode, default the IoU match threshold to 0.5 if the caller left
    # the OKS default of 0.0.
    if match_method == "mask" and match_threshold == 0:
        match_threshold = 0.5

    # Mask eval matches GT vs predicted MASKS (on ``frame.masks``), independent of
    # whether the frame's keypoint instances are user- or predicted-labeled. The
    # ``user_labels_only`` frame filter (find_frame_pairs) keeps only frames with
    # USER keypoint instances, which silently drops EVERY frame when the GT was
    # built from predicted poses (e.g. pseudo-mask GT from predicted skeletons),
    # raising "Empty Frame Pairs". Mask mode therefore never applies that FRAME
    # filter. The caller's ``user_labels_only`` intent is preserved separately to
    # govern the ground-truth MASK filter: a labels file with stray
    # PredictedInstances gives each a mask (masks are built per-instance), and under
    # user-only labels those must not be treated as ground truth (they would be
    # spurious false negatives that cap recall). Callers evaluating pseudo-mask GT
    # pass ``user_labels_only=False``.
    exclude_predicted_instance_masks = user_labels_only
    if match_method in ("mask", "semantic"):
        user_labels_only = False

    logger.info("Matching videos and frames...")
    # Get match stats before creating evaluator
    match_result = ground_truth_instances.match(predicted_instances)
    logger.info(
        f"  Videos matched: {match_result.n_videos_matched}/{len(match_result.video_map)}"
    )

    logger.info("Matching instances...")
    evaluator = Evaluator(
        ground_truth_instances=ground_truth_instances,
        predicted_instances=predicted_instances,
        oks_stddev=oks_stddev,
        oks_scale=oks_scale,
        match_threshold=match_threshold,
        user_labels_only=user_labels_only,
        match_method=match_method,
        anchor_ind=anchor_ind,
        exclude_predicted_instance_masks=exclude_predicted_instance_masks,
    )
    logger.info(
        f"  Frame pairs: {len(evaluator.frame_pairs)}, "
        f"Matched instances: {len(evaluator.positive_pairs)}, "
        f"Unmatched GT: {len(evaluator.false_negatives)}"
    )

    logger.info("Computing evaluation metrics...")
    metrics = evaluator.evaluate()

    if match_method == "centroid":
        # Centroid mode: report detection + distance metrics only (no
        # oks_voc.*/mOKS/PCK/visibility keys exist).
        det = metrics["detection_metrics"]
        dist = metrics["distance_metrics"]
        logger.info("Evaluation Results (centroid mode):")
        logger.info(f"  Precision: {det['precision']:.4f}")
        logger.info(f"  Recall: {det['recall']:.4f}")
        logger.info(f"  F1: {det['f1']:.4f}")
        logger.info(f"  Counts: TP={det['n_tp']}, FP={det['n_fp']}, FN={det['n_fn']}")
        logger.info(f"  Average Distance: {dist['avg']:.2f} px")
        logger.info(f"  dist.p50: {dist['p50']:.2f} px")
        logger.info(f"  dist.p90: {dist['p90']:.2f} px")
        logger.info(f"  dist.p95: {dist['p95']:.2f} px")
        logger.info(f"  dist.p99: {dist['p99']:.2f} px")

        if save_metrics:
            logger.info(f"Saving metrics to {save_metrics}...")
            save_path = Path(save_metrics)
            # Writes the pickled ``.npz`` (back-compat) plus a JSON sibling
            # with the same stem so the app can read metrics without unpickling.
            _write_metrics(save_path, metrics)
            logger.info(f"Metrics saved successfully to {save_path}")

        return metrics

    if match_method == "mask":
        # Mask mode: report detection (IoU-matched) + mask-IoU quality + COCO
        # mask AP/AR (no oks_voc.*/mOKS/PCK/visibility keys exist).
        det = metrics["detection_metrics"]
        mm = metrics["mask_metrics"]
        mvoc = metrics["mask_voc_metrics"]
        logger.info("Evaluation Results (mask mode):")
        logger.info(f"  Precision: {det['precision']:.4f}")
        logger.info(f"  Recall: {det['recall']:.4f}")
        logger.info(f"  F1: {det['f1']:.4f}")
        logger.info(f"  Counts: TP={det['n_tp']}, FP={det['n_fp']}, FN={det['n_fn']}")
        logger.info(f"  Mean mask IoU: {mm['mean_iou']:.4f}")
        logger.info(f"  mask IoU p50: {mm['p50']:.4f}")
        logger.info(f"  mask IoU p25: {mm['p25']:.4f}")
        logger.info(f"  Mean boundary IoU: {mm['mean_boundary_iou']:.4f}")
        logger.info(f"  Mean clDice (centerline): {mm['mean_cldice']:.4f}")
        logger.info(f"  mAP @[.5:.95]: {mvoc['mask_voc.mAP']:.4f}")
        logger.info(
            f"  AP50: {mvoc['mask_voc.AP50']:.4f}  AP75: {mvoc['mask_voc.AP75']:.4f}"
        )
        logger.info(f"  AR @[.5:.95]: {mvoc['mask_voc.AR']:.4f}")
        e0, e1 = mvoc["mask_voc.size_edges"]
        logger.info(
            f"  AP by size [percentile, edges={e0:.0f}/{e1:.0f} px^2]: "
            f"S={mvoc['mask_voc.AP_small']:.4f} "
            f"M={mvoc['mask_voc.AP_medium']:.4f} L={mvoc['mask_voc.AP_large']:.4f} "
            f"(GT S/M/L={mvoc['mask_voc.n_gt_small']}/"
            f"{mvoc['mask_voc.n_gt_medium']}/{mvoc['mask_voc.n_gt_large']})"
        )
        logger.info(
            f"  AP by size [COCO 1024/9216 px^2]: "
            f"S={mvoc['mask_voc.coco.AP_small']:.4f} "
            f"M={mvoc['mask_voc.coco.AP_medium']:.4f} "
            f"L={mvoc['mask_voc.coco.AP_large']:.4f} "
            f"(GT S/M/L={mvoc['mask_voc.coco.n_gt_small']}/"
            f"{mvoc['mask_voc.coco.n_gt_medium']}/{mvoc['mask_voc.coco.n_gt_large']})"
        )
        logger.info(
            f"  Fragmentation: oversegmentation={mm['oversegmentation']}, "
            f"undersegmentation={mm['undersegmentation']}"
        )

        if save_metrics:
            logger.info(f"Saving metrics to {save_metrics}...")
            save_path = Path(save_metrics)
            # Writes the pickled ``.npz`` (back-compat) plus a JSON sibling
            # with the same stem so the app can read metrics without unpickling.
            _write_metrics(save_path, metrics)
            logger.info(f"Metrics saved successfully to {save_path}")

        return metrics

    if match_method == "semantic":
        # Semantic (whole-frame foreground) mode: matching-free IoU / clDice /
        # boundary-IoU only (no detection / mask-AP keys exist).
        sm = metrics["semantic_metrics"]
        logger.info("Evaluation Results (semantic / whole-frame foreground mode):")
        logger.info(f"  Frames scored (non-empty GT fg): {sm['n_frames']}")
        logger.info(f"  Mean foreground IoU: {sm['mean_iou']:.4f}")
        logger.info(f"  Mean clDice (centerline): {sm['mean_cldice']:.4f}")
        logger.info(f"  Mean boundary IoU: {sm['mean_boundary_iou']:.4f}")

        if save_metrics:
            logger.info(f"Saving metrics to {save_metrics}...")
            save_path = Path(save_metrics)
            # Writes the pickled ``.npz`` (back-compat) plus a JSON sibling
            # with the same stem so the app can read metrics without unpickling.
            _write_metrics(save_path, metrics)
            logger.info(f"Metrics saved successfully to {save_path}")

        return metrics

    # Compute PCK at specific thresholds (5 and 10 pixels)
    dists = metrics["distance_metrics"]["dists"]
    dists_clean = np.copy(dists)
    dists_clean[np.isnan(dists_clean)] = np.inf
    # Guard the empty-match case (0 matched instances for the whole split) so
    # this doesn't hit "Mean of empty slice" on top of the evaluate()-level
    # log line already emitted for it.
    pck_5 = float((dists_clean < 5).mean()) if dists_clean.size else np.nan
    pck_10 = float((dists_clean < 10).mean()) if dists_clean.size else np.nan

    # Print key metrics
    logger.info("Evaluation Results:")
    logger.info(f"  mOKS: {metrics['mOKS']['mOKS']:.4f}")
    logger.info(f"  mAP (OKS VOC): {metrics['voc_metrics']['oks_voc.mAP']:.4f}")
    logger.info(f"  mAR (OKS VOC): {metrics['voc_metrics']['oks_voc.mAR']:.4f}")
    logger.info(f"  Average Distance: {metrics['distance_metrics']['avg']:.2f} px")
    logger.info(f"  dist.p50: {metrics['distance_metrics']['p50']:.2f} px")
    logger.info(f"  dist.p95: {metrics['distance_metrics']['p95']:.2f} px")
    logger.info(f"  dist.p99: {metrics['distance_metrics']['p99']:.2f} px")
    logger.info(f"  mPCK: {metrics['pck_metrics']['mPCK']:.4f}")
    logger.info(f"  PCK@5px: {pck_5:.4f}")
    logger.info(f"  PCK@10px: {pck_10:.4f}")
    logger.info(
        f"  Visibility Precision: {metrics['visibility_metrics']['precision']:.4f}"
    )
    logger.info(f"  Visibility Recall: {metrics['visibility_metrics']['recall']:.4f}")

    # Save metrics if path provided
    if save_metrics:
        logger.info(f"Saving metrics to {save_metrics}...")
        save_path = Path(save_metrics)

        # Save metrics in SLEAP 1.4 format (single "metrics" key) plus a JSON
        # sibling (same stem) that the app metrics UI can read without
        # unpickling the numpy object array.
        _write_metrics(save_path, metrics)
        logger.info(f"Metrics saved successfully to {save_path}")

    return metrics