evaluation

sleap_nn.evaluation

This module is to compute evaluation metrics for trained models.

Classes:

Name	Description
Evaluator	Compute the standard evaluation metrics with the predicted and the ground-truth Labels.
IdentityMetrics	Identity-persistence metrics for one tracked prediction.
MOTMetrics	TrackEval-faithful CLEAR-MOT, Identity and HOTA metrics for one prediction.
MatchInstance	Class to have a new structure for sio.Instance object.
NothingToScoreError	Raised by run_identity_evaluation(raise_if_empty=True) when nothing is scoreable.

Functions:

Name	Description
bbox_frame_pairs	Build :func:compute_bbox_metrics frames from GT and predicted labels.
bbox_iou_matrix	Pairwise IoU of (n, 4) and (m, 4) xyxy boxes -> (n, m).
compare_identity_metrics	Render a Markdown comparison table across tracker arms.
compute_bbox_metrics	COCO-style detection metrics over a set of frames.
compute_distance_match_score	Compute a pixel-distance-based match score for degenerate-scale GT instances.
compute_dists	Compute Euclidean distances between matched pairs of instances.
compute_gt_centroids	Compute ground-truth centroids for a numpy array of instance keypoints.
compute_instance_area	Compute the area of the bounding box of a set of keypoints.
compute_mot_metrics	Score a tracked prediction with TrackEval-faithful CLEAR-MOT, IDF1 and HOTA.
compute_oks	Compute the object keypoints similarity between sets of points.
embedding_full_eval	Combined retrieval + verification + kNN-accuracy metrics dict.
embedding_leave_self_out_eval	Leave-self-out retrieval/verification/kNN over one labeled embedding set.
find_frame_pairs	Find corresponding frames across two sets of labels.
get_instances	Get a list of instances of type MatchInstance from the Labeled Frame.
identity_metrics	Score a tracked prediction against tracked ground truth.
knn_classify	Cosine k-NN classification (see :func:_knn_vote). Returns (pred, conf).
load_metrics	Load metrics from a model folder or metrics file.
mask_cldice	Centerline Dice (clDice) between two binary masks.
match_centroids	Match predicted centroids to ground truth using Hungarian algorithm.
match_frame_pairs	Match all ground truth and predicted instances within each pair of frames.
match_instances	Match pairs of instances between ground truth and predictions in a frame.
match_masks	Match predicted masks to ground-truth masks by IoU (Hungarian).
motion_diagnostic	Judge whether a labels file is continuous video or temporally sparse samples.
retrieval_metrics	Rank-1 (CMC@1) + mAP of queries against a gallery (cosine similarity).
run_evaluation	Evaluate SLEAP-NN model predictions against ground truth labels.
run_identity_evaluation	Evaluate identity persistence of a tracked prediction against tracked GT.
verification_metrics	ROC-AUC + EER over all query x gallery pairs (same vs different identity).

Evaluator

Compute the standard evaluation metrics with the predicted and the ground-truth Labels.

This class is used to calculate the common metrics for pose estimation models which includes voc metrics (with oks and pck), mOKS, distance metrics, pck metrics and visibility metrics.

Parameters:

Name	Type	Description	Default
ground_truth_instances	Labels	The sio.Labels dataset object with ground truth labels.	required
predicted_instances	Labels	The sio.Labels dataset object with predicted labels.	required
oks_stddev	float	The standard deviation to use for calculating object keypoint similarity; see compute_oks function for details.	0.025
oks_scale	Optional[float]	The scale to use for calculating object keypoint similarity; see compute_oks function for details.	None
match_threshold	float	The threshold to use when determining which instances match between ground truth and predicted frames. For match_method="oks" this is an OKS threshold; for match_method="centroid" this is a PIXEL distance threshold.	0
user_labels_only	bool	If False, predicted instances in the ground truth frame may be considered for matching.	True
match_method	str	Either "oks" (default, full-skeleton OKS matching) or "centroid" (single-point distance matching for centroid-only / single-node predictions).	'oks'
anchor_ind	Optional[int]	For match_method="centroid", the index of the GT skeleton node used to compute each ground-truth centroid (see :func:compute_gt_centroids and #586). None falls back to the NaN-ignoring mean of visible nodes.	None
centroid_method	Optional[str]	For match_method="centroid", how the GT centroid is derived -- "center_of_mass", "bbox_center", "geometric_median" or "anchor". None (default) infers it from anchor_ind. Must match what the model was trained on, or the distance metric compares two different definitions of "centroid"; :func:run_evaluation reads it off the training config.	None
centroid_fallback	Optional[str]	Reduce method used when the anchor node is not visible.	None

Methods:

Name	Description
__init__	Initialize the Evaluator class with ground-truth and predicted labels.
detection_metrics	Compute detection metrics (precision/recall/F1) over TP/FP/FN counts.
distance_metrics	Compute the Euclidean distance error at different percentiles using the pairwise distances.
evaluate	Return the evaluation metrics.
mOKS	Return the meanOKS value.
mask_metrics	Compute mask-IoU summary statistics for match_method="mask".
mask_voc_metrics	COCO-style score-ranked mask Average Precision / Recall.
pck_metrics	Compute PCK across a range of thresholds using the pair-wise distances.
semantic_metrics	Aggregate whole-frame foreground metrics for match_method="semantic".
visibility_metrics	Compute node visibility metrics for the matched pair of instances.
voc_metrics	Compute VOC metrics for a matched pairs of instances positive pairs and false negatives.

Source code in sleap_nn/evaluation.py

class Evaluator:
    """Compute the standard evaluation metrics with the predicted and the ground-truth Labels.

    This class is used to calculate the common metrics for pose estimation models which
    includes voc metrics (with oks and pck), mOKS, distance metrics, pck metrics and
    visibility metrics.

    Args:
        ground_truth_instances: The `sio.Labels` dataset object with ground truth labels.
        predicted_instances: The `sio.Labels` dataset object with predicted labels.
        oks_stddev: The standard deviation to use for calculating object
            keypoint similarity; see `compute_oks` function for details.
        oks_scale: The scale to use for calculating object
            keypoint similarity; see `compute_oks` function for details.
        match_threshold: The threshold to use when determining which instances
            match between ground truth and predicted frames. For
            ``match_method="oks"`` this is an OKS threshold; for
            ``match_method="centroid"`` this is a PIXEL distance threshold.
        user_labels_only: If False, predicted instances in the ground truth frame may be
            considered for matching.
        match_method: Either ``"oks"`` (default, full-skeleton OKS matching) or
            ``"centroid"`` (single-point distance matching for centroid-only /
            single-node predictions).
        anchor_ind: For ``match_method="centroid"``, the index of the GT
            skeleton node used to compute each ground-truth centroid (see
            :func:`compute_gt_centroids` and #586). ``None`` falls back to the
            NaN-ignoring mean of visible nodes.
        centroid_method: For ``match_method="centroid"``, how the GT centroid is
            derived -- ``"center_of_mass"``, ``"bbox_center"``,
            ``"geometric_median"`` or ``"anchor"``. ``None`` (default) infers it
            from ``anchor_ind``. Must match what the model was trained on, or the
            distance metric compares two different definitions of "centroid";
            :func:`run_evaluation` reads it off the training config.
        centroid_fallback: Reduce method used when the anchor node is not visible.

    """

    def __init__(
        self,
        ground_truth_instances: sio.Labels,
        predicted_instances: sio.Labels,
        oks_stddev: float = 0.025,
        oks_scale: Optional[float] = None,
        match_threshold: float = 0,
        user_labels_only: bool = True,
        match_method: str = "oks",
        anchor_ind: Optional[int] = None,
        centroid_method: Optional[str] = None,
        centroid_fallback: Optional[str] = None,
        exclude_predicted_instance_masks: bool = False,
    ):
        """Initialize the Evaluator class with ground-truth and predicted labels.

        ``exclude_predicted_instance_masks`` (``match_method="mask"`` only) drops
        masks linked to a ``PredictedInstance`` from the ground-truth labels, so a
        labels file that carries stray predicted instances (each of which gets a
        mask when masks are built from poses) does not treat them as ground truth.
        It is kept separate from ``user_labels_only`` (which controls the frame-pair
        filter) because mask mode disables that frame filter -- see
        :func:`run_evaluation`.
        """
        self.ground_truth_instances = ground_truth_instances
        self.predicted_instances = predicted_instances
        self.match_threshold = match_threshold
        self.oks_stddev = oks_stddev
        self.oks_scale = oks_scale
        self.user_labels_only = user_labels_only
        self.match_method = match_method
        self.anchor_ind = anchor_ind
        self.centroid_method = centroid_method
        self.centroid_fallback = centroid_fallback
        self.exclude_predicted_instance_masks = exclude_predicted_instance_masks
        # Populated only in centroid / mask mode.
        self.false_positives = []
        # Matched-pair IoUs, populated only in mask mode.
        self.mask_ious = np.array([])
        # Per-frame mask records + matched TP mask pairs, populated only in mask
        # mode (feed mask_voc_metrics / boundary-IoU / fragmentation / per-size).
        self._mask_frames = []
        self._matched_mask_pairs = []
        # Per-frame (iou, cldice, boundary_iou) triples, populated only in
        # match_method="semantic" (whole-frame foreground, no matching).
        self._semantic_rows = []

        self._process_frames()

    def _process_frames(self):
        self.frame_pairs = find_frame_pairs(
            self.ground_truth_instances,
            self.predicted_instances,
            self.user_labels_only,
            keep_user_centroid_frames=self.match_method == "centroid",
        )
        if not self.frame_pairs:
            message = "Empty Frame Pairs. No match found for the video frames"
            logger.error(message)
            raise Exception(message)

        if self.match_method == "centroid":
            self._process_frames_centroid()
            return

        if self.match_method == "mask":
            self._process_frames_mask()
            return

        if self.match_method == "semantic":
            self._process_frames_semantic()
            return

        self.positive_pairs, self.false_negatives = match_frame_pairs(
            self.frame_pairs,
            stddev=self.oks_stddev,
            scale=self.oks_scale,
            threshold=self.match_threshold,
        )

        self.dists_dict = compute_dists(self.positive_pairs)

    def _process_frames_centroid(self):
        """Match predicted vs GT centroids by pixel distance (per frame).

        Each predicted instance is collapsed to its single centroid point (its
        sole visible point / node-0 for a 1-node prediction). Ground-truth
        centroids are computed via :func:`compute_gt_centroids` to exactly
        mirror the centroid target used during training (#586). Matching uses
        :func:`match_centroids` with ``self.match_threshold`` as a PIXEL
        distance. Populates ``positive_pairs`` as ``(gt_inst, pr_inst, dist)``
        3-tuples, ``false_negatives`` (unmatched GT), and ``false_positives``
        (unmatched predictions).
        """
        self.positive_pairs = []
        self.false_negatives = []
        self.false_positives = []

        for frame_gt, frame_pr in self.frame_pairs:
            # A mask-only or centroid-annotation-only ground-truth frame has no
            # instances; its centroids ARE the ground truth (#586).
            if not get_instances(frame_gt) and _user_centroids(frame_gt):
                frame_gt = attrs.evolve(
                    frame_gt, instances=_instances_from_user_centroids(frame_gt)
                )
                gt_from_centroid_annotations = True
            else:
                gt_from_centroid_annotations = False
            gt_match_instances = get_instances(frame_gt)
            pr_match_instances = get_instances(frame_pr)

            # Collapse each predicted instance to its single centroid point.
            pred_centroids = np.array(
                [
                    self._collapse_pred_centroid(m.instance.numpy())
                    for m in pr_match_instances
                ]
            ).reshape(-1, 2)

            # GT centroids come from generate_centroids itself (#586) -- except
            # when they came from `Centroid` annotations, which are already the
            # centroid: the wrapper is one node, so `anchor_ind` (an index into
            # the POSE skeleton) does not apply to it.
            gt_centroids = np.array(
                [
                    compute_gt_centroids(
                        m.instance.numpy(),
                        None if gt_from_centroid_annotations else self.anchor_ind,
                        method=(
                            None
                            if gt_from_centroid_annotations
                            else self.centroid_method
                        ),
                        fallback=(
                            None
                            if gt_from_centroid_annotations
                            else self.centroid_fallback
                        ),
                    )
                    for m in gt_match_instances
                ]
            ).reshape(-1, 2)

            # Drop NaN centroids before Hungarian matching: scipy's cdist /
            # linear_sum_assignment reject NaN, and a fully-occluded (all-NaN)
            # GT instance is common in real labels. Index maps translate the
            # filtered match indices back to the original instance lists so
            # FN/FP/positive-pair attribution stays correct. (A NaN-row GT is
            # counted as an automatic false negative — matching the legacy
            # CentroidEvaluationCallback; a NaN-row prediction is not a real
            # detection and is simply excluded.)
            gt_valid = ~np.isnan(gt_centroids).any(axis=1)
            pred_valid = ~np.isnan(pred_centroids).any(axis=1)
            gt_map = np.flatnonzero(gt_valid)
            pred_map = np.flatnonzero(pred_valid)

            matched_pred, matched_gt, unmatched_pred, unmatched_gt = match_centroids(
                pred_centroids[pred_valid],
                gt_centroids[gt_valid],
                max_distance=self.match_threshold,
            )

            for p_local, g_local in zip(matched_pred, matched_gt):
                p_idx = int(pred_map[int(p_local)])
                g_idx = int(gt_map[int(g_local)])
                dist = float(
                    np.linalg.norm(pred_centroids[p_idx] - gt_centroids[g_idx])
                )
                self.positive_pairs.append(
                    (gt_match_instances[g_idx], pr_match_instances[p_idx], dist)
                )

            for g_local in unmatched_gt:
                self.false_negatives.append(
                    gt_match_instances[int(gt_map[int(g_local)])]
                )
            # Fully-occluded (all-NaN) GT instances -> automatic false negatives.
            for g_idx in np.flatnonzero(~gt_valid):
                self.false_negatives.append(gt_match_instances[int(g_idx)])

            for p_local in unmatched_pred:
                self.false_positives.append(
                    pr_match_instances[int(pred_map[int(p_local)])]
                )

        # Build the dists dict directly from matched-pair centroid distances so
        # distance_metrics() works uniformly across match methods.
        dists = np.array([dist for _, _, dist in self.positive_pairs])
        self.dists_dict = {
            "dists": dists,
            "frame_idxs": [gt.frame_idx for gt, _, _ in self.positive_pairs],
            "video_paths": [gt.video_path for gt, _, _ in self.positive_pairs],
        }

    def _process_frames_mask(self):
        """Match predicted vs GT segmentation masks by IoU (per frame).

        Pulls per-instance boolean masks from ``LabeledFrame.masks`` on each
        paired frame and matches them with :func:`match_masks` using
        ``self.match_threshold`` as the IoU threshold. Populates
        ``positive_pairs`` as ``(frame_gt, frame_pr, iou)`` 3-tuples (the frame
        objects are stored only as tokens; detection counting uses the list
        lengths, and per-pair IoUs feed :meth:`mask_metrics`), plus
        ``false_negatives`` (unmatched GT masks) and ``false_positives``
        (unmatched predicted masks). No keypoint distances exist for masks, so
        ``dists_dict`` is left empty (``distance_metrics`` reports NaN; IoU is
        reported via :meth:`mask_metrics`).
        """
        self.positive_pairs = []
        self.false_negatives = []
        self.false_positives = []
        ious: List[float] = []
        # Per-frame decoded masks + scores + IoU/intersection matrices, reused by
        # mask_voc_metrics (score-ranked COCO AP) and the fragmentation/per-size
        # breakdowns without re-decoding RLE masks.
        self._mask_frames = []
        # Matched (pred_mask, gt_mask) TP pairs (aligned to ``self.mask_ious``),
        # used for boundary-IoU scoring.
        self._matched_mask_pairs = []

        for frame_gt, frame_pr in self.frame_pairs:
            # Ground-truth masks drop any PredictedInstance-linked masks when the
            # caller asked for user-only labels; predicted-side masks are the
            # model's output and are always kept in full.
            gt_masks = _frame_masks(
                frame_gt,
                drop_predicted_instances=self.exclude_predicted_instance_masks,
            )
            pr_masks = _frame_masks(frame_pr)
            pr_scores = _frame_pred_scores(frame_pr)
            iou_mat, inter_mat = _mask_pair_stats(pr_masks, gt_masks)
            self._mask_frames.append(
                {
                    "pred_masks": pr_masks,
                    "pred_scores": pr_scores,
                    "gt_masks": gt_masks,
                    "iou": iou_mat,
                    "inter": inter_mat,
                    "gt_areas": np.array([int(m.sum()) for m in gt_masks], dtype=float),
                    "pred_areas": np.array(
                        [int(m.sum()) for m in pr_masks], dtype=float
                    ),
                }
            )

            matched_pred, matched_gt, unmatched_pred, unmatched_gt, pair_ious = (
                match_masks(pr_masks, gt_masks, min_iou=self.match_threshold)
            )

            for iou in pair_ious:
                self.positive_pairs.append((frame_gt, frame_pr, float(iou)))
                ious.append(float(iou))
            for p_idx, g_idx in zip(matched_pred, matched_gt):
                self._matched_mask_pairs.append(
                    (pr_masks[int(p_idx)], gt_masks[int(g_idx)])
                )
            for _ in unmatched_gt:
                self.false_negatives.append(frame_gt)
            for _ in unmatched_pred:
                self.false_positives.append(frame_pr)

        self.mask_ious = np.asarray(ious, dtype=float)
        self.dists_dict = {"dists": np.array([]), "frame_idxs": [], "video_paths": []}

    def _process_frames_semantic(self):
        """Whole-frame foreground evaluation (no instance matching).

        For semantic (binary foreground/background) segmentation there is a single
        foreground mask per frame and no instance grouping, so there is nothing to
        match. Each paired frame's predicted and ground-truth masks are unioned
        into one foreground mask (:func:`_union_frame_fg`) and scored directly with
        :func:`_mask_iou`, :func:`mask_cldice`, and :func:`_boundary_iou`. Frames
        whose GROUND-TRUTH foreground is empty are skipped (there is no foreground
        to score).

        Populates ``self._semantic_rows`` as ``(iou, cldice, boundary_iou)``
        triples (consumed by :meth:`semantic_metrics`). The matching-based
        attributes (``positive_pairs`` / ``false_negatives`` / ``false_positives``
        / ``dists_dict``) are left empty so the shared plumbing degrades gracefully
        (semantic mode reports only ``semantic_metrics``).
        """
        self.positive_pairs = []
        self.false_negatives = []
        self.false_positives = []
        self._semantic_rows = []

        for frame_gt, frame_pr in self.frame_pairs:
            gt_fg = _union_frame_fg(frame_gt)
            if not gt_fg.any():
                # No ground-truth foreground: nothing to score on this frame.
                continue
            pr_fg = _union_frame_fg(frame_pr)
            iou = _mask_iou(pr_fg, gt_fg)
            cldice = mask_cldice(pr_fg, gt_fg)
            biou = _boundary_iou(pr_fg, gt_fg)
            self._semantic_rows.append((iou, cldice, biou))

        self.dists_dict = {"dists": np.array([]), "frame_idxs": [], "video_paths": []}

    @staticmethod
    def _collapse_pred_centroid(points: np.ndarray) -> np.ndarray:
        """Collapse a predicted instance to its single centroid point.

        For a 1-node ('centroid') prediction this is node-0. For predictions
        with multiple nodes (e.g. a single-instance model used as a detector)
        we take the single visible point, falling back to node-0.
        """
        points = np.asarray(points, dtype=np.float64).reshape(-1, 2)
        visible = ~np.isnan(points).any(axis=-1)
        if visible.any():
            return points[np.argmax(visible)]
        return points[0]

    def voc_metrics(
        self,
        match_score_by="oks",
        match_score_thresholds: np.ndarray = np.linspace(
            0.5, 0.95, 10
        ),  # 0.5:0.05:0.95
        recall_thresholds: np.ndarray = np.linspace(0, 1, 101),  # 0.0:0.01:1.00
    ):
        """Compute VOC metrics for a matched pairs of instances positive pairs and false negatives.

        Args:
            match_score_by: The score to be used for computing the metrics. "ock" or "pck"
            match_score_thresholds: Score thresholds at which to consider matches as a true
                positive match.
            recall_thresholds: Recall thresholds at which to evaluate Average Precision.

        Returns:
            A dictionary of VOC metrics.
        """
        if match_score_by == "oks":
            match_scores = np.array([oks for _, _, oks in self.positive_pairs])
            name = "oks_voc"
        elif match_score_by == "pck":
            name = "pck_voc"
            if not self.positive_pairs:
                # Guard the empty-match case: the (n_pairs, n_nodes, n_thresholds)
                # ``pcks`` array is empty along the pairs axis, so reducing it with
                # nested .mean() calls would hit "Mean of empty slice".
                match_scores = np.array([])
            else:
                pck_metrics = self.pck_metrics()
                match_scores = pck_metrics["pcks"].mean(axis=-1).mean(axis=-1)
        else:
            message = "Invalid Option for match_score_by. Choose either `oks` or `pck`"
            logger.error(message)
            raise Exception(message)

        detection_scores = np.array(
            [pp[1].instance.score for pp in self.positive_pairs]
        )

        inds = np.argsort(-detection_scores, kind="mergesort")
        detection_scores = detection_scores[inds]
        match_scores = match_scores[inds]

        precisions = []
        recalls = []

        npig = len(self.positive_pairs) + len(
            self.false_negatives
        )  # total number of GT instances

        for match_score_threshold in match_score_thresholds:
            tp = np.cumsum(match_scores >= match_score_threshold)
            fp = np.cumsum(match_scores < match_score_threshold)

            if tp.size == 0:
                return {
                    name + ".match_score_thresholds": 0,
                    name + ".recall_thresholds": 0,
                    name + ".match_scores": 0,
                    name + ".precisions": 0,
                    name + ".recalls": 0,
                    name + ".AP": 0,
                    name + ".AR": 0,
                    name + ".mAP": 0,
                    name + ".mAR": 0,
                }

            rc = tp / npig
            pr = tp / (fp + tp + np.spacing(1))

            recall = rc[-1]  # best recall at this OKS threshold

            # Ensure strictly decreasing precisions.
            for i in range(len(pr) - 1, 0, -1):
                if pr[i] > pr[i - 1]:
                    pr[i - 1] = pr[i]

            # Find best precision at each recall threshold.
            rc_inds = np.searchsorted(rc, recall_thresholds, side="left")
            precision = np.zeros(rc_inds.shape)
            is_valid_rc_ind = rc_inds < len(pr)
            precision[is_valid_rc_ind] = pr[rc_inds[is_valid_rc_ind]]

            precisions.append(precision)
            recalls.append(recall)

        precisions = np.array(precisions)
        recalls = np.array(recalls)

        AP = precisions.mean(
            axis=1
        )  # AP = average precision over fixed set of recall thresholds
        AR = recalls  # AR = max recall given a fixed number of detections per image

        mAP = precisions.mean()  # mAP = mean over all OKS thresholds
        mAR = recalls.mean()  # mAR = mean over all OKS thresholds

        return {
            name + ".match_score_thresholds": match_score_thresholds,
            name + ".recall_thresholds": recall_thresholds,
            name + ".match_scores": match_scores,
            name + ".precisions": precisions,
            name + ".recalls": recalls,
            name + ".AP": AP,
            name + ".AR": AR,
            name + ".mAP": mAP,
            name + ".mAR": mAR,
        }

    def mOKS(self):
        """Return the meanOKS value."""
        pair_oks = np.array([oks for _, _, oks in self.positive_pairs])
        return {"mOKS": float(pair_oks.mean()) if pair_oks.size else np.nan}

    def distance_metrics(self):
        """Compute the Euclidean distance error at different percentiles using the pairwise distances.

        Returns:
            A dictionary of distance metrics.
        """
        dists = self.dists_dict["dists"]
        results = {
            "frame_idxs": self.dists_dict["frame_idxs"],
            "video_paths": self.dists_dict["video_paths"],
            "dists": dists,
            # Guard the empty / all-NaN matched set (zero true positives in a
            # split) so np.nanmean doesn't emit a "Mean of empty slice" warning.
            "avg": (
                float(np.nanmean(dists))
                if np.asarray(dists).size and not np.all(np.isnan(dists))
                else np.nan
            ),
            "p50": np.nan,
            "p75": np.nan,
            "p90": np.nan,
            "p95": np.nan,
            "p99": np.nan,
        }

        is_non_nan = ~np.isnan(dists)
        if np.any(is_non_nan):
            non_nans = dists[is_non_nan]
            for ptile in (50, 75, 90, 95, 99):
                results[f"p{ptile}"] = np.percentile(non_nans, ptile)

        return results

    def detection_metrics(self) -> dict:
        """Compute detection metrics (precision/recall/F1) over TP/FP/FN counts.

        Used by both ``match_method="centroid"`` and ``match_method="mask"``
        (it only reads the matched/unmatched list lengths and ``dists_dict``).
        Mirrors ``CentroidEvaluationCallback._compute_metrics``. For centroid
        mode the localization-error percentiles are computed over the Euclidean
        distances of matched centroid pairs; for mask mode ``dists_dict`` is
        empty so those percentiles are NaN (per-pair IoU is reported separately
        via :meth:`mask_metrics`). Not used for ``match_method="oks"`` (which
        reports OKS-based VOC metrics instead).

        Returns:
            A dict with ``precision``, ``recall``, ``f1``, ``n_tp``, ``n_fp``,
            ``n_fn`` and localization-error percentiles ``avg``/``p50``/``p75``/
            ``p90``/``p95``/``p99`` (NaN when there are no matched pairs).
        """
        n_tp = len(self.positive_pairs)
        n_fp = len(self.false_positives)
        n_fn = len(self.false_negatives)

        precision = n_tp / (n_tp + n_fp) if (n_tp + n_fp) > 0 else 0.0
        recall = n_tp / (n_tp + n_fn) if (n_tp + n_fn) > 0 else 0.0
        f1 = (
            2 * precision * recall / (precision + recall)
            if (precision + recall) > 0
            else 0.0
        )

        dists = self.dists_dict["dists"]
        results = {
            "precision": precision,
            "recall": recall,
            "f1": f1,
            "n_tp": n_tp,
            "n_fp": n_fp,
            "n_fn": n_fn,
            "avg": np.nan,
            "p50": np.nan,
            "p75": np.nan,
            "p90": np.nan,
            "p95": np.nan,
            "p99": np.nan,
        }

        is_non_nan = ~np.isnan(dists) if len(dists) else np.array([], dtype=bool)
        if np.any(is_non_nan):
            non_nans = dists[is_non_nan]
            results["avg"] = float(np.mean(non_nans))
            for ptile in (50, 75, 90, 95, 99):
                results[f"p{ptile}"] = float(np.percentile(non_nans, ptile))

        return results

    def mask_metrics(self) -> dict:
        """Compute mask-IoU summary statistics for ``match_method="mask"``.

        Reports complementary IoU summaries, panoptic-quality, boundary-IoU,
        fragmentation, and per-object-size breakdowns:

        * ``mean_iou`` (and ``min``/``max``/percentiles) over the matched (TP)
          pairs only — COCO-style segmentation quality, blind to misses.
        * ``mean_iou_all_gt`` — IoU averaged over *all* ground-truth masks,
          where an unmatched GT (a miss) contributes ``0``. This penalizes
          recall and complements the TP-only mean.
        * Panoptic Quality ``pq = sq * rq`` with ``sq = mean_iou`` (segmentation
          quality) and ``rq = TP / (TP + 0.5*FP + 0.5*FN)`` (recognition
          quality, == detection F1). See Kirillov et al., "Panoptic
          Segmentation" (2019).
        * ``mean_boundary_iou`` — boundary IoU over the matched pairs (Cheng et
          al., 2021), more sensitive to contour error than mask IoU.
        * ``mean_cldice`` — centerline Dice over the matched pairs (Shit et al.,
          CVPR 2021), connectivity-aware and nearly width-insensitive; a fairer
          score than IoU for thin/tubular structures. NaN if scikit-image is
          unavailable.
        * ``oversegmentation`` / ``undersegmentation`` — fragmentation counts:
          GT masks split across >=2 predictions, and predictions spanning >=2
          GT masks (each with >=10% area overlap). The headline over-/under-
          segmentation failure mode is invisible to the 1-to-1 match.
        * ``per_size`` — COCO small/medium/large breakdown of GT count, TP
          count, and TP-only mean IoU (buckets sum to the GT total).

        Returns:
            A dict with ``mean_iou``, ``min``, ``max``, percentiles ``p25``/
            ``p50``/``p75``, ``mean_iou_all_gt``, ``pq``/``sq``/``rq``,
            ``mean_boundary_iou``, ``oversegmentation``/``undersegmentation``,
            ``per_size``, the TP count ``n_matched`` (plus ``n_fp``/``n_fn``),
            and the raw ``ious`` array. Quantities are NaN when undefined.
        """
        ious = np.asarray(self.mask_ious, dtype=float)
        n_tp = len(self.positive_pairs)
        n_fp = len(self.false_positives)
        n_fn = len(self.false_negatives)
        over, under = self._fragmentation_counts()
        results = {
            "mean_iou": np.nan,
            "min": np.nan,
            "max": np.nan,
            "p25": np.nan,
            "p50": np.nan,
            "p75": np.nan,
            "mean_iou_all_gt": np.nan,
            "pq": np.nan,
            "sq": np.nan,
            "rq": np.nan,
            "mean_boundary_iou": np.nan,
            "mean_cldice": np.nan,
            "oversegmentation": over,
            "undersegmentation": under,
            "per_size": self._mask_per_size_stats(),
            "n_matched": int(ious.size),
            "n_fp": n_fp,
            "n_fn": n_fn,
            "ious": ious,
        }
        if ious.size:
            results["mean_iou"] = float(np.mean(ious))
            results["min"] = float(np.min(ious))
            results["max"] = float(np.max(ious))
            for ptile in (25, 50, 75):
                results[f"p{ptile}"] = float(np.percentile(ious, ptile))

        if self._matched_mask_pairs:
            boundary_ious = np.array(
                [_boundary_iou(p, g) for p, g in self._matched_mask_pairs],
                dtype=float,
            )
            results["mean_boundary_iou"] = float(np.mean(boundary_ious))
            # Centerline Dice (clDice): connectivity/width-tolerant, fairer than
            # IoU for thin structures. NaN entries (scikit-image missing) drop out.
            cldices = np.array(
                [mask_cldice(p, g) for p, g in self._matched_mask_pairs],
                dtype=float,
            )
            cldices = cldices[~np.isnan(cldices)]
            if cldices.size:
                results["mean_cldice"] = float(np.mean(cldices))

        iou_sum = float(np.sum(ious)) if ious.size else 0.0
        # Miss-penalizing mean: averaged over every GT mask (TP + FN).
        n_gt = n_tp + n_fn
        if n_gt > 0:
            results["mean_iou_all_gt"] = iou_sum / n_gt
        # Panoptic quality: SQ = TP-only mean IoU, RQ = detection F1, PQ = SQ*RQ
        # = iou_sum / (TP + 0.5*FP + 0.5*FN).
        pq_denom = n_tp + 0.5 * n_fp + 0.5 * n_fn
        if pq_denom > 0:
            results["sq"] = results["mean_iou"]
            results["rq"] = n_tp / pq_denom
            results["pq"] = iou_sum / pq_denom
        return results

    def semantic_metrics(self) -> dict:
        """Aggregate whole-frame foreground metrics for ``match_method="semantic"``.

        Averages the per-frame foreground IoU, centerline Dice (clDice), and
        boundary IoU computed by :meth:`_process_frames_semantic` over all frames
        with non-empty ground-truth foreground. clDice entries that are NaN
        (scikit-image unavailable) are dropped from the clDice mean; if every entry
        is NaN the reported ``mean_cldice`` is NaN.

        Returns:
            A dict with ``mean_iou``, ``mean_cldice``, ``mean_boundary_iou``, the
            per-frame ``ious`` / ``cldices`` / ``boundary_ious`` arrays, and
            ``n_frames`` (frames scored). Means are NaN when no frame was scored.
        """
        rows = np.asarray(self._semantic_rows, dtype=float).reshape(-1, 3)
        ious = rows[:, 0]
        cldices = rows[:, 1]
        bious = rows[:, 2]
        cld_valid = cldices[~np.isnan(cldices)]
        return {
            "mean_iou": float(np.mean(ious)) if ious.size else float("nan"),
            "mean_cldice": (
                float(np.mean(cld_valid)) if cld_valid.size else float("nan")
            ),
            "mean_boundary_iou": (
                float(np.mean(bious)) if bious.size else float("nan")
            ),
            "ious": ious,
            "cldices": cldices,
            "boundary_ious": bious,
            "n_frames": int(ious.size),
        }

    def _fragmentation_counts(self, overlap_frac: float = 0.1) -> Tuple[int, int]:
        """Count over-/under-segmented instances across all mask frames.

        A prediction "covers" a GT mask when their intersection is at least
        ``overlap_frac`` of the GT area. Over-segmentation counts GT masks
        covered by >=2 predictions (one animal split into fragments);
        under-segmentation counts predictions covering >=2 GT masks (one mask
        merging neighbors). Both directly surface the failure mode the 1-to-1
        Hungarian match hides (extra fragments otherwise just become FPs).
        """
        over = under = 0
        for f in self._mask_frames:
            inter = f["inter"]
            gt_areas = f["gt_areas"]
            n_pred, n_gt = inter.shape
            if n_pred == 0 or n_gt == 0:
                continue
            # Fraction of each GT (cols) covered by each prediction (rows).
            cov_gt = inter / np.maximum(gt_areas[None, :], 1.0)
            covers = cov_gt >= overlap_frac
            over += int(np.count_nonzero(covers.sum(axis=0) >= 2))  # GT split
            under += int(np.count_nonzero(covers.sum(axis=1) >= 2))  # pred merged
        return over, under

    def _per_size_breakdown(
        self,
        gt_areas_all: np.ndarray,
        tp_iou: np.ndarray,
        tp_gt_area: np.ndarray,
        edges: np.ndarray,
    ) -> dict:
        """small/medium/large GT count, TP count and TP mean IoU under ``edges``.

        ``n_gt`` over the three buckets sums to the total GT count (every GT area
        falls in exactly one half-open bucket).
        """
        out = {"edges": [float(e) for e in edges]}
        for idx, bucket in enumerate(_SIZE_KEYS):
            in_gt = _size_mask(gt_areas_all, idx, edges)
            in_tp = (
                _size_mask(tp_gt_area, idx, edges)
                if tp_gt_area.size
                else np.array([], dtype=bool)
            )
            out[bucket] = {
                "n_gt": int(np.count_nonzero(in_gt)),
                "n_tp": int(np.count_nonzero(in_tp)),
                "mean_iou": (
                    float(np.mean(tp_iou[in_tp])) if np.any(in_tp) else np.nan
                ),
            }
        return out

    def _mask_per_size_stats(self) -> dict:
        """Per-object-size GT/TP/IoU breakdown under both bucketing schemes.

        GT objects are bucketed by mask area (``mask.sum()``). The primary
        scheme (top-level ``small``/``medium``/``large`` keys) uses
        dataset-relative percentile edges (terciles by default) so the buckets
        adapt to the actual mask scale; the COCO fixed-cutoff scheme (small <
        32^2 <= medium < 96^2 <= large) is reported additionally under
        ``"coco"`` for cross-dataset comparability.
        """
        gt_areas_all = np.array(
            [a for f in self._mask_frames for a in f["gt_areas"]], dtype=float
        )
        tp_iou = np.asarray(self.mask_ious, dtype=float)
        tp_gt_area = np.array(
            [int(g.sum()) for _, g in self._matched_mask_pairs], dtype=float
        )
        pct_edges = _percentile_size_edges(gt_areas_all)
        out = self._per_size_breakdown(gt_areas_all, tp_iou, tp_gt_area, pct_edges)
        out["scheme"] = "percentile"
        out["coco"] = self._per_size_breakdown(
            gt_areas_all, tp_iou, tp_gt_area, COCO_SIZE_EDGES
        )
        return out

    def _match_masks_coco(
        self, iou_threshold: float
    ) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
        """Greedy score-ranked pred->GT matching at one IoU threshold (COCO).

        Per frame, predictions are considered in descending score order; each
        claims the highest-IoU not-yet-claimed GT whose IoU >= ``iou_threshold``
        (a TP), else it is a FP. Mirrors ``pycocotools`` matching.

        Returns:
            ``(scores, matched, matched_gt_area, pred_area)`` flat arrays over
            every prediction across all frames (aligned). ``matched`` is the
            TP flag; ``matched_gt_area`` is the area of the claimed GT (NaN for
            a FP); ``pred_area`` is the prediction's own area.
        """
        scores, matched, matched_gt_area, pred_area = [], [], [], []
        for f in self._mask_frames:
            iou = f["iou"]
            pred_scores = f["pred_scores"]
            gt_areas = f["gt_areas"]
            pred_areas = f["pred_areas"]
            n_pred, n_gt = iou.shape
            order = (
                np.argsort(-pred_scores, kind="mergesort")
                if n_pred
                else np.array([], dtype=int)
            )
            gt_taken = np.zeros(n_gt, dtype=bool)
            for p in order:
                scores.append(float(pred_scores[p]))
                pred_area.append(float(pred_areas[p]))
                if n_gt == 0:
                    matched.append(False)
                    matched_gt_area.append(np.nan)
                    continue
                row = iou[p].copy()
                row[gt_taken] = -1.0
                g = int(np.argmax(row))
                if row[g] >= iou_threshold:
                    gt_taken[g] = True
                    matched.append(True)
                    matched_gt_area.append(float(gt_areas[g]))
                else:
                    matched.append(False)
                    matched_gt_area.append(np.nan)
        return (
            np.array(scores, dtype=float),
            np.array(matched, dtype=bool),
            np.array(matched_gt_area, dtype=float),
            np.array(pred_area, dtype=float),
        )

    def mask_voc_metrics(
        self,
        iou_thresholds: np.ndarray = MASK_IOU_THRESHOLDS,
        recall_thresholds: np.ndarray = np.linspace(0, 1, 101),
        size_percentiles: Tuple[float, float] = DEFAULT_SIZE_PERCENTILES,
    ) -> dict:
        """COCO-style score-ranked mask Average Precision / Recall.

        Re-matches predictions to GT independently at each IoU threshold
        (:meth:`_match_masks_coco`), score-ranks the resulting TP/FP flags, and
        integrates the precision-recall curve (101-point interpolation, mirrors
        :meth:`voc_metrics`). Reports overall AP@[.5:.95]/AP50/AP75/AR plus a
        per-object-size AP breakdown under two bucketing schemes (GT outside a
        bucket is ignored, as in ``pycocotools`` ``areaRng``): the primary
        (default) buckets use dataset-relative percentile edges (terciles), and
        the COCO fixed-cutoff buckets are reported additionally under the
        ``mask_voc.coco.`` prefix — analogous to the dual OKS/PCK VOC.

        Args:
            iou_thresholds: IoU thresholds to average AP over.
            recall_thresholds: Recall grid for 101-point interpolation.
            size_percentiles: Two percentiles of the GT area distribution
                delimiting the primary small/medium/large buckets.

        Returns:
            A dict keyed under ``"mask_voc."``: ``AP`` (per-threshold array),
            ``mAP``, ``AP50``, ``AP75``, ``AR``, ``recalls``, ``iou_thresholds``,
            ``n_gt``; primary per-size ``AP_small``/``AP_medium``/``AP_large``,
            ``n_gt_small``/``..._medium``/``..._large``, ``size_scheme`` and
            ``size_edges``; and COCO per-size ``coco.AP_*``/``coco.n_gt_*``/
            ``coco.size_edges``. AP values are NaN when the relevant GT set is
            empty.
        """
        iou_thresholds = np.asarray(iou_thresholds, dtype=float)
        recall_thresholds = np.asarray(recall_thresholds, dtype=float)
        gt_areas_all = np.array(
            [a for f in self._mask_frames for a in f["gt_areas"]], dtype=float
        )
        npig = int(gt_areas_all.size)

        # Primary (percentile, dataset-relative) + additional (COCO) edges.
        schemes = {
            "percentile": _percentile_size_edges(gt_areas_all, size_percentiles),
            "coco": COCO_SIZE_EDGES,
        }
        n_gt_size = {
            name: [
                int(np.count_nonzero(_size_mask(gt_areas_all, i, edges)))
                for i in range(len(_SIZE_KEYS))
            ]
            for name, edges in schemes.items()
        }

        ap_overall = np.full(iou_thresholds.size, np.nan)
        recall_overall = np.full(iou_thresholds.size, np.nan)
        ap_size = {
            name: [np.full(iou_thresholds.size, np.nan) for _ in _SIZE_KEYS]
            for name in schemes
        }

        for ti, thr in enumerate(iou_thresholds):
            scores, matched, matched_gt_area, pred_area = self._match_masks_coco(
                float(thr)
            )
            ap_overall[ti], recall_overall[ti] = _ap_from_pr(
                scores, matched, npig, recall_thresholds
            )
            for name, edges in schemes.items():
                for i in range(len(_SIZE_KEYS)):
                    # COCO areaRng: keep TPs whose matched GT is in-bucket and
                    # FPs whose own area is in-bucket; ignore everything else.
                    keep_tp = matched & _size_mask(matched_gt_area, i, edges)
                    keep_fp = (~matched) & _size_mask(pred_area, i, edges)
                    keep = keep_tp | keep_fp
                    ap_size[name][i][ti], _ = _ap_from_pr(
                        scores[keep],
                        keep_tp[keep],
                        n_gt_size[name][i],
                        recall_thresholds,
                    )

        def _nanmean(arr: np.ndarray) -> float:
            return float(np.nanmean(arr)) if np.any(~np.isnan(arr)) else np.nan

        def _at(target: float) -> float:
            return float(ap_overall[int(np.argmin(np.abs(iou_thresholds - target)))])

        results = {
            "mask_voc.iou_thresholds": iou_thresholds,
            "mask_voc.AP": ap_overall,
            "mask_voc.recalls": recall_overall,
            "mask_voc.mAP": _nanmean(ap_overall),
            "mask_voc.AR": _nanmean(recall_overall),
            "mask_voc.AP50": _at(0.5),
            "mask_voc.AP75": _at(0.75),
            "mask_voc.n_gt": npig,
            "mask_voc.size_scheme": "percentile",
            "mask_voc.size_edges": [float(e) for e in schemes["percentile"]],
            "mask_voc.coco.size_edges": [float(e) for e in schemes["coco"]],
        }
        # Primary (percentile) per-size keys are unprefixed; COCO is additional.
        for name, prefix in (("percentile", "mask_voc."), ("coco", "mask_voc.coco.")):
            for i, bucket in enumerate(_SIZE_KEYS):
                results[f"{prefix}AP_{bucket}"] = _nanmean(ap_size[name][i])
                results[f"{prefix}n_gt_{bucket}"] = n_gt_size[name][i]
        return results

    def pck_metrics(self, thresholds: np.ndarray = np.linspace(1, 10, 10)):
        """Compute PCK across a range of thresholds using the pair-wise distances.

        Args:
            thresholds: A list of distance thresholds in pixels.

        Returns:
            A dictionary of PCK metrics evaluated at each threshold.
        """
        dists = self.dists_dict["dists"]
        dists = np.copy(dists)
        dists[np.isnan(dists)] = np.inf
        pcks = np.expand_dims(dists, -1) < np.reshape(thresholds, (1, 1, -1))

        # Guard the empty-match case (0 positive pairs for the whole split) so
        # the nested .mean() reductions below don't hit "Mean of empty slice".
        if dists.size == 0:
            mPCK_parts = np.array([])
            mPCK = np.nan
            pck5 = np.nan
            pck10 = np.nan
        else:
            mPCK_parts = pcks.mean(axis=0).mean(axis=-1)
            mPCK = float(mPCK_parts.mean())

            # Precompute PCK at common thresholds
            idx_5 = np.argmin(np.abs(thresholds - 5))
            idx_10 = np.argmin(np.abs(thresholds - 10))
            pck5 = float(pcks[:, :, idx_5].mean())
            pck10 = float(pcks[:, :, idx_10].mean())

        return {
            "thresholds": thresholds,
            "pcks": pcks,
            "mPCK_parts": mPCK_parts,
            "mPCK": mPCK,
            "PCK@5": pck5,
            "PCK@10": pck10,
        }

    def visibility_metrics(self):
        """Compute node visibility metrics for the matched pair of instances.

        Returns:
            A dictionary of visibility metrics, including the confusion matrix.
        """
        vis_tp = 0
        vis_fn = 0
        vis_fp = 0
        vis_tn = 0

        for instance_gt, instance_pr, _ in self.positive_pairs:
            missing_nodes_gt = np.isnan(instance_gt.instance.numpy()).any(axis=-1)
            missing_nodes_pr = np.isnan(instance_pr.instance.numpy()).any(axis=-1)

            vis_tn += ((missing_nodes_gt) & (missing_nodes_pr)).sum()
            vis_fn += ((~missing_nodes_gt) & (missing_nodes_pr)).sum()
            vis_fp += ((missing_nodes_gt) & (~missing_nodes_pr)).sum()
            vis_tp += ((~missing_nodes_gt) & (~missing_nodes_pr)).sum()

        return {
            "tp": vis_tp,
            "fp": vis_fp,
            "tn": vis_tn,
            "fn": vis_fn,
            "precision": vis_tp / (vis_tp + vis_fp) if (vis_tp + vis_fp) else np.nan,
            "recall": vis_tp / (vis_tp + vis_fn) if (vis_tp + vis_fn) else np.nan,
        }

    def evaluate(self):
        """Return the evaluation metrics."""
        if self.match_method == "centroid":
            # Single-node / centroid-only: OKS/PCK/mOKS/visibility are
            # degenerate for one node, so we only report detection +
            # distance metrics. We intentionally do NOT compute OKS for a
            # single node (no magic OKS-scale constant) — the OKS path stays
            # only for match_method="oks".
            return {
                "detection_metrics": self.detection_metrics(),
                "distance_metrics": self.distance_metrics(),
            }

        if self.match_method == "mask":
            # Instance segmentation: detection (precision/recall/F1 over
            # IoU-matched masks) + mask-IoU quality + COCO-style score-ranked
            # mask AP/AR. OKS/PCK/visibility are keypoint-only and not computed.
            return {
                "detection_metrics": self.detection_metrics(),
                "mask_metrics": self.mask_metrics(),
                "mask_voc_metrics": self.mask_voc_metrics(),
            }

        if self.match_method == "semantic":
            # Whole-frame binary foreground segmentation: no instances to match, so
            # report only matching-free foreground IoU / clDice / boundary-IoU.
            return {"semantic_metrics": self.semantic_metrics()}

        if not self.positive_pairs:
            # 0 matched instances for the whole split (e.g. a collapsed model
            # predicting nothing, or predictions that never clear the OKS
            # threshold) -- every metric below is undefined by construction.
            # The individual methods already guard their own NaN/empty-array
            # math, so this is just one clear line instead of relying on the
            # reader to infer "collapsed model" from a wall of NaNs.
            logger.info(
                "0 matched instances: metrics undefined (model predicted "
                "nothing usable, or training likely collapsed)."
            )

        metrics = {}
        metrics["voc_metrics"] = self.voc_metrics(match_score_by="oks")
        metrics["voc_metrics"].update(self.voc_metrics(match_score_by="pck"))
        metrics["mOKS"] = self.mOKS()
        metrics["distance_metrics"] = self.distance_metrics()
        metrics["pck_metrics"] = self.pck_metrics()
        metrics["visibility_metrics"] = self.visibility_metrics()

        return metrics

__init__(ground_truth_instances, predicted_instances, oks_stddev=0.025, oks_scale=None, match_threshold=0, user_labels_only=True, match_method='oks', anchor_ind=None, centroid_method=None, centroid_fallback=None, exclude_predicted_instance_masks=False)

Initialize the Evaluator class with ground-truth and predicted labels.

exclude_predicted_instance_masks (match_method="mask" only) drops masks linked to a PredictedInstance from the ground-truth labels, so a labels file that carries stray predicted instances (each of which gets a mask when masks are built from poses) does not treat them as ground truth. It is kept separate from user_labels_only (which controls the frame-pair filter) because mask mode disables that frame filter -- see :func:run_evaluation.

Source code in sleap_nn/evaluation.py

def __init__(
    self,
    ground_truth_instances: sio.Labels,
    predicted_instances: sio.Labels,
    oks_stddev: float = 0.025,
    oks_scale: Optional[float] = None,
    match_threshold: float = 0,
    user_labels_only: bool = True,
    match_method: str = "oks",
    anchor_ind: Optional[int] = None,
    centroid_method: Optional[str] = None,
    centroid_fallback: Optional[str] = None,
    exclude_predicted_instance_masks: bool = False,
):
    """Initialize the Evaluator class with ground-truth and predicted labels.

    ``exclude_predicted_instance_masks`` (``match_method="mask"`` only) drops
    masks linked to a ``PredictedInstance`` from the ground-truth labels, so a
    labels file that carries stray predicted instances (each of which gets a
    mask when masks are built from poses) does not treat them as ground truth.
    It is kept separate from ``user_labels_only`` (which controls the frame-pair
    filter) because mask mode disables that frame filter -- see
    :func:`run_evaluation`.
    """
    self.ground_truth_instances = ground_truth_instances
    self.predicted_instances = predicted_instances
    self.match_threshold = match_threshold
    self.oks_stddev = oks_stddev
    self.oks_scale = oks_scale
    self.user_labels_only = user_labels_only
    self.match_method = match_method
    self.anchor_ind = anchor_ind
    self.centroid_method = centroid_method
    self.centroid_fallback = centroid_fallback
    self.exclude_predicted_instance_masks = exclude_predicted_instance_masks
    # Populated only in centroid / mask mode.
    self.false_positives = []
    # Matched-pair IoUs, populated only in mask mode.
    self.mask_ious = np.array([])
    # Per-frame mask records + matched TP mask pairs, populated only in mask
    # mode (feed mask_voc_metrics / boundary-IoU / fragmentation / per-size).
    self._mask_frames = []
    self._matched_mask_pairs = []
    # Per-frame (iou, cldice, boundary_iou) triples, populated only in
    # match_method="semantic" (whole-frame foreground, no matching).
    self._semantic_rows = []

    self._process_frames()

detection_metrics()

Compute detection metrics (precision/recall/F1) over TP/FP/FN counts.

Used by both match_method="centroid" and match_method="mask" (it only reads the matched/unmatched list lengths and dists_dict). Mirrors CentroidEvaluationCallback._compute_metrics. For centroid mode the localization-error percentiles are computed over the Euclidean distances of matched centroid pairs; for mask mode dists_dict is empty so those percentiles are NaN (per-pair IoU is reported separately via :meth:mask_metrics). Not used for match_method="oks" (which reports OKS-based VOC metrics instead).

Returns:

Type	Description
dict	A dict with precision, recall, f1, n_tp, n_fp, n_fn and localization-error percentiles avg/p50/p75/ p90/p95/p99 (NaN when there are no matched pairs).

Source code in sleap_nn/evaluation.py

def detection_metrics(self) -> dict:
    """Compute detection metrics (precision/recall/F1) over TP/FP/FN counts.

    Used by both ``match_method="centroid"`` and ``match_method="mask"``
    (it only reads the matched/unmatched list lengths and ``dists_dict``).
    Mirrors ``CentroidEvaluationCallback._compute_metrics``. For centroid
    mode the localization-error percentiles are computed over the Euclidean
    distances of matched centroid pairs; for mask mode ``dists_dict`` is
    empty so those percentiles are NaN (per-pair IoU is reported separately
    via :meth:`mask_metrics`). Not used for ``match_method="oks"`` (which
    reports OKS-based VOC metrics instead).

    Returns:
        A dict with ``precision``, ``recall``, ``f1``, ``n_tp``, ``n_fp``,
        ``n_fn`` and localization-error percentiles ``avg``/``p50``/``p75``/
        ``p90``/``p95``/``p99`` (NaN when there are no matched pairs).
    """
    n_tp = len(self.positive_pairs)
    n_fp = len(self.false_positives)
    n_fn = len(self.false_negatives)

    precision = n_tp / (n_tp + n_fp) if (n_tp + n_fp) > 0 else 0.0
    recall = n_tp / (n_tp + n_fn) if (n_tp + n_fn) > 0 else 0.0
    f1 = (
        2 * precision * recall / (precision + recall)
        if (precision + recall) > 0
        else 0.0
    )

    dists = self.dists_dict["dists"]
    results = {
        "precision": precision,
        "recall": recall,
        "f1": f1,
        "n_tp": n_tp,
        "n_fp": n_fp,
        "n_fn": n_fn,
        "avg": np.nan,
        "p50": np.nan,
        "p75": np.nan,
        "p90": np.nan,
        "p95": np.nan,
        "p99": np.nan,
    }

    is_non_nan = ~np.isnan(dists) if len(dists) else np.array([], dtype=bool)
    if np.any(is_non_nan):
        non_nans = dists[is_non_nan]
        results["avg"] = float(np.mean(non_nans))
        for ptile in (50, 75, 90, 95, 99):
            results[f"p{ptile}"] = float(np.percentile(non_nans, ptile))

    return results

distance_metrics()

Compute the Euclidean distance error at different percentiles using the pairwise distances.

Returns:

Type	Description
	A dictionary of distance metrics.

Source code in sleap_nn/evaluation.py

def distance_metrics(self):
    """Compute the Euclidean distance error at different percentiles using the pairwise distances.

    Returns:
        A dictionary of distance metrics.
    """
    dists = self.dists_dict["dists"]
    results = {
        "frame_idxs": self.dists_dict["frame_idxs"],
        "video_paths": self.dists_dict["video_paths"],
        "dists": dists,
        # Guard the empty / all-NaN matched set (zero true positives in a
        # split) so np.nanmean doesn't emit a "Mean of empty slice" warning.
        "avg": (
            float(np.nanmean(dists))
            if np.asarray(dists).size and not np.all(np.isnan(dists))
            else np.nan
        ),
        "p50": np.nan,
        "p75": np.nan,
        "p90": np.nan,
        "p95": np.nan,
        "p99": np.nan,
    }

    is_non_nan = ~np.isnan(dists)
    if np.any(is_non_nan):
        non_nans = dists[is_non_nan]
        for ptile in (50, 75, 90, 95, 99):
            results[f"p{ptile}"] = np.percentile(non_nans, ptile)

    return results

evaluate()

Return the evaluation metrics.

Source code in sleap_nn/evaluation.py

def evaluate(self):
    """Return the evaluation metrics."""
    if self.match_method == "centroid":
        # Single-node / centroid-only: OKS/PCK/mOKS/visibility are
        # degenerate for one node, so we only report detection +
        # distance metrics. We intentionally do NOT compute OKS for a
        # single node (no magic OKS-scale constant) — the OKS path stays
        # only for match_method="oks".
        return {
            "detection_metrics": self.detection_metrics(),
            "distance_metrics": self.distance_metrics(),
        }

    if self.match_method == "mask":
        # Instance segmentation: detection (precision/recall/F1 over
        # IoU-matched masks) + mask-IoU quality + COCO-style score-ranked
        # mask AP/AR. OKS/PCK/visibility are keypoint-only and not computed.
        return {
            "detection_metrics": self.detection_metrics(),
            "mask_metrics": self.mask_metrics(),
            "mask_voc_metrics": self.mask_voc_metrics(),
        }

    if self.match_method == "semantic":
        # Whole-frame binary foreground segmentation: no instances to match, so
        # report only matching-free foreground IoU / clDice / boundary-IoU.
        return {"semantic_metrics": self.semantic_metrics()}

    if not self.positive_pairs:
        # 0 matched instances for the whole split (e.g. a collapsed model
        # predicting nothing, or predictions that never clear the OKS
        # threshold) -- every metric below is undefined by construction.
        # The individual methods already guard their own NaN/empty-array
        # math, so this is just one clear line instead of relying on the
        # reader to infer "collapsed model" from a wall of NaNs.
        logger.info(
            "0 matched instances: metrics undefined (model predicted "
            "nothing usable, or training likely collapsed)."
        )

    metrics = {}
    metrics["voc_metrics"] = self.voc_metrics(match_score_by="oks")
    metrics["voc_metrics"].update(self.voc_metrics(match_score_by="pck"))
    metrics["mOKS"] = self.mOKS()
    metrics["distance_metrics"] = self.distance_metrics()
    metrics["pck_metrics"] = self.pck_metrics()
    metrics["visibility_metrics"] = self.visibility_metrics()

    return metrics

mOKS()

Return the meanOKS value.

Source code in sleap_nn/evaluation.py

def mOKS(self):
    """Return the meanOKS value."""
    pair_oks = np.array([oks for _, _, oks in self.positive_pairs])
    return {"mOKS": float(pair_oks.mean()) if pair_oks.size else np.nan}

mask_metrics()

Compute mask-IoU summary statistics for match_method="mask".

Reports complementary IoU summaries, panoptic-quality, boundary-IoU, fragmentation, and per-object-size breakdowns:

• mean_iou (and min/max/percentiles) over the matched (TP) pairs only — COCO-style segmentation quality, blind to misses.
• mean_iou_all_gt — IoU averaged over all ground-truth masks, where an unmatched GT (a miss) contributes 0. This penalizes recall and complements the TP-only mean.
• Panoptic Quality pq = sq * rq with sq = mean_iou (segmentation quality) and rq = TP / (TP + 0.5*FP + 0.5*FN) (recognition quality, == detection F1). See Kirillov et al., "Panoptic Segmentation" (2019).
• mean_boundary_iou — boundary IoU over the matched pairs (Cheng et al., 2021), more sensitive to contour error than mask IoU.
• mean_cldice — centerline Dice over the matched pairs (Shit et al., CVPR 2021), connectivity-aware and nearly width-insensitive; a fairer score than IoU for thin/tubular structures. NaN if scikit-image is unavailable.
• oversegmentation / undersegmentation — fragmentation counts: GT masks split across >=2 predictions, and predictions spanning >=2 GT masks (each with >=10% area overlap). The headline over-/under- segmentation failure mode is invisible to the 1-to-1 match.
• per_size — COCO small/medium/large breakdown of GT count, TP count, and TP-only mean IoU (buckets sum to the GT total).

Returns:

Type	Description
dict	A dict with mean_iou, min, max, percentiles p25/ p50/p75, mean_iou_all_gt, pq/sq/rq, mean_boundary_iou, oversegmentation/undersegmentation, per_size, the TP count n_matched (plus n_fp/n_fn), and the raw ious array. Quantities are NaN when undefined.

Source code in sleap_nn/evaluation.py

def mask_metrics(self) -> dict:
    """Compute mask-IoU summary statistics for ``match_method="mask"``.

    Reports complementary IoU summaries, panoptic-quality, boundary-IoU,
    fragmentation, and per-object-size breakdowns:

    * ``mean_iou`` (and ``min``/``max``/percentiles) over the matched (TP)
      pairs only — COCO-style segmentation quality, blind to misses.
    * ``mean_iou_all_gt`` — IoU averaged over *all* ground-truth masks,
      where an unmatched GT (a miss) contributes ``0``. This penalizes
      recall and complements the TP-only mean.
    * Panoptic Quality ``pq = sq * rq`` with ``sq = mean_iou`` (segmentation
      quality) and ``rq = TP / (TP + 0.5*FP + 0.5*FN)`` (recognition
      quality, == detection F1). See Kirillov et al., "Panoptic
      Segmentation" (2019).
    * ``mean_boundary_iou`` — boundary IoU over the matched pairs (Cheng et
      al., 2021), more sensitive to contour error than mask IoU.
    * ``mean_cldice`` — centerline Dice over the matched pairs (Shit et al.,
      CVPR 2021), connectivity-aware and nearly width-insensitive; a fairer
      score than IoU for thin/tubular structures. NaN if scikit-image is
      unavailable.
    * ``oversegmentation`` / ``undersegmentation`` — fragmentation counts:
      GT masks split across >=2 predictions, and predictions spanning >=2
      GT masks (each with >=10% area overlap). The headline over-/under-
      segmentation failure mode is invisible to the 1-to-1 match.
    * ``per_size`` — COCO small/medium/large breakdown of GT count, TP
      count, and TP-only mean IoU (buckets sum to the GT total).

    Returns:
        A dict with ``mean_iou``, ``min``, ``max``, percentiles ``p25``/
        ``p50``/``p75``, ``mean_iou_all_gt``, ``pq``/``sq``/``rq``,
        ``mean_boundary_iou``, ``oversegmentation``/``undersegmentation``,
        ``per_size``, the TP count ``n_matched`` (plus ``n_fp``/``n_fn``),
        and the raw ``ious`` array. Quantities are NaN when undefined.
    """
    ious = np.asarray(self.mask_ious, dtype=float)
    n_tp = len(self.positive_pairs)
    n_fp = len(self.false_positives)
    n_fn = len(self.false_negatives)
    over, under = self._fragmentation_counts()
    results = {
        "mean_iou": np.nan,
        "min": np.nan,
        "max": np.nan,
        "p25": np.nan,
        "p50": np.nan,
        "p75": np.nan,
        "mean_iou_all_gt": np.nan,
        "pq": np.nan,
        "sq": np.nan,
        "rq": np.nan,
        "mean_boundary_iou": np.nan,
        "mean_cldice": np.nan,
        "oversegmentation": over,
        "undersegmentation": under,
        "per_size": self._mask_per_size_stats(),
        "n_matched": int(ious.size),
        "n_fp": n_fp,
        "n_fn": n_fn,
        "ious": ious,
    }
    if ious.size:
        results["mean_iou"] = float(np.mean(ious))
        results["min"] = float(np.min(ious))
        results["max"] = float(np.max(ious))
        for ptile in (25, 50, 75):
            results[f"p{ptile}"] = float(np.percentile(ious, ptile))

    if self._matched_mask_pairs:
        boundary_ious = np.array(
            [_boundary_iou(p, g) for p, g in self._matched_mask_pairs],
            dtype=float,
        )
        results["mean_boundary_iou"] = float(np.mean(boundary_ious))
        # Centerline Dice (clDice): connectivity/width-tolerant, fairer than
        # IoU for thin structures. NaN entries (scikit-image missing) drop out.
        cldices = np.array(
            [mask_cldice(p, g) for p, g in self._matched_mask_pairs],
            dtype=float,
        )
        cldices = cldices[~np.isnan(cldices)]
        if cldices.size:
            results["mean_cldice"] = float(np.mean(cldices))

    iou_sum = float(np.sum(ious)) if ious.size else 0.0
    # Miss-penalizing mean: averaged over every GT mask (TP + FN).
    n_gt = n_tp + n_fn
    if n_gt > 0:
        results["mean_iou_all_gt"] = iou_sum / n_gt
    # Panoptic quality: SQ = TP-only mean IoU, RQ = detection F1, PQ = SQ*RQ
    # = iou_sum / (TP + 0.5*FP + 0.5*FN).
    pq_denom = n_tp + 0.5 * n_fp + 0.5 * n_fn
    if pq_denom > 0:
        results["sq"] = results["mean_iou"]
        results["rq"] = n_tp / pq_denom
        results["pq"] = iou_sum / pq_denom
    return results

mask_voc_metrics(iou_thresholds=MASK_IOU_THRESHOLDS, recall_thresholds=np.linspace(0, 1, 101), size_percentiles=DEFAULT_SIZE_PERCENTILES)

COCO-style score-ranked mask Average Precision / Recall.

Re-matches predictions to GT independently at each IoU threshold (:meth:_match_masks_coco), score-ranks the resulting TP/FP flags, and integrates the precision-recall curve (101-point interpolation, mirrors :meth:voc_metrics). Reports overall AP@[.5:.95]/AP50/AP75/AR plus a per-object-size AP breakdown under two bucketing schemes (GT outside a bucket is ignored, as in pycocotools areaRng): the primary (default) buckets use dataset-relative percentile edges (terciles), and the COCO fixed-cutoff buckets are reported additionally under the mask_voc.coco. prefix — analogous to the dual OKS/PCK VOC.

Parameters:

Name	Type	Description	Default
iou_thresholds	ndarray	IoU thresholds to average AP over.	MASK_IOU_THRESHOLDS
recall_thresholds	ndarray	Recall grid for 101-point interpolation.	linspace(0, 1, 101)
size_percentiles	Tuple[float, float]	Two percentiles of the GT area distribution delimiting the primary small/medium/large buckets.	DEFAULT_SIZE_PERCENTILES

Returns:

Type	Description
dict	A dict keyed under "mask_voc.": AP (per-threshold array), mAP, AP50, AP75, AR, recalls, iou_thresholds, n_gt; primary per-size AP_small/AP_medium/AP_large, n_gt_small/..._medium/..._large, size_scheme and size_edges; and COCO per-size coco.AP_*/coco.n_gt_*/ coco.size_edges. AP values are NaN when the relevant GT set is empty.

Source code in sleap_nn/evaluation.py

def mask_voc_metrics(
    self,
    iou_thresholds: np.ndarray = MASK_IOU_THRESHOLDS,
    recall_thresholds: np.ndarray = np.linspace(0, 1, 101),
    size_percentiles: Tuple[float, float] = DEFAULT_SIZE_PERCENTILES,
) -> dict:
    """COCO-style score-ranked mask Average Precision / Recall.

    Re-matches predictions to GT independently at each IoU threshold
    (:meth:`_match_masks_coco`), score-ranks the resulting TP/FP flags, and
    integrates the precision-recall curve (101-point interpolation, mirrors
    :meth:`voc_metrics`). Reports overall AP@[.5:.95]/AP50/AP75/AR plus a
    per-object-size AP breakdown under two bucketing schemes (GT outside a
    bucket is ignored, as in ``pycocotools`` ``areaRng``): the primary
    (default) buckets use dataset-relative percentile edges (terciles), and
    the COCO fixed-cutoff buckets are reported additionally under the
    ``mask_voc.coco.`` prefix — analogous to the dual OKS/PCK VOC.

    Args:
        iou_thresholds: IoU thresholds to average AP over.
        recall_thresholds: Recall grid for 101-point interpolation.
        size_percentiles: Two percentiles of the GT area distribution
            delimiting the primary small/medium/large buckets.

    Returns:
        A dict keyed under ``"mask_voc."``: ``AP`` (per-threshold array),
        ``mAP``, ``AP50``, ``AP75``, ``AR``, ``recalls``, ``iou_thresholds``,
        ``n_gt``; primary per-size ``AP_small``/``AP_medium``/``AP_large``,
        ``n_gt_small``/``..._medium``/``..._large``, ``size_scheme`` and
        ``size_edges``; and COCO per-size ``coco.AP_*``/``coco.n_gt_*``/
        ``coco.size_edges``. AP values are NaN when the relevant GT set is
        empty.
    """
    iou_thresholds = np.asarray(iou_thresholds, dtype=float)
    recall_thresholds = np.asarray(recall_thresholds, dtype=float)
    gt_areas_all = np.array(
        [a for f in self._mask_frames for a in f["gt_areas"]], dtype=float
    )
    npig = int(gt_areas_all.size)

    # Primary (percentile, dataset-relative) + additional (COCO) edges.
    schemes = {
        "percentile": _percentile_size_edges(gt_areas_all, size_percentiles),
        "coco": COCO_SIZE_EDGES,
    }
    n_gt_size = {
        name: [
            int(np.count_nonzero(_size_mask(gt_areas_all, i, edges)))
            for i in range(len(_SIZE_KEYS))
        ]
        for name, edges in schemes.items()
    }

    ap_overall = np.full(iou_thresholds.size, np.nan)
    recall_overall = np.full(iou_thresholds.size, np.nan)
    ap_size = {
        name: [np.full(iou_thresholds.size, np.nan) for _ in _SIZE_KEYS]
        for name in schemes
    }

    for ti, thr in enumerate(iou_thresholds):
        scores, matched, matched_gt_area, pred_area = self._match_masks_coco(
            float(thr)
        )
        ap_overall[ti], recall_overall[ti] = _ap_from_pr(
            scores, matched, npig, recall_thresholds
        )
        for name, edges in schemes.items():
            for i in range(len(_SIZE_KEYS)):
                # COCO areaRng: keep TPs whose matched GT is in-bucket and
                # FPs whose own area is in-bucket; ignore everything else.
                keep_tp = matched & _size_mask(matched_gt_area, i, edges)
                keep_fp = (~matched) & _size_mask(pred_area, i, edges)
                keep = keep_tp | keep_fp
                ap_size[name][i][ti], _ = _ap_from_pr(
                    scores[keep],
                    keep_tp[keep],
                    n_gt_size[name][i],
                    recall_thresholds,
                )

    def _nanmean(arr: np.ndarray) -> float:
        return float(np.nanmean(arr)) if np.any(~np.isnan(arr)) else np.nan

    def _at(target: float) -> float:
        return float(ap_overall[int(np.argmin(np.abs(iou_thresholds - target)))])

    results = {
        "mask_voc.iou_thresholds": iou_thresholds,
        "mask_voc.AP": ap_overall,
        "mask_voc.recalls": recall_overall,
        "mask_voc.mAP": _nanmean(ap_overall),
        "mask_voc.AR": _nanmean(recall_overall),
        "mask_voc.AP50": _at(0.5),
        "mask_voc.AP75": _at(0.75),
        "mask_voc.n_gt": npig,
        "mask_voc.size_scheme": "percentile",
        "mask_voc.size_edges": [float(e) for e in schemes["percentile"]],
        "mask_voc.coco.size_edges": [float(e) for e in schemes["coco"]],
    }
    # Primary (percentile) per-size keys are unprefixed; COCO is additional.
    for name, prefix in (("percentile", "mask_voc."), ("coco", "mask_voc.coco.")):
        for i, bucket in enumerate(_SIZE_KEYS):
            results[f"{prefix}AP_{bucket}"] = _nanmean(ap_size[name][i])
            results[f"{prefix}n_gt_{bucket}"] = n_gt_size[name][i]
    return results

pck_metrics(thresholds=np.linspace(1, 10, 10))

Compute PCK across a range of thresholds using the pair-wise distances.

Parameters:

Name	Type	Description	Default
thresholds	ndarray	A list of distance thresholds in pixels.	linspace(1, 10, 10)

Returns:

Type	Description
	A dictionary of PCK metrics evaluated at each threshold.

Source code in sleap_nn/evaluation.py

def pck_metrics(self, thresholds: np.ndarray = np.linspace(1, 10, 10)):
    """Compute PCK across a range of thresholds using the pair-wise distances.

    Args:
        thresholds: A list of distance thresholds in pixels.

    Returns:
        A dictionary of PCK metrics evaluated at each threshold.
    """
    dists = self.dists_dict["dists"]
    dists = np.copy(dists)
    dists[np.isnan(dists)] = np.inf
    pcks = np.expand_dims(dists, -1) < np.reshape(thresholds, (1, 1, -1))

    # Guard the empty-match case (0 positive pairs for the whole split) so
    # the nested .mean() reductions below don't hit "Mean of empty slice".
    if dists.size == 0:
        mPCK_parts = np.array([])
        mPCK = np.nan
        pck5 = np.nan
        pck10 = np.nan
    else:
        mPCK_parts = pcks.mean(axis=0).mean(axis=-1)
        mPCK = float(mPCK_parts.mean())

        # Precompute PCK at common thresholds
        idx_5 = np.argmin(np.abs(thresholds - 5))
        idx_10 = np.argmin(np.abs(thresholds - 10))
        pck5 = float(pcks[:, :, idx_5].mean())
        pck10 = float(pcks[:, :, idx_10].mean())

    return {
        "thresholds": thresholds,
        "pcks": pcks,
        "mPCK_parts": mPCK_parts,
        "mPCK": mPCK,
        "PCK@5": pck5,
        "PCK@10": pck10,
    }

semantic_metrics()

Aggregate whole-frame foreground metrics for match_method="semantic".

Averages the per-frame foreground IoU, centerline Dice (clDice), and boundary IoU computed by :meth:_process_frames_semantic over all frames with non-empty ground-truth foreground. clDice entries that are NaN (scikit-image unavailable) are dropped from the clDice mean; if every entry is NaN the reported mean_cldice is NaN.

Returns:

Type	Description
dict	A dict with mean_iou, mean_cldice, mean_boundary_iou, the per-frame ious / cldices / boundary_ious arrays, and n_frames (frames scored). Means are NaN when no frame was scored.

Source code in sleap_nn/evaluation.py

def semantic_metrics(self) -> dict:
    """Aggregate whole-frame foreground metrics for ``match_method="semantic"``.

    Averages the per-frame foreground IoU, centerline Dice (clDice), and
    boundary IoU computed by :meth:`_process_frames_semantic` over all frames
    with non-empty ground-truth foreground. clDice entries that are NaN
    (scikit-image unavailable) are dropped from the clDice mean; if every entry
    is NaN the reported ``mean_cldice`` is NaN.

    Returns:
        A dict with ``mean_iou``, ``mean_cldice``, ``mean_boundary_iou``, the
        per-frame ``ious`` / ``cldices`` / ``boundary_ious`` arrays, and
        ``n_frames`` (frames scored). Means are NaN when no frame was scored.
    """
    rows = np.asarray(self._semantic_rows, dtype=float).reshape(-1, 3)
    ious = rows[:, 0]
    cldices = rows[:, 1]
    bious = rows[:, 2]
    cld_valid = cldices[~np.isnan(cldices)]
    return {
        "mean_iou": float(np.mean(ious)) if ious.size else float("nan"),
        "mean_cldice": (
            float(np.mean(cld_valid)) if cld_valid.size else float("nan")
        ),
        "mean_boundary_iou": (
            float(np.mean(bious)) if bious.size else float("nan")
        ),
        "ious": ious,
        "cldices": cldices,
        "boundary_ious": bious,
        "n_frames": int(ious.size),
    }

visibility_metrics()

Compute node visibility metrics for the matched pair of instances.

Returns:

Type	Description
	A dictionary of visibility metrics, including the confusion matrix.

Source code in sleap_nn/evaluation.py

def visibility_metrics(self):
    """Compute node visibility metrics for the matched pair of instances.

    Returns:
        A dictionary of visibility metrics, including the confusion matrix.
    """
    vis_tp = 0
    vis_fn = 0
    vis_fp = 0
    vis_tn = 0

    for instance_gt, instance_pr, _ in self.positive_pairs:
        missing_nodes_gt = np.isnan(instance_gt.instance.numpy()).any(axis=-1)
        missing_nodes_pr = np.isnan(instance_pr.instance.numpy()).any(axis=-1)

        vis_tn += ((missing_nodes_gt) & (missing_nodes_pr)).sum()
        vis_fn += ((~missing_nodes_gt) & (missing_nodes_pr)).sum()
        vis_fp += ((missing_nodes_gt) & (~missing_nodes_pr)).sum()
        vis_tp += ((~missing_nodes_gt) & (~missing_nodes_pr)).sum()

    return {
        "tp": vis_tp,
        "fp": vis_fp,
        "tn": vis_tn,
        "fn": vis_fn,
        "precision": vis_tp / (vis_tp + vis_fp) if (vis_tp + vis_fp) else np.nan,
        "recall": vis_tp / (vis_tp + vis_fn) if (vis_tp + vis_fn) else np.nan,
    }

voc_metrics(match_score_by='oks', match_score_thresholds=np.linspace(0.5, 0.95, 10), recall_thresholds=np.linspace(0, 1, 101))

Compute VOC metrics for a matched pairs of instances positive pairs and false negatives.

Parameters:

Name	Type	Description	Default
match_score_by		The score to be used for computing the metrics. "ock" or "pck"	'oks'
match_score_thresholds	ndarray	Score thresholds at which to consider matches as a true positive match.	linspace(0.5, 0.95, 10)
recall_thresholds	ndarray	Recall thresholds at which to evaluate Average Precision.	linspace(0, 1, 101)

Returns:

Type	Description
	A dictionary of VOC metrics.

Source code in sleap_nn/evaluation.py

def voc_metrics(
    self,
    match_score_by="oks",
    match_score_thresholds: np.ndarray = np.linspace(
        0.5, 0.95, 10
    ),  # 0.5:0.05:0.95
    recall_thresholds: np.ndarray = np.linspace(0, 1, 101),  # 0.0:0.01:1.00
):
    """Compute VOC metrics for a matched pairs of instances positive pairs and false negatives.

    Args:
        match_score_by: The score to be used for computing the metrics. "ock" or "pck"
        match_score_thresholds: Score thresholds at which to consider matches as a true
            positive match.
        recall_thresholds: Recall thresholds at which to evaluate Average Precision.

    Returns:
        A dictionary of VOC metrics.
    """
    if match_score_by == "oks":
        match_scores = np.array([oks for _, _, oks in self.positive_pairs])
        name = "oks_voc"
    elif match_score_by == "pck":
        name = "pck_voc"
        if not self.positive_pairs:
            # Guard the empty-match case: the (n_pairs, n_nodes, n_thresholds)
            # ``pcks`` array is empty along the pairs axis, so reducing it with
            # nested .mean() calls would hit "Mean of empty slice".
            match_scores = np.array([])
        else:
            pck_metrics = self.pck_metrics()
            match_scores = pck_metrics["pcks"].mean(axis=-1).mean(axis=-1)
    else:
        message = "Invalid Option for match_score_by. Choose either `oks` or `pck`"
        logger.error(message)
        raise Exception(message)

    detection_scores = np.array(
        [pp[1].instance.score for pp in self.positive_pairs]
    )

    inds = np.argsort(-detection_scores, kind="mergesort")
    detection_scores = detection_scores[inds]
    match_scores = match_scores[inds]

    precisions = []
    recalls = []

    npig = len(self.positive_pairs) + len(
        self.false_negatives
    )  # total number of GT instances

    for match_score_threshold in match_score_thresholds:
        tp = np.cumsum(match_scores >= match_score_threshold)
        fp = np.cumsum(match_scores < match_score_threshold)

        if tp.size == 0:
            return {
                name + ".match_score_thresholds": 0,
                name + ".recall_thresholds": 0,
                name + ".match_scores": 0,
                name + ".precisions": 0,
                name + ".recalls": 0,
                name + ".AP": 0,
                name + ".AR": 0,
                name + ".mAP": 0,
                name + ".mAR": 0,
            }

        rc = tp / npig
        pr = tp / (fp + tp + np.spacing(1))

        recall = rc[-1]  # best recall at this OKS threshold

        # Ensure strictly decreasing precisions.
        for i in range(len(pr) - 1, 0, -1):
            if pr[i] > pr[i - 1]:
                pr[i - 1] = pr[i]

        # Find best precision at each recall threshold.
        rc_inds = np.searchsorted(rc, recall_thresholds, side="left")
        precision = np.zeros(rc_inds.shape)
        is_valid_rc_ind = rc_inds < len(pr)
        precision[is_valid_rc_ind] = pr[rc_inds[is_valid_rc_ind]]

        precisions.append(precision)
        recalls.append(recall)

    precisions = np.array(precisions)
    recalls = np.array(recalls)

    AP = precisions.mean(
        axis=1
    )  # AP = average precision over fixed set of recall thresholds
    AR = recalls  # AR = max recall given a fixed number of detections per image

    mAP = precisions.mean()  # mAP = mean over all OKS thresholds
    mAR = recalls.mean()  # mAR = mean over all OKS thresholds

    return {
        name + ".match_score_thresholds": match_score_thresholds,
        name + ".recall_thresholds": recall_thresholds,
        name + ".match_scores": match_scores,
        name + ".precisions": precisions,
        name + ".recalls": recalls,
        name + ".AP": AP,
        name + ".AR": AR,
        name + ".mAP": mAP,
        name + ".mAR": mAR,
    }

IdentityMetrics

Identity-persistence metrics for one tracked prediction.

These are sleap-nn's own (legacy) definitions. Four of them differ from TrackEval, the reference MOT evaluation code; quote :func:compute_mot_metrics (:class:MOTMetrics) when comparing with published MOT numbers:

1. ID switches. Each frame is matched by a plain Hungarian assignment, with no preference for keeping last frame's match. TrackEval's CLEAR-MOT first keeps the previous frame's pairs that are still above threshold, so id_switches can exceed its IDSW when a competing prediction scores higher for a frame.
2. IDF1. IDTP counts one-to-one co-matches from those per-frame assignments. TrackEval counts every frame on which a ground-truth / predicted identity pair is above threshold, matched or not, so its IDTP (and IDF1) can be higher.
3. Mostly tracked is coverage >= mt_threshold here; TrackEval uses > 0.8 (a trajectory matched on exactly 4 of 5 frames is MT here, PT in TrackEval).
4. Fragmentation counts matched -> unmatched -> re-matched interruptions over the frames the ground truth is present. TrackEval counts tracked segments minus one, and its segments are also broken by frames where the ground truth is absent but predictions are present.

Attributes:

Name	Type	Description
id_switches	int	CLEAR-MOT ID switches, summed over ground-truth trajectories.
idf1	float	Identity F1 (Ristani et al.). Includes detection errors: every unmatched tracked GT detection is an IDFN, every unmatched tracked prediction an IDFP.
idp	float	Identity precision.
idr	float	Identity recall.
idf1_matched	float	IDF1 over matched detections only (IDTP / n_matched): the share of matched detections carrying their trajectory's assigned identity, with the detector's misses and false positives factored out. The number to compare tracker arms by over frozen detections.
mostly_tracked	int	Ground-truth trajectories covered at or above mt_threshold.
partly_tracked	int	Ground-truth trajectories between the two coverage cuts.
mostly_lost	int	Ground-truth trajectories covered below ml_threshold.
fragmentations	int	Matched -> unmatched -> matched interruptions of a ground-truth trajectory.
mean_gt_coverage	float	Mean share of each trajectory's frames that matched.
mean_track_purity	float	Length-weighted mean dominant-identity share per predicted track.
n_gt_dets	int	Tracked ground-truth detections scored.
n_gt_untracked	int	Untracked ground-truth detections in the scored frames (don't-care: matched against, never scored).
n_pred_dets	int	Predicted detections in the scored frames.
n_matched	int	Tracked ground-truth/predicted pairs matched above threshold.
n_dont_care	int	Tracked predictions matched to an untracked ground-truth detection -- neither hits nor false positives.
n_frames_compared	int	Ground-truth frames scored (every GT frame of every scored video).
n_frames_missing_pred	int	Of those, frames the prediction does not have; their tracked GT detections all score as misses.
n_videos_compared	int	Videos scored.
n_gt_tracks	int	Distinct ground-truth trajectories -- (video, name) pairs, or names under global_identity.
n_pred_tracks	int	Distinct predicted identities, keyed the same way.
n_pred_untracked	int	Predicted detections with no track (after resolving through a pose <-> mask link).
match_method	str	How detections were matched: "oks", "iou" or "distance" (single-node skeletons, in pixels).
match_threshold	float	The threshold match_method was applied with.
global_identity	bool	Whether trajectories were keyed by name across videos.
notes	List[str]	Human-readable caveats raised while comparing.

Methods:

Name	Description
as_dict	Return the metrics as a plain, JSON-serializable dict.
summary	Return a one-line summary of the headline metrics.

Source code in sleap_nn/evaluation.py

@attrs.define(auto_attribs=True, slots=True)
class IdentityMetrics:
    """Identity-persistence metrics for one tracked prediction.

    These are sleap-nn's own (legacy) definitions. Four of them differ from
    TrackEval, the reference MOT evaluation code; quote
    :func:`compute_mot_metrics` (:class:`MOTMetrics`) when comparing with
    published MOT numbers:

    1. **ID switches.** Each frame is matched by a plain Hungarian assignment,
       with no preference for keeping last frame's match. TrackEval's CLEAR-MOT
       first keeps the previous frame's pairs that are still above threshold, so
       ``id_switches`` can exceed its ``IDSW`` when a competing prediction scores
       higher for a frame.
    2. **IDF1.** IDTP counts one-to-one co-matches from those per-frame
       assignments. TrackEval counts every frame on which a ground-truth /
       predicted identity pair is above threshold, matched or not, so its IDTP
       (and IDF1) can be higher.
    3. **Mostly tracked** is ``coverage >= mt_threshold`` here; TrackEval uses
       ``> 0.8`` (a trajectory matched on exactly 4 of 5 frames is MT here, PT
       in TrackEval).
    4. **Fragmentation** counts matched -> unmatched -> re-matched interruptions
       over the frames the ground truth is present. TrackEval counts tracked
       segments minus one, and its segments are also broken by frames where the
       ground truth is absent but predictions are present.

    Attributes:
        id_switches: CLEAR-MOT ID switches, summed over ground-truth trajectories.
        idf1: Identity F1 (Ristani et al.). Includes detection errors: every
            unmatched tracked GT detection is an IDFN, every unmatched tracked
            prediction an IDFP.
        idp: Identity precision.
        idr: Identity recall.
        idf1_matched: IDF1 over matched detections only (``IDTP / n_matched``):
            the share of matched detections carrying their trajectory's assigned
            identity, with the detector's misses and false positives factored
            out. The number to compare tracker arms by over frozen detections.
        mostly_tracked: Ground-truth trajectories covered at or above
            ``mt_threshold``.
        partly_tracked: Ground-truth trajectories between the two coverage cuts.
        mostly_lost: Ground-truth trajectories covered below ``ml_threshold``.
        fragmentations: Matched -> unmatched -> matched interruptions of a
            ground-truth trajectory.
        mean_gt_coverage: Mean share of each trajectory's frames that matched.
        mean_track_purity: Length-weighted mean dominant-identity share per
            predicted track.
        n_gt_dets: Tracked ground-truth detections scored.
        n_gt_untracked: Untracked ground-truth detections in the scored frames
            (don't-care: matched against, never scored).
        n_pred_dets: Predicted detections in the scored frames.
        n_matched: Tracked ground-truth/predicted pairs matched above threshold.
        n_dont_care: Tracked predictions matched to an untracked ground-truth
            detection -- neither hits nor false positives.
        n_frames_compared: Ground-truth frames scored (every GT frame of every
            scored video).
        n_frames_missing_pred: Of those, frames the prediction does not have;
            their tracked GT detections all score as misses.
        n_videos_compared: Videos scored.
        n_gt_tracks: Distinct ground-truth trajectories -- ``(video, name)``
            pairs, or names under ``global_identity``.
        n_pred_tracks: Distinct predicted identities, keyed the same way.
        n_pred_untracked: Predicted detections with no track (after resolving
            through a pose <-> mask link).
        match_method: How detections were matched: ``"oks"``, ``"iou"`` or
            ``"distance"`` (single-node skeletons, in pixels).
        match_threshold: The threshold ``match_method`` was applied with.
        global_identity: Whether trajectories were keyed by name across videos.
        notes: Human-readable caveats raised while comparing.
    """

    # Headline.
    id_switches: int = 0
    idf1: float = float("nan")
    idp: float = float("nan")
    idr: float = float("nan")
    idf1_matched: float = float("nan")
    # Coverage -- guards against winning on switches by tracking less.
    mostly_tracked: int = 0
    partly_tracked: int = 0
    mostly_lost: int = 0
    fragmentations: int = 0
    mean_gt_coverage: float = float("nan")
    # Purity.
    mean_track_purity: float = float("nan")
    # Detection accounting.
    n_gt_dets: int = 0
    n_gt_untracked: int = 0
    n_pred_dets: int = 0
    n_matched: int = 0
    n_dont_care: int = 0
    n_frames_compared: int = 0
    n_frames_missing_pred: int = 0
    n_videos_compared: int = 0
    n_gt_tracks: int = 0
    n_pred_tracks: int = 0
    n_pred_untracked: int = 0
    # How the comparison was made.
    match_method: str = ""
    match_threshold: float = float("nan")
    global_identity: bool = False
    notes: List[str] = attrs.field(factory=list)

    def as_dict(self) -> Dict[str, Any]:
        """Return the metrics as a plain, JSON-serializable dict."""
        return attrs.asdict(self)

    def summary(self) -> str:
        """Return a one-line summary of the headline metrics."""
        missing = (
            f", {self.n_frames_missing_pred} without a prediction"
            if self.n_frames_missing_pred
            else ""
        )
        return (
            f"IDSW={self.id_switches}  IDF1={self.idf1:.4f} "
            f"(P={self.idp:.4f} R={self.idr:.4f}; matched-only "
            f"{self.idf1_matched:.4f})  "
            f"MT/PT/ML={self.mostly_tracked}/{self.partly_tracked}/{self.mostly_lost}  "
            f"Frag={self.fragmentations}  purity={self.mean_track_purity:.4f}  "
            f"cov={self.mean_gt_coverage:.4f}  "
            f"[{self.n_matched}/{self.n_gt_dets} GT dets matched, "
            f"{self.n_pred_dets} pred, {self.n_frames_compared} frames{missing}]"
        )

as_dict()

Return the metrics as a plain, JSON-serializable dict.

Source code in sleap_nn/evaluation.py

def as_dict(self) -> Dict[str, Any]:
    """Return the metrics as a plain, JSON-serializable dict."""
    return attrs.asdict(self)

summary()

Return a one-line summary of the headline metrics.

Source code in sleap_nn/evaluation.py

def summary(self) -> str:
    """Return a one-line summary of the headline metrics."""
    missing = (
        f", {self.n_frames_missing_pred} without a prediction"
        if self.n_frames_missing_pred
        else ""
    )
    return (
        f"IDSW={self.id_switches}  IDF1={self.idf1:.4f} "
        f"(P={self.idp:.4f} R={self.idr:.4f}; matched-only "
        f"{self.idf1_matched:.4f})  "
        f"MT/PT/ML={self.mostly_tracked}/{self.partly_tracked}/{self.mostly_lost}  "
        f"Frag={self.fragmentations}  purity={self.mean_track_purity:.4f}  "
        f"cov={self.mean_gt_coverage:.4f}  "
        f"[{self.n_matched}/{self.n_gt_dets} GT dets matched, "
        f"{self.n_pred_dets} pred, {self.n_frames_compared} frames{missing}]"
    )

MOTMetrics

TrackEval-faithful CLEAR-MOT, Identity and HOTA metrics for one prediction.

Field names are snake_case versions of TrackEval's (mota = MOTA, det_a = DetA, ...). HOTA-family scalars are the mean over the 19 localization thresholds alpha = 0.05, ..., 0.95 (:data:sleap_nn.tracking.mot_metrics.HOTA_ALPHAS), as TrackEval reports them; the *_alpha lists hold the per-alpha values. Videos are combined as TrackEval combines sequences: counts summed, ratios recomputed.

Attributes:

Name	Type	Description
mota	float	Multi-object tracking accuracy, (TP - FP - IDSW) / n_gt.
motp	float	Mean similarity of the matched pairs (OKS / IoU, or the mapped distance similarity -- see :func:compute_mot_metrics).
moda	float	Detection accuracy, (TP - FP) / n_gt.
idsw	int	CLEAR-MOT identity switches.
frag	int	Fragmentations: tracked segments minus one, per trajectory.
mt	int	Ground-truth trajectories matched on more than 80% of their frames.
pt	int	Trajectories matched on 20-80% of their frames.
ml	int	Trajectories matched on less than 20% of their frames.
clr_tp	int	CLEAR-MOT true positives (matched pairs).
clr_fn	int	CLEAR-MOT false negatives (unmatched tracked ground truth).
clr_fp	int	CLEAR-MOT false positives (unmatched tracked predictions).
idf1	float	Identity F1 (Ristani et al.).
idp	float	Identity precision.
idr	float	Identity recall.
idtp	int	Identity true positives.
idfp	int	Identity false positives.
idfn	int	Identity false negatives.
hota	float	Higher-order tracking accuracy, sqrt(DetA * AssA).
det_a	float	Detection accuracy.
ass_a	float	Association accuracy.
loc_a	float	Localization accuracy (1 at an alpha with no true positive).
det_re	float	Detection recall.
det_pr	float	Detection precision.
ass_re	float	Association recall.
ass_pr	float	Association precision.
owta	float	Open-world tracking accuracy, sqrt(DetRe * AssA).
hota_0	float	HOTA at the first alpha, 0.05 (TrackEval's HOTA(0)).
loc_a_0	float	LocA at the first alpha, 0.05 (TrackEval's LocA(0)).
hota_alpha	List[float]	HOTA per alpha.
det_a_alpha	List[float]	DetA per alpha.
ass_a_alpha	List[float]	AssA per alpha.
loc_a_alpha	List[float]	LocA per alpha.
n_gt_dets	int	Tracked ground-truth detections scored (clr_tp + clr_fn).
n_gt_untracked	int	Untracked ground-truth detections in the scored frames (don't-care: matched against, never scored).
n_pred_dets	int	Tracked predicted detections scored (clr_tp + clr_fp): after untracked ones and those on untracked ground truth are removed.
n_pred_untracked	int	Predicted detections with no track, removed before scoring.
n_dont_care	int	Tracked predictions removed because they matched an untracked ground-truth detection.
n_frames	int	Ground-truth frames scored (every GT frame of every scored video).
n_frames_missing_pred	int	Of those, frames the prediction does not have.
n_videos	int	Videos scored.
match_method	str	"oks", "iou" or "distance".
match_threshold	float	The match threshold in match_method's units (OKS / IoU, or pixels for "distance").
oks_stddev	float	The OKS keypoint spread used ("oks" only).
global_identity	bool	Whether trajectories were keyed by name across videos.
notes	List[str]	Human-readable caveats raised while comparing.

Methods:

Name	Description
as_dict	Return the metrics as a plain, JSON-serializable dict.
summary	Return a one-line summary of the headline metrics.

Source code in sleap_nn/evaluation.py

@attrs.define(auto_attribs=True, slots=True)
class MOTMetrics:
    """TrackEval-faithful CLEAR-MOT, Identity and HOTA metrics for one prediction.

    Field names are snake_case versions of TrackEval's (``mota`` = ``MOTA``,
    ``det_a`` = ``DetA``, ...). HOTA-family scalars are the mean over the 19
    localization thresholds ``alpha = 0.05, ..., 0.95``
    (:data:`sleap_nn.tracking.mot_metrics.HOTA_ALPHAS`), as TrackEval reports
    them; the ``*_alpha`` lists hold the per-alpha values. Videos are combined
    as TrackEval combines sequences: counts summed, ratios recomputed.

    Attributes:
        mota: Multi-object tracking accuracy, ``(TP - FP - IDSW) / n_gt``.
        motp: Mean similarity of the matched pairs (OKS / IoU, or the mapped
            distance similarity -- see :func:`compute_mot_metrics`).
        moda: Detection accuracy, ``(TP - FP) / n_gt``.
        idsw: CLEAR-MOT identity switches.
        frag: Fragmentations: tracked segments minus one, per trajectory.
        mt: Ground-truth trajectories matched on more than 80% of their frames.
        pt: Trajectories matched on 20-80% of their frames.
        ml: Trajectories matched on less than 20% of their frames.
        clr_tp: CLEAR-MOT true positives (matched pairs).
        clr_fn: CLEAR-MOT false negatives (unmatched tracked ground truth).
        clr_fp: CLEAR-MOT false positives (unmatched tracked predictions).
        idf1: Identity F1 (Ristani et al.).
        idp: Identity precision.
        idr: Identity recall.
        idtp: Identity true positives.
        idfp: Identity false positives.
        idfn: Identity false negatives.
        hota: Higher-order tracking accuracy, ``sqrt(DetA * AssA)``.
        det_a: Detection accuracy.
        ass_a: Association accuracy.
        loc_a: Localization accuracy (1 at an alpha with no true positive).
        det_re: Detection recall.
        det_pr: Detection precision.
        ass_re: Association recall.
        ass_pr: Association precision.
        owta: Open-world tracking accuracy, ``sqrt(DetRe * AssA)``.
        hota_0: HOTA at the first alpha, 0.05 (TrackEval's ``HOTA(0)``).
        loc_a_0: LocA at the first alpha, 0.05 (TrackEval's ``LocA(0)``).
        hota_alpha: HOTA per alpha.
        det_a_alpha: DetA per alpha.
        ass_a_alpha: AssA per alpha.
        loc_a_alpha: LocA per alpha.
        n_gt_dets: Tracked ground-truth detections scored (``clr_tp + clr_fn``).
        n_gt_untracked: Untracked ground-truth detections in the scored frames
            (don't-care: matched against, never scored).
        n_pred_dets: Tracked predicted detections scored (``clr_tp + clr_fp``):
            after untracked ones and those on untracked ground truth are
            removed.
        n_pred_untracked: Predicted detections with no track, removed before
            scoring.
        n_dont_care: Tracked predictions removed because they matched an
            untracked ground-truth detection.
        n_frames: Ground-truth frames scored (every GT frame of every scored
            video).
        n_frames_missing_pred: Of those, frames the prediction does not have.
        n_videos: Videos scored.
        match_method: ``"oks"``, ``"iou"`` or ``"distance"``.
        match_threshold: The match threshold in ``match_method``'s units (OKS /
            IoU, or pixels for ``"distance"``).
        oks_stddev: The OKS keypoint spread used (``"oks"`` only).
        global_identity: Whether trajectories were keyed by name across videos.
        notes: Human-readable caveats raised while comparing.
    """

    # CLEAR-MOT.
    mota: float = float("nan")
    motp: float = float("nan")
    moda: float = float("nan")
    idsw: int = 0
    frag: int = 0
    mt: int = 0
    pt: int = 0
    ml: int = 0
    clr_tp: int = 0
    clr_fn: int = 0
    clr_fp: int = 0
    # Identity.
    idf1: float = float("nan")
    idp: float = float("nan")
    idr: float = float("nan")
    idtp: int = 0
    idfp: int = 0
    idfn: int = 0
    # HOTA (means over alphas).
    hota: float = float("nan")
    det_a: float = float("nan")
    ass_a: float = float("nan")
    loc_a: float = float("nan")
    det_re: float = float("nan")
    det_pr: float = float("nan")
    ass_re: float = float("nan")
    ass_pr: float = float("nan")
    owta: float = float("nan")
    hota_0: float = float("nan")
    loc_a_0: float = float("nan")
    hota_alpha: List[float] = attrs.field(factory=list)
    det_a_alpha: List[float] = attrs.field(factory=list)
    ass_a_alpha: List[float] = attrs.field(factory=list)
    loc_a_alpha: List[float] = attrs.field(factory=list)
    # Detection accounting.
    n_gt_dets: int = 0
    n_gt_untracked: int = 0
    n_pred_dets: int = 0
    n_pred_untracked: int = 0
    n_dont_care: int = 0
    n_frames: int = 0
    n_frames_missing_pred: int = 0
    n_videos: int = 0
    # How the comparison was made.
    match_method: str = ""
    match_threshold: float = float("nan")
    oks_stddev: float = 0.025
    global_identity: bool = False
    notes: List[str] = attrs.field(factory=list)

    def as_dict(self) -> Dict[str, Any]:
        """Return the metrics as a plain, JSON-serializable dict."""
        return attrs.asdict(self)

    def summary(self) -> str:
        """Return a one-line summary of the headline metrics."""
        return (
            f"MOTA={self.mota:.4f} MOTP={self.motp:.4f} IDSW={self.idsw} "
            f"Frag={self.frag} MT/PT/ML={self.mt}/{self.pt}/{self.ml}  "
            f"IDF1={self.idf1:.4f} (P={self.idp:.4f} R={self.idr:.4f})  "
            f"HOTA={self.hota:.4f} (DetA={self.det_a:.4f} AssA={self.ass_a:.4f} "
            f"LocA={self.loc_a:.4f})  "
            f"[{self.clr_tp} TP / {self.clr_fn} FN / {self.clr_fp} FP over "
            f"{self.n_gt_dets} GT dets, {self.n_frames} frames]"
        )

as_dict()

Return the metrics as a plain, JSON-serializable dict.

Source code in sleap_nn/evaluation.py

def as_dict(self) -> Dict[str, Any]:
    """Return the metrics as a plain, JSON-serializable dict."""
    return attrs.asdict(self)

summary()

Return a one-line summary of the headline metrics.

Source code in sleap_nn/evaluation.py

def summary(self) -> str:
    """Return a one-line summary of the headline metrics."""
    return (
        f"MOTA={self.mota:.4f} MOTP={self.motp:.4f} IDSW={self.idsw} "
        f"Frag={self.frag} MT/PT/ML={self.mt}/{self.pt}/{self.ml}  "
        f"IDF1={self.idf1:.4f} (P={self.idp:.4f} R={self.idr:.4f})  "
        f"HOTA={self.hota:.4f} (DetA={self.det_a:.4f} AssA={self.ass_a:.4f} "
        f"LocA={self.loc_a:.4f})  "
        f"[{self.clr_tp} TP / {self.clr_fn} FN / {self.clr_fp} FP over "
        f"{self.n_gt_dets} GT dets, {self.n_frames} frames]"
    )

MatchInstance

Class to have a new structure for sio.Instance object.

Source code in sleap_nn/evaluation.py

@attrs.define(auto_attribs=True, slots=True)
class MatchInstance:
    """Class to have a new structure for sio.Instance object."""

    instance: sio.Instance
    frame_idx: int
    video_path: str

NothingToScoreError

Bases: ValueError

Raised by run_identity_evaluation(raise_if_empty=True) when nothing is scoreable.

That is: no tracked predicted detection on the carrier, no frame in common with the ground truth, or no tracked ground-truth detection in the frames compared. The message says which, and what to do about it.

Source code in sleap_nn/evaluation.py

class NothingToScoreError(ValueError):
    """Raised by ``run_identity_evaluation(raise_if_empty=True)`` when nothing is scoreable.

    That is: no tracked predicted detection on the carrier, no frame in common
    with the ground truth, or no tracked ground-truth detection in the frames
    compared. The message says which, and what to do about it.
    """

bbox_frame_pairs(labels_gt, labels_pr, class_agnostic=None)

Build :func:compute_bbox_metrics frames from GT and predicted labels.

Every GT frame with at least one user box is evaluated (its predictions -- the PredictedBoundingBoxes of the matching predicted frame -- may be none, making all its boxes false negatives). Predicted frames with no GT frame are not scored (unlabeled frames carry no ground truth).

Parameters:

Name	Type	Description	Default
labels_gt	Labels	Ground truth with user boxes on LabeledFrame.bboxes.	required
labels_pr	Labels	Predictions with PredictedBoundingBoxes.	required
class_agnostic	Optional[bool]	Score every box as one class. None (default) decides from the data: class-agnostic when the GT and the predictions each carry at most one category name -- a single-class detector maps every box to its one class whatever it is called, so an uncategorized GT ("object", e.g. boxes derived from poses) and predictions named after the model's one class ("mouse") are the same class.	None

Returns:

Type	Description
Tuple[List[Dict[str, ndarray]], List[str]]	(frames, categories).

Source code in sleap_nn/evaluation.py

def bbox_frame_pairs(
    labels_gt: sio.Labels,
    labels_pr: sio.Labels,
    class_agnostic: Optional[bool] = None,
) -> Tuple[List[Dict[str, np.ndarray]], List[str]]:
    """Build :func:`compute_bbox_metrics` frames from GT and predicted labels.

    Every GT frame with at least one user box is evaluated (its predictions --
    the ``PredictedBoundingBox``es of the matching predicted frame -- may be none,
    making all its boxes false negatives). Predicted frames with no GT frame are
    not scored (unlabeled frames carry no ground truth).

    Args:
        labels_gt: Ground truth with user boxes on ``LabeledFrame.bboxes``.
        labels_pr: Predictions with ``PredictedBoundingBox``es.
        class_agnostic: Score every box as one class. ``None`` (default) decides
            from the data: class-agnostic when the GT and the predictions each
            carry at most one category name -- a single-class detector maps every
            box to its one class whatever it is called, so an uncategorized GT
            (``"object"``, e.g. boxes derived from poses) and predictions named
            after the model's one class (``"mouse"``) are the same class.

    Returns:
        ``(frames, categories)``.
    """
    from sleap_nn.data.bboxes import bbox_category_name, frame_user_bboxes

    categories: List[str] = []
    match_result = labels_gt.match(labels_pr)
    # `sio.Video` defines `__len__` but not `__bool__`: a matched video whose
    # shape is unknown is falsy, so test against None explicitly.
    gt_to_pr = {
        id(v_gt): v_pr
        for v_pr, v_gt in match_result.video_map.items()
        if v_gt is not None
    }
    if class_agnostic is None:
        gt_names = {
            bbox_category_name(b) for lf in labels_gt for b in frame_user_bboxes(lf)
        }
        pr_names = {
            bbox_category_name(b)
            for lf in labels_pr
            for b in lf.bboxes
            if b.is_predicted
        }
        class_agnostic = len(gt_names) <= 1 and len(pr_names) <= 1
    frames = []
    for lf in labels_gt.labeled_frames:
        gt_boxes = frame_user_bboxes(lf)
        if not gt_boxes:
            continue
        pred_boxes = []
        video_pr = gt_to_pr.get(id(lf.video))
        if video_pr is not None:
            for lf_pr in labels_pr.find(video_pr, frame_idx=lf.frame_idx):
                pred_boxes.extend(b for b in lf_pr.bboxes if b.is_predicted)
        gt_b, gt_c, _ = _frame_bbox_arrays_by_name(gt_boxes, categories)
        pr_b, pr_c, pr_s = _frame_bbox_arrays_by_name(pred_boxes, categories)
        if class_agnostic:
            gt_c, pr_c = np.zeros_like(gt_c), np.zeros_like(pr_c)
        frames.append(
            {
                "video_idx": labels_gt.videos.index(lf.video),
                "frame_idx": lf.frame_idx,
                "gt_bboxes": gt_b,
                "gt_classes": gt_c,
                "pred_bboxes": pr_b,
                "pred_scores": pr_s,
                "pred_classes": pr_c,
            }
        )
    if class_agnostic:
        categories = ["object"]
    return frames, categories

bbox_iou_matrix(a, b)

Pairwise IoU of (n, 4) and (m, 4) xyxy boxes -> (n, m).

Source code in sleap_nn/evaluation.py

def bbox_iou_matrix(a: np.ndarray, b: np.ndarray) -> np.ndarray:
    """Pairwise IoU of ``(n, 4)`` and ``(m, 4)`` xyxy boxes -> ``(n, m)``."""
    a = np.asarray(a, dtype=np.float64).reshape(-1, 4)
    b = np.asarray(b, dtype=np.float64).reshape(-1, 4)
    if a.shape[0] == 0 or b.shape[0] == 0:
        return np.zeros((a.shape[0], b.shape[0]))
    x1 = np.maximum(a[:, None, 0], b[None, :, 0])
    y1 = np.maximum(a[:, None, 1], b[None, :, 1])
    x2 = np.minimum(a[:, None, 2], b[None, :, 2])
    y2 = np.minimum(a[:, None, 3], b[None, :, 3])
    inter = np.clip(x2 - x1, 0, None) * np.clip(y2 - y1, 0, None)
    area_a = (a[:, 2] - a[:, 0]) * (a[:, 3] - a[:, 1])
    area_b = (b[:, 2] - b[:, 0]) * (b[:, 3] - b[:, 1])
    union = area_a[:, None] + area_b[None, :] - inter
    return np.where(union > 0, inter / np.maximum(union, 1e-12), 0.0)

compare_identity_metrics(arms)

Render a Markdown comparison table across tracker arms.

Parameters:

Name	Type	Description	Default
arms	Dict[str, IdentityMetrics]	Mapping of arm name (e.g. "geometry", "fused") to its :class:IdentityMetrics.	required

Returns:

Type	Description
str	A Markdown table, one row per arm, with a footer naming the direction of improvement for each column.

Source code in sleap_nn/evaluation.py

def compare_identity_metrics(arms: Dict[str, IdentityMetrics]) -> str:
    """Render a Markdown comparison table across tracker arms.

    Args:
        arms: Mapping of arm name (e.g. ``"geometry"``, ``"fused"``) to its
            :class:`IdentityMetrics`.

    Returns:
        A Markdown table, one row per arm, with a footer naming the direction of
        improvement for each column.
    """
    columns = [
        ("IDSW", "id_switches", "{:d}"),
        ("IDF1", "idf1", "{:.4f}"),
        ("IDF1m", "idf1_matched", "{:.4f}"),
        ("purity", "mean_track_purity", "{:.4f}"),
        ("cov", "mean_gt_coverage", "{:.4f}"),
        ("MT", "mostly_tracked", "{:d}"),
        ("ML", "mostly_lost", "{:d}"),
        ("Frag", "fragmentations", "{:d}"),
        ("matched", "n_matched", "{:d}"),
    ]
    lines = [
        "| arm | " + " | ".join(label for label, _attr, _fmt in columns) + " |",
        "|---|" + "|".join("---" for _ in columns) + "|",
    ]
    for name, metrics in arms.items():
        cells = []
        for _label, attr, fmt in columns:
            value = getattr(metrics, attr)
            cells.append(
                "n/a"
                if value is None or (isinstance(value, float) and np.isnan(value))
                else fmt.format(value)
            )
        lines.append(f"| {name} | " + " | ".join(cells) + " |")
    lines.append("")
    lines.append(
        "Lower is better: IDSW, ML, Frag. Higher is better: IDF1, IDF1m, purity, "
        "cov, MT. IDF1m = IDF1 over matched detections only (detector errors "
        "factored out)."
    )
    return "\n".join(lines)

compute_bbox_metrics(frames, num_classes=None, iou_thresholds=BBOX_IOU_THRESHOLDS, max_detections=100)

COCO-style detection metrics over a set of frames.

Parameters:

Name	Type	Description	Default
frames	List[Dict[str, ndarray]]	One dict per evaluated frame with gt_bboxes (n, 4), gt_classes (n,), pred_bboxes (m, 4), pred_scores (m,) and pred_classes (m,) (xyxy, any consistent pixel space).	required
num_classes	Optional[int]	Number of classes (inferred from the class ids if None).	None
iou_thresholds	ndarray	IoU thresholds averaged by mAP / AR.	BBOX_IOU_THRESHOLDS
max_detections	int	Highest-scoring detections kept per frame (COCO's 100).	100

Returns:

Type	Description
Dict[str, Any]	A dict with bbox_mAP (AP@[.5:.95]), bbox_AP50, bbox_AP75, bbox_AR, the IoU-0.5 operating point (bbox_precision, bbox_recall, bbox_f1 at bbox_f1_threshold, bbox_mean_iou, bbox_n_tp/fp/fn), n_frames / n_gt / n_pred, per-class ap_per_class ((C, T)) and the IoU-0.5 pr_curve (scores, precision, recall).

Source code in sleap_nn/evaluation.py

def compute_bbox_metrics(
    frames: List[Dict[str, np.ndarray]],
    num_classes: Optional[int] = None,
    iou_thresholds: np.ndarray = BBOX_IOU_THRESHOLDS,
    max_detections: int = 100,
) -> Dict[str, Any]:
    """COCO-style detection metrics over a set of frames.

    Args:
        frames: One dict per evaluated frame with ``gt_bboxes (n, 4)``,
            ``gt_classes (n,)``, ``pred_bboxes (m, 4)``, ``pred_scores (m,)`` and
            ``pred_classes (m,)`` (xyxy, any consistent pixel space).
        num_classes: Number of classes (inferred from the class ids if ``None``).
        iou_thresholds: IoU thresholds averaged by mAP / AR.
        max_detections: Highest-scoring detections kept per frame (COCO's 100).

    Returns:
        A dict with ``bbox_mAP`` (AP@[.5:.95]), ``bbox_AP50``, ``bbox_AP75``,
        ``bbox_AR``, the IoU-0.5 operating point (``bbox_precision``,
        ``bbox_recall``, ``bbox_f1`` at ``bbox_f1_threshold``, ``bbox_mean_iou``,
        ``bbox_n_tp/fp/fn``), ``n_frames`` / ``n_gt`` / ``n_pred``, per-class
        ``ap_per_class`` (``(C, T)``) and the IoU-0.5 ``pr_curve`` (scores,
        precision, recall).
    """
    recall_grid = np.linspace(0.0, 1.0, 101)
    iou_thresholds = np.asarray(iou_thresholds, dtype=float)
    if num_classes is None:
        ids = [
            int(c)
            for f in frames
            for c in list(f["gt_classes"]) + list(f["pred_classes"])
        ]
        num_classes = max(ids) + 1 if ids else 1

    # Per (class, threshold): detection scores, TP flags, matched IoUs.
    scores = [[[] for _ in iou_thresholds] for _ in range(num_classes)]
    tps = [[[] for _ in iou_thresholds] for _ in range(num_classes)]
    ious = [[[] for _ in iou_thresholds] for _ in range(num_classes)]
    n_gt_per_class = np.zeros(num_classes, dtype=np.int64)
    n_pred = 0
    for f in frames:
        gt_b = np.asarray(f["gt_bboxes"], dtype=np.float64).reshape(-1, 4)
        gt_c = np.asarray(f["gt_classes"], dtype=np.int64).reshape(-1)
        pr_b = np.asarray(f["pred_bboxes"], dtype=np.float64).reshape(-1, 4)
        pr_s = np.asarray(f["pred_scores"], dtype=np.float64).reshape(-1)
        pr_c = np.asarray(f["pred_classes"], dtype=np.int64).reshape(-1)
        keep = np.isfinite(pr_s) & np.isfinite(pr_b).all(axis=1)
        pr_b, pr_s, pr_c = pr_b[keep], pr_s[keep], pr_c[keep]
        order = np.argsort(-pr_s, kind="mergesort")[:max_detections]
        pr_b, pr_s, pr_c = pr_b[order], pr_s[order], pr_c[order]
        n_pred += len(pr_s)
        for c in range(num_classes):
            g = gt_b[gt_c == c]
            p_mask = pr_c == c
            p, s = pr_b[p_mask], pr_s[p_mask]
            n_gt_per_class[c] += len(g)
            iou = bbox_iou_matrix(p, g)
            for t, thr in enumerate(iou_thresholds):
                gt_idx, m_iou = _greedy_match_bboxes(iou, thr)
                scores[c][t].append(s)
                tps[c][t].append(gt_idx >= 0)
                ious[c][t].append(m_iou)

    ap = np.full((num_classes, len(iou_thresholds)), np.nan)
    ar = np.full((num_classes, len(iou_thresholds)), np.nan)
    for c in range(num_classes):
        for t in range(len(iou_thresholds)):
            s = np.concatenate(scores[c][t]) if scores[c][t] else np.zeros(0)
            tp = np.concatenate(tps[c][t]) if tps[c][t] else np.zeros(0, bool)
            ap[c, t], ar[c, t] = _ap_from_pr(s, tp, n_gt_per_class[c], recall_grid)

    def _nanmean(x):
        x = np.asarray(x, dtype=float)
        return float(np.nanmean(x)) if np.isfinite(x).any() else float("nan")

    t50 = int(np.argmin(np.abs(iou_thresholds - 0.5)))
    t75 = int(np.argmin(np.abs(iou_thresholds - 0.75)))

    # IoU-0.5 operating point over all classes: the F1-maximizing score threshold.
    s50 = np.concatenate([np.concatenate(scores[c][t50]) for c in range(num_classes)])
    tp50 = np.concatenate([np.concatenate(tps[c][t50]) for c in range(num_classes)])
    iou50 = np.concatenate([np.concatenate(ious[c][t50]) for c in range(num_classes)])
    n_gt = int(n_gt_per_class.sum())
    order = np.argsort(-s50, kind="mergesort")
    s50, tp50, iou50 = s50[order], tp50[order], iou50[order]
    cum_tp = np.cumsum(tp50)
    cum_fp = np.cumsum(~tp50)
    precision = cum_tp / np.maximum(cum_tp + cum_fp, 1)
    recall = cum_tp / max(n_gt, 1)
    f1 = 2 * precision * recall / np.maximum(precision + recall, 1e-12)
    if f1.size:
        best = int(np.argmax(f1))
        n_tp, n_fp = int(cum_tp[best]), int(cum_fp[best])
        op = dict(
            bbox_precision=float(precision[best]),
            bbox_recall=float(recall[best]) if n_gt else float("nan"),
            bbox_f1=float(f1[best]),
            bbox_f1_threshold=float(s50[best]),
            bbox_mean_iou=(
                float(iou50[: best + 1][tp50[: best + 1]].mean()) if n_tp else 0.0
            ),
            bbox_n_tp=n_tp,
            bbox_n_fp=n_fp,
            bbox_n_fn=n_gt - n_tp,
        )
    else:
        op = dict(
            bbox_precision=float("nan"),
            bbox_recall=0.0 if n_gt else float("nan"),
            bbox_f1=0.0,
            bbox_f1_threshold=float("nan"),
            bbox_mean_iou=float("nan"),
            bbox_n_tp=0,
            bbox_n_fp=0,
            bbox_n_fn=n_gt,
        )

    return {
        "bbox_mAP": _nanmean(ap),
        "bbox_AP50": _nanmean(ap[:, t50]),
        "bbox_AP75": _nanmean(ap[:, t75]),
        "bbox_AR": _nanmean(ar),
        **op,
        "n_frames": len(frames),
        "n_gt": n_gt,
        "n_pred": int(n_pred),
        "iou_thresholds": iou_thresholds,
        "ap_per_class": ap,
        "ar_per_class": ar,
        "pr_curve": {"scores": s50, "precision": precision, "recall": recall},
    }

compute_distance_match_score(points_gt, points_pr, pixel_threshold=50.0)

Compute a pixel-distance-based match score for degenerate-scale GT instances.

Used as a fallback for GT instances whose visible-keypoint bounding box has zero area (see _DEGENERATE_AREA_EPS), where compute_oks degenerates into a strict equality test. Mirrors the pixel-distance matching already used for centroid-only models (match_method="centroid"), but restricted to the nodes that are visible in both the ground truth and predicted instance.

Parameters:

Name	Type	Description	Default
points_gt	ndarray	Ground truth instances of shape (n_gt, n_nodes, n_ed).	required
points_pr	ndarray	Predicted instances of shape (n_pr, n_nodes, n_ed).	required
pixel_threshold	float	Distance (in pixels) at which the score reaches 0.	50.0

Returns:

Type	Description
ndarray	Match scores of shape (n_gt, n_pr) in the range [0, 1], with 1.0 denoting a perfect match and 0.0 denoting no jointly-visible nodes or a mean distance at or beyond pixel_threshold. Comparable in scale to compute_oks's output, so the two can be combined and thresholded uniformly.

Source code in sleap_nn/evaluation.py

def compute_distance_match_score(
    points_gt: np.ndarray,
    points_pr: np.ndarray,
    pixel_threshold: float = 50.0,
) -> np.ndarray:
    """Compute a pixel-distance-based match score for degenerate-scale GT instances.

    Used as a fallback for GT instances whose visible-keypoint bounding box has zero
    area (see `_DEGENERATE_AREA_EPS`), where `compute_oks` degenerates into a strict
    equality test. Mirrors the pixel-distance matching already used for centroid-only
    models (`match_method="centroid"`), but restricted to the nodes that are visible
    in both the ground truth and predicted instance.

    Args:
        points_gt: Ground truth instances of shape (n_gt, n_nodes, n_ed).
        points_pr: Predicted instances of shape (n_pr, n_nodes, n_ed).
        pixel_threshold: Distance (in pixels) at which the score reaches 0.

    Returns:
        Match scores of shape (n_gt, n_pr) in the range [0, 1], with 1.0 denoting a
        perfect match and 0.0 denoting no jointly-visible nodes or a mean distance at
        or beyond `pixel_threshold`. Comparable in scale to `compute_oks`'s output, so
        the two can be combined and thresholded uniformly.
    """
    if points_gt.ndim == 2:
        points_gt = np.expand_dims(points_gt, axis=0)
    if points_pr.ndim == 2:
        points_pr = np.expand_dims(points_pr, axis=0)

    n_gt = points_gt.shape[0]
    n_pr = points_pr.shape[0]
    scores = np.zeros((n_gt, n_pr))
    for i in range(n_gt):
        for j in range(n_pr):
            jointly_visible = ~np.isnan(points_gt[i]).any(axis=-1) & ~np.isnan(
                points_pr[j]
            ).any(axis=-1)
            if not jointly_visible.any():
                continue
            dists = np.linalg.norm(
                points_gt[i, jointly_visible] - points_pr[j, jointly_visible], axis=-1
            )
            mean_dist = float(np.mean(dists))
            scores[i, j] = max(0.0, 1.0 - mean_dist / pixel_threshold)
    return scores

compute_dists(positive_pairs)

Compute Euclidean distances between matched pairs of instances.

Parameters:

Name	Type	Description	Default
positive_pairs	List[Tuple[Instance, PredictedInstance, Any]]	A list of tuples of the form (instance_gt, instance_pr, _) containing the matched pair of instances.	required

Returns:

Type	Description
Dict[str, Union[ndarray, List[int], List[str]]]	A dictionary with the following keys: dists: An array of pairwise distances of shape (n_positive_pairs, n_nodes) frame_idxs: A list of frame indices corresponding to the dists video_paths: A list of video paths corresponding to the dists

Source code in sleap_nn/evaluation.py

def compute_dists(
    positive_pairs: List[Tuple[sio.Instance, sio.PredictedInstance, Any]],
) -> Dict[str, Union[np.ndarray, List[int], List[str]]]:
    """Compute Euclidean distances between matched pairs of instances.

    Args:
        positive_pairs: A list of tuples of the form `(instance_gt, instance_pr, _)`
            containing the matched pair of instances.

    Returns:
        A dictionary with the following keys:
            dists: An array of pairwise distances of shape `(n_positive_pairs, n_nodes)`
            frame_idxs: A list of frame indices corresponding to the `dists`
            video_paths: A list of video paths corresponding to the `dists`
    """
    dists = []
    frame_idxs = []
    video_paths = []
    for instance_gt, instance_pr, _ in positive_pairs:
        points_gt = instance_gt.instance.numpy()
        points_pr = instance_pr.instance.numpy()

        dists.append(np.linalg.norm(points_pr - points_gt, axis=-1))
        frame_idxs.append(instance_gt.frame_idx)
        video_paths.append(instance_gt.video_path)

    dists = np.array(dists)

    # Bundle everything into a dictionary
    dists_dict = {
        "dists": dists,
        "frame_idxs": frame_idxs,
        "video_paths": video_paths,
    }

    return dists_dict

compute_gt_centroids(instance_gt_points, anchor_ind=None, method=None, fallback=None)

Compute ground-truth centroids for a numpy array of instance keypoints.

A thin numpy-in/numpy-out wrapper around :func:sleap_nn.data.instance_centroids.generate_centroids, which is the single definition of what a centroid MEANS (see also #586). It used to be a hand-written numpy mirror; delegating removes the drift that sleap_nn.inference.centroid_convert warns about — evaluation now cannot disagree with the trained target about the centroid, including for the bbox_center / geometric_median methods.

Parameters:

Name	Type	Description	Default
instance_gt_points	ndarray	Ground-truth keypoints of shape (n_instances, n_nodes, 2) or (n_nodes, 2). Missing/occluded nodes are NaN.	required
anchor_ind	Optional[int]	Index of the node to use as the anchor. Required by (and only used by) method="anchor"; when that node is NaN for an instance, that instance falls back to fallback.	None
method	Optional[str]	One of sleap_nn.data.instance_centroids.CENTROID_METHODS. None (default) infers it from anchor_ind — the historical behavior: the anchor node when given, else the NaN-ignoring mean of visible nodes.	None
fallback	Optional[str]	Reduce method for a missing anchor. None means "center_of_mass".	None

Returns:

Type	Description
ndarray	Centroids of shape (n_instances, 2) (or (2,) for a single instance input), reducing the node axis.

Source code in sleap_nn/evaluation.py

def compute_gt_centroids(
    instance_gt_points: np.ndarray,
    anchor_ind: Optional[int] = None,
    method: Optional[str] = None,
    fallback: Optional[str] = None,
) -> np.ndarray:
    """Compute ground-truth centroids for a numpy array of instance keypoints.

    A thin numpy-in/numpy-out wrapper around
    :func:`sleap_nn.data.instance_centroids.generate_centroids`, which is the
    single definition of what a centroid MEANS (see also #586). It used to be a
    hand-written numpy mirror; delegating removes the drift that
    ``sleap_nn.inference.centroid_convert`` warns about — evaluation now cannot
    disagree with the trained target about the centroid, including for the
    ``bbox_center`` / ``geometric_median`` methods.

    Args:
        instance_gt_points: Ground-truth keypoints of shape ``(n_instances,
            n_nodes, 2)`` or ``(n_nodes, 2)``. Missing/occluded nodes are NaN.
        anchor_ind: Index of the node to use as the anchor. Required by (and only
            used by) ``method="anchor"``; when that node is NaN for an instance,
            that instance falls back to ``fallback``.
        method: One of ``sleap_nn.data.instance_centroids.CENTROID_METHODS``.
            ``None`` (default) infers it from ``anchor_ind`` — the historical
            behavior: the anchor node when given, else the NaN-ignoring mean of
            visible nodes.
        fallback: Reduce method for a missing anchor. ``None`` means
            ``"center_of_mass"``.

    Returns:
        Centroids of shape ``(n_instances, 2)`` (or ``(2,)`` for a single
        instance input), reducing the node axis.
    """
    points = np.asarray(instance_gt_points, dtype=np.float64)
    centroids = generate_centroids(
        torch.from_numpy(points),
        anchor_ind=anchor_ind,
        method=method,
        fallback=fallback,
    )
    return centroids.numpy()

compute_instance_area(points)

Compute the area of the bounding box of a set of keypoints.

Parameters:

Name	Type	Description	Default
points	ndarray	A numpy array of coordinates.	required

Returns:

Type	Description
ndarray	The area of the bounding box of the points.

Source code in sleap_nn/evaluation.py

def compute_instance_area(points: np.ndarray) -> np.ndarray:
    """Compute the area of the bounding box of a set of keypoints.

    Args:
        points: A numpy array of coordinates.

    Returns:
        The area of the bounding box of the points.
    """
    if points.ndim == 2:
        points = np.expand_dims(points, axis=0)

    min_pt = np.nanmin(points, axis=-2)
    max_pt = np.nanmax(points, axis=-2)

    return np.prod(max_pt - min_pt, axis=-1)

compute_mot_metrics(gt_labels, pred_labels, carrier='pose', *, match_threshold=None, oks_stddev=0.025, user_labels_only=False, global_identity=False)

Score a tracked prediction with TrackEval-faithful CLEAR-MOT, IDF1 and HOTA.

Reproduces TrackEval 1.3.0 (the reference HOTA / MOTChallenge code) on the same detections, matcher and conventions as :func:identity_metrics:

• Matching. OKS for "pose" (pixel distance for single-node skeletons), mask IoU for "mask"; tracks are read off each detection or through its pose <-> mask link (:func:_identity_tracks). CLEAR-MOT and IDF1 match at similarity >= match_threshold; HOTA scores over its 19 thresholds. For "distance" the pixel distance d becomes the similarity max(0, 1 - d / (2 * match_threshold)), so the 0.5 cut is exactly match_threshold pixels (50 px by default) and MOTP / LocA are in that unit.
• Sequences. Each video is one sequence; videos are combined as TrackEval combines sequences (counts summed, HOTA's AssA / LocA weighted by true positives). With global_identity all videos form ONE sequence keyed by track name, with CLEAR-MOT's frame-to-frame state reset at each video boundary (no ID switch, continuity bonus or fragment across it); keeping a name across videos is then scored by IDF1 and HOTA's AssA.
• Frames. Every ground-truth frame of each video the prediction covers is a timestep; a frame missing from the prediction scores as misses.
• Untracked ground truth is don't-care, as MOTChallenge treats distractors: one Hungarian matching against ALL ground truth at the match threshold, predictions matched to an untracked ground truth removed, then the untracked ground truth removed.
• Untracked predictions are removed before scoring and counted in n_pred_untracked.
• Corrected predictions are left out of the ground truth, before the don't-care matching: a prediction a user label on the same frame replaces (a pose corrected in the GUI) is not ground truth, the user copy is (:func:_superseded_ground_truth, shared with :func:identity_metrics).

Parameters:

Name	Type	Description	Default
gt_labels	Labels	Ground truth with track set on the detections to score.	required
pred_labels	Labels	Prediction with track set by the tracker under test.	required
carrier	str	"pose" (instances) or "mask" (segmentation masks).	'pose'
match_threshold	Optional[float]	Minimum OKS / IoU for a pair to match, or for single-node skeletons the maximum centroid distance in PIXELS. None (default): 0.5 for OKS / IoU, 50 px for distance (IDENTITY_MATCH_THRESHOLDS).	None
oks_stddev	float	Per-keypoint OKS spread (:func:compute_oks), a finite number > 0. Default 0.025, the same as :func:identity_metrics.	0.025
user_labels_only	bool	Drop model output from the GROUND-TRUTH side (see :func:identity_metrics for why this is off by default).	False
global_identity	bool	Key trajectories by track name across videos.	False

Returns:

Name	Type	Description
A	MOTMetrics	class:MOTMetrics. When the files share no frame the metrics are NaN and notes says so -- check n_frames before quoting a number.

Raises:

Type	Description
ValueError	If the carrier is unknown, oks_stddev is not a finite number greater than 0, a distance threshold is not positive, or a track occurs on two scored detections of one frame (TrackEval refuses ambiguous identities too).

Source code in sleap_nn/evaluation.py

def compute_mot_metrics(
    gt_labels: sio.Labels,
    pred_labels: sio.Labels,
    carrier: str = "pose",
    *,
    match_threshold: Optional[float] = None,
    oks_stddev: float = 0.025,
    user_labels_only: bool = False,
    global_identity: bool = False,
) -> MOTMetrics:
    """Score a tracked prediction with TrackEval-faithful CLEAR-MOT, IDF1 and HOTA.

    Reproduces TrackEval 1.3.0 (the reference HOTA / MOTChallenge code) on the
    same detections, matcher and conventions as :func:`identity_metrics`:

    - **Matching.** OKS for ``"pose"`` (pixel distance for single-node
      skeletons), mask IoU for ``"mask"``; tracks are read off each detection
      or through its pose <-> mask link (:func:`_identity_tracks`). CLEAR-MOT
      and IDF1 match at ``similarity >= match_threshold``; HOTA scores over its
      19 thresholds. For ``"distance"`` the pixel distance ``d`` becomes the
      similarity ``max(0, 1 - d / (2 * match_threshold))``, so the 0.5 cut is
      exactly ``match_threshold`` pixels (50 px by default) and MOTP / LocA are
      in that unit.
    - **Sequences.** Each video is one sequence; videos are combined as
      TrackEval combines sequences (counts summed, HOTA's AssA / LocA weighted
      by true positives). With ``global_identity`` all videos form ONE
      sequence keyed by track name, with CLEAR-MOT's frame-to-frame state reset
      at each video boundary (no ID switch, continuity bonus or fragment
      across it); keeping a name across videos is then scored by IDF1 and
      HOTA's AssA.
    - **Frames.** Every ground-truth frame of each video the prediction covers
      is a timestep; a frame missing from the prediction scores as misses.
    - **Untracked ground truth is don't-care**, as MOTChallenge treats
      distractors: one Hungarian matching against ALL ground truth at the match
      threshold, predictions matched to an untracked ground truth removed, then
      the untracked ground truth removed.
    - **Untracked predictions** are removed before scoring and counted in
      ``n_pred_untracked``.
    - **Corrected predictions are left out of the ground truth**, before the
      don't-care matching: a prediction a user label on the same frame replaces
      (a pose corrected in the GUI) is not ground truth, the user copy is
      (:func:`_superseded_ground_truth`, shared with :func:`identity_metrics`).

    Args:
        gt_labels: Ground truth with ``track`` set on the detections to score.
        pred_labels: Prediction with ``track`` set by the tracker under test.
        carrier: ``"pose"`` (instances) or ``"mask"`` (segmentation masks).
        match_threshold: Minimum OKS / IoU for a pair to match, or for
            single-node skeletons the maximum centroid distance in PIXELS.
            ``None`` (default): 0.5 for OKS / IoU, 50 px for distance
            (``IDENTITY_MATCH_THRESHOLDS``).
        oks_stddev: Per-keypoint OKS spread (:func:`compute_oks`), a finite
            number > 0. Default 0.025, the same as :func:`identity_metrics`.
        user_labels_only: Drop model output from the GROUND-TRUTH side (see
            :func:`identity_metrics` for why this is off by default).
        global_identity: Key trajectories by track name across videos.

    Returns:
        A :class:`MOTMetrics`. When the files share no frame the metrics are
        NaN and ``notes`` says so -- check ``n_frames`` before quoting a number.

    Raises:
        ValueError: If the carrier is unknown, ``oks_stddev`` is not a finite
            number greater than 0, a distance threshold is not positive, or a
            track occurs on two scored detections of one frame (TrackEval
            refuses ambiguous identities too).
    """
    from collections import defaultdict

    from sleap_nn.tracking import mot_metrics as mm

    carrier = _validate_carrier(carrier)
    oks_stddev = _check_oks_stddev(oks_stddev)
    method = _identity_match_method(gt_labels, pred_labels, carrier)
    threshold = (
        IDENTITY_MATCH_THRESHOLDS[method]
        if match_threshold is None
        else float(match_threshold)
    )
    if method == "distance" and not threshold > 0:
        raise ValueError(
            f"match_threshold must be a positive pixel distance for single-node "
            f"skeletons, got {threshold!r}."
        )
    # CLEAR-MOT / IDF1 threshold on the similarity scale.
    sim_threshold = 0.5 if method == "distance" else threshold
    metrics = MOTMetrics(
        match_method=method,
        match_threshold=threshold,
        oks_stddev=oks_stddev,
        global_identity=global_identity,
    )

    gt_by_key, pred_by_key, gt_keys, scored_videos, notes = _align_identity_frames(
        gt_labels, pred_labels
    )
    metrics.notes.extend(notes)
    if not scored_videos:
        return metrics

    # One sequence per video, or one overall under global identities.
    frames_by_seq: Dict[str, List[Tuple[np.ndarray, np.ndarray, np.ndarray]]] = (
        defaultdict(list)
    )
    seq_resets: Dict[str, List[int]] = defaultdict(list)
    gt_ids: Dict[Tuple[str, str], int] = {}
    pred_ids: Dict[Tuple[str, str], int] = {}
    last_video: Optional[str] = None
    n_gt_superseded = 0

    def trajectory(video: str, track: sio.Track) -> Tuple[str, str]:
        return ("" if global_identity else video, track.name)

    def check_unique(keys: List[Tuple[str, str]], side: str, key: Tuple) -> None:
        names = [name for _video, name in keys]
        duplicates = sorted({n for n in names if names.count(n) > 1})
        if duplicates:
            hint = (
                " Predictions a user label on the same frame replaces (same "
                "track, or linked by from_predicted) are already left out, so "
                "two ground-truth detections share a track: two user labels, or "
                "a stale prediction carrying a user track. Fix the tracks, or "
                "pass user_labels_only=True (--user_labels_only) to score the "
                "user labels only."
                if side == "Ground-truth" and not user_labels_only
                else " Fix the tracks (or drop the duplicate detections) before "
                "scoring."
            )
            raise ValueError(
                f"{side} track(s) {duplicates} occur on more than one scored "
                f"detection in video {key[0]!r}, frame {key[1]} -- a track can "
                f"hold one detection per frame.{hint}"
            )

    for key in gt_keys:
        video = key[0]
        seq_key = "" if global_identity else video
        if global_identity and last_video is not None and video != last_video:
            seq_resets[seq_key].append(len(frames_by_seq[seq_key]))
        last_video = video
        gt_frame, pred_frame = gt_by_key[key], pred_by_key.get(key)
        if pred_frame is None:
            metrics.n_frames_missing_pred += 1

        gt_all = _identity_dets(gt_frame, carrier)
        gt_keep, n_superseded = _ground_truth_keep(
            gt_frame, carrier, gt_all, user_labels_only
        )
        n_gt_superseded += n_superseded
        gt_all_tracks = _identity_tracks(gt_frame, carrier, gt_all)
        gt_trk = [gt_all_tracks[i] for i in gt_keep]
        gt_tracked = np.array([t is not None for t in gt_trk], dtype=bool)

        if pred_frame is None:
            pred_all: List[Any] = []
            pred_all_tracks: List[Optional[sio.Track]] = []
        else:
            pred_all = _identity_dets(pred_frame, carrier)
            pred_all_tracks = _identity_tracks(pred_frame, carrier, pred_all)
        pred_keep = [j for j, t in enumerate(pred_all_tracks) if t is not None]
        metrics.n_pred_untracked += len(pred_all) - len(pred_keep)
        metrics.n_gt_untracked += int((~gt_tracked).sum())

        if carrier == "mask":
            gt_arrays = _frame_masks(gt_frame)
            match_gt = [gt_arrays[i] for i in gt_keep]
            pred_arrays = _frame_masks(pred_frame) if pred_frame is not None else []
            match_pred = [pred_arrays[j] for j in pred_keep]
        else:
            match_gt = [gt_all[i] for i in gt_keep]
            match_pred = [pred_all[j] for j in pred_keep]

        sim = _mot_similarity(match_gt, match_pred, method, threshold, oks_stddev)
        keep_pred = _drop_dont_care_predictions(sim, gt_tracked, sim_threshold)
        metrics.n_dont_care += int((~keep_pred).sum())
        sim = sim[np.ix_(gt_tracked, keep_pred)]

        gt_traj = [trajectory(video, t) for t in gt_trk if t is not None]
        pred_traj = [
            trajectory(video, pred_all_tracks[pred_keep[j]])
            for j in np.flatnonzero(keep_pred)
        ]
        check_unique(gt_traj, "Ground-truth", key)
        check_unique(pred_traj, "Predicted", key)
        frames_by_seq[seq_key].append(
            (
                np.array([gt_ids.setdefault(k, len(gt_ids)) for k in gt_traj], int),
                np.array(
                    [pred_ids.setdefault(k, len(pred_ids)) for k in pred_traj], int
                ),
                sim,
            )
        )

    per_seq = []
    for seq_key, frames in frames_by_seq.items():
        seq = mm.MOTSequence(
            gt_ids=[g for g, _p, _s in frames],
            pred_ids=[p for _g, p, _s in frames],
            similarity=[s for _g, _p, s in frames],
            resets=tuple(seq_resets[seq_key]),
        )
        per_seq.append(mm.mot_metrics(seq, threshold=sim_threshold))
    res = mm.combine_sequences(per_seq)

    metrics.mota, metrics.motp, metrics.moda = (
        float(res["MOTA"]),
        float(res["MOTP"]),
        float(res["MODA"]),
    )
    metrics.idsw, metrics.frag = int(res["IDSW"]), int(res["Frag"])
    metrics.mt, metrics.pt, metrics.ml = int(res["MT"]), int(res["PT"]), int(res["ML"])
    metrics.clr_tp, metrics.clr_fn, metrics.clr_fp = (
        int(res["CLR_TP"]),
        int(res["CLR_FN"]),
        int(res["CLR_FP"]),
    )
    metrics.idf1, metrics.idp, metrics.idr = (
        float(res["IDF1"]),
        float(res["IDP"]),
        float(res["IDR"]),
    )
    metrics.idtp, metrics.idfp, metrics.idfn = (
        int(res["IDTP"]),
        int(res["IDFP"]),
        int(res["IDFN"]),
    )
    for attr, field in (
        ("hota", "HOTA"),
        ("det_a", "DetA"),
        ("ass_a", "AssA"),
        ("loc_a", "LocA"),
        ("det_re", "DetRe"),
        ("det_pr", "DetPr"),
        ("ass_re", "AssRe"),
        ("ass_pr", "AssPr"),
        ("owta", "OWTA"),
    ):
        setattr(metrics, attr, float(np.mean(res[field])))
    metrics.hota_0, metrics.loc_a_0 = float(res["HOTA(0)"]), float(res["LocA(0)"])
    metrics.hota_alpha = [float(v) for v in res["HOTA"]]
    metrics.det_a_alpha = [float(v) for v in res["DetA"]]
    metrics.ass_a_alpha = [float(v) for v in res["AssA"]]
    metrics.loc_a_alpha = [float(v) for v in res["LocA"]]
    metrics.n_gt_dets = metrics.clr_tp + metrics.clr_fn
    metrics.n_pred_dets = metrics.clr_tp + metrics.clr_fp
    metrics.n_frames = len(gt_keys)
    metrics.n_videos = len(scored_videos)

    if n_gt_superseded:
        metrics.notes.append(_superseded_note(n_gt_superseded))
    if metrics.n_gt_untracked:
        metrics.notes.append(
            f"{metrics.n_gt_untracked} untracked ground-truth detections are "
            f"don't-care; {metrics.n_dont_care} tracked predictions matched them "
            "and were removed before scoring"
        )
    if metrics.n_dont_care and metrics.n_gt_dets and not metrics.clr_tp:
        metrics.notes.append(_dont_care_took_every_match_note())
    if metrics.n_gt_dets and metrics.n_pred_dets and not metrics.clr_tp:
        metrics.notes.append(
            "no tracked prediction matched a tracked ground-truth detection "
            f"({_threshold_phrase(method, threshold)}) -- check the match "
            "threshold and the carrier"
        )
    return metrics

compute_oks(points_gt, points_pr, scale=None, stddev=0.025, use_cocoeval=True)

Compute the object keypoints similarity between sets of points.

Parameters:

Name	Type	Description	Default
points_gt	ndarray	Ground truth instances of shape (n_gt, n_nodes, n_ed), where n_nodes is the number of body parts/keypoint types, and n_ed is the number of Euclidean dimensions (typically 2 or 3). Keypoints that are missing/not visible should be represented as NaNs.	required
points_pr	ndarray	Predicted instance of shape (n_pr, n_nodes, n_ed).	required
use_cocoeval	bool	Indicates whether the OKS score is calculated like cocoeval method or not. True indicating the score is calculated using the cocoeval method (widely used and the code can be found here at https://github.com/cocodataset/cocoapi/blob/8c9bcc3cf640524c4c20a9c40e89cb6a2f2fa0e9/PythonAPI/pycocotools/cocoeval.py#L192C5-L233C20) and False indicating the score is calculated using the method exactly as given in the paper referenced in the Notes below.	True
scale	Optional[float]	Size scaling factor to use when weighing the scores, typically the area of the bounding box of the instance (in pixels). This should be of the length n_gt. If a scalar is provided, the same number is used for all ground truth instances. If set to None, the bounding box area of the ground truth instances will be calculated.	None
stddev	float	The standard deviation associated with the spread in the localization accuracy of each node/keypoint type. This should be of the length n_nodes. "Easier" keypoint types will have lower values to reflect the smaller spread expected in localizing it.	0.025

Returns:

Type	Description
ndarray	The object keypoints similarity between every pair of ground truth and predicted instance, a numpy array of of shape (n_gt, n_pr) in the range of [0, 1.0], with 1.0 denoting a perfect match.

Notes

It's important to set the stddev appropriately when accounting for the difficulty of each keypoint type. For reference, the median value for all keypoint types in COCO is 0.072. The "easiest" keypoint is the left eye, with stddev of 0.025, since it is easy to precisely locate the eyes when labeling. The "hardest" keypoint is the left hip, with stddev of 0.107, since it's hard to locate the left hip bone without external anatomical features and since it is often occluded by clothing.

The implementation here is based off of the descriptions in: Ronch & Perona. "Benchmarking and Error Diagnosis in Multi-Instance Pose Estimation." ICCV (2017).

Source code in sleap_nn/evaluation.py

def compute_oks(
    points_gt: np.ndarray,
    points_pr: np.ndarray,
    scale: Optional[float] = None,
    stddev: float = 0.025,
    use_cocoeval: bool = True,
) -> np.ndarray:
    """Compute the object keypoints similarity between sets of points.

    Args:
        points_gt: Ground truth instances of shape (n_gt, n_nodes, n_ed),
            where n_nodes is the number of body parts/keypoint types, and n_ed
            is the number of Euclidean dimensions (typically 2 or 3). Keypoints
            that are missing/not visible should be represented as NaNs.
        points_pr: Predicted instance of shape (n_pr, n_nodes, n_ed).
        use_cocoeval: Indicates whether the OKS score is calculated like cocoeval
            method or not. True indicating the score is calculated using the
            cocoeval method (widely used and the code can be found here at
            https://github.com/cocodataset/cocoapi/blob/8c9bcc3cf640524c4c20a9c40e89cb6a2f2fa0e9/PythonAPI/pycocotools/cocoeval.py#L192C5-L233C20)
            and False indicating the score is calculated using the method exactly
            as given in the paper referenced in the Notes below.
        scale: Size scaling factor to use when weighing the scores, typically
            the area of the bounding box of the instance (in pixels). This
            should be of the length n_gt. If a scalar is provided, the same
            number is used for all ground truth instances. If set to None, the
            bounding box area of the ground truth instances will be calculated.
        stddev: The standard deviation associated with the spread in the
            localization accuracy of each node/keypoint type. This should be of
            the length n_nodes. "Easier" keypoint types will have lower values
            to reflect the smaller spread expected in localizing it.

    Returns:
        The object keypoints similarity between every pair of ground truth and
        predicted instance, a numpy array of of shape (n_gt, n_pr) in the range
        of [0, 1.0], with 1.0 denoting a perfect match.

    Notes:
        It's important to set the stddev appropriately when accounting for the
        difficulty of each keypoint type. For reference, the median value for
        all keypoint types in COCO is 0.072. The "easiest" keypoint is the left
        eye, with stddev of 0.025, since it is easy to precisely locate the
        eyes when labeling. The "hardest" keypoint is the left hip, with stddev
        of 0.107, since it's hard to locate the left hip bone without external
        anatomical features and since it is often occluded by clothing.

        The implementation here is based off of the descriptions in:
        Ronch & Perona. "Benchmarking and Error Diagnosis in Multi-Instance Pose
        Estimation." ICCV (2017).
    """
    if points_gt.ndim == 2:
        points_gt = np.expand_dims(points_gt, axis=0)
    if points_pr.ndim == 2:
        points_pr = np.expand_dims(points_pr, axis=0)

    if scale is None:
        scale = compute_instance_area(points_gt)

    n_gt, n_nodes, n_ed = points_gt.shape  # n_ed = 2 or 3 (euclidean dimensions)
    n_pr = points_pr.shape[0]

    # If scalar scale was provided, use the same for each ground truth instance.
    if np.isscalar(scale):
        scale = np.full(n_gt, scale)

    # If scalar standard deviation was provided, use the same for each node.
    if np.isscalar(stddev):
        stddev = np.full(n_nodes, stddev)

    # Compute displacement between each pair.
    displacement = np.reshape(points_gt, (n_gt, 1, n_nodes, n_ed)) - np.reshape(
        points_pr, (1, n_pr, n_nodes, n_ed)
    )
    assert displacement.shape == (n_gt, n_pr, n_nodes, n_ed)

    # Convert to pairwise Euclidean distances.
    distance = (displacement**2).sum(axis=-1)  # (n_gt, n_pr, n_nodes)
    assert distance.shape == (n_gt, n_pr, n_nodes)

    # Compute the normalization factor per keypoint.
    if use_cocoeval:
        # If use_cocoeval is True, then compute normalization factor according to cocoeval.
        spread_factor = (2 * stddev) ** 2
        scale_factor = 2 * (scale + np.spacing(1))
    else:
        # If use_cocoeval is False, then compute normalization factor according to the paper.
        spread_factor = stddev**2
        scale_factor = 2 * ((scale + np.spacing(1)) ** 2)
    normalization_factor = np.reshape(spread_factor, (1, 1, n_nodes)) * np.reshape(
        scale_factor, (n_gt, 1, 1)
    )
    assert normalization_factor.shape == (n_gt, 1, n_nodes)

    # Since a "miss" is considered as KS < 0.5, we'll set the
    # distances for predicted points that are missing to inf.
    missing_pr = np.any(np.isnan(points_pr), axis=-1)  # (n_pr, n_nodes)
    assert missing_pr.shape == (n_pr, n_nodes)
    distance[:, missing_pr] = np.inf

    # Compute the keypoint similarity as per the top of Eq. 1.
    ks = np.exp(-(distance / normalization_factor))  # (n_gt, n_pr, n_nodes)
    assert ks.shape == (n_gt, n_pr, n_nodes)

    # Set the KS for missing ground truth points to 0.
    # This is equivalent to the visibility delta function of the bottom
    # of Eq. 1.
    missing_gt = np.any(np.isnan(points_gt), axis=-1)  # (n_gt, n_nodes)
    assert missing_gt.shape == (n_gt, n_nodes)
    # BROADCAST, don't boolean-index. `ks` is (n_gt, n_pr, n_nodes) while the mask
    # is (n_gt, 1, n_nodes); numpy requires a boolean index to match the indexed
    # array's shape exactly, so `ks[mask] = 0` raised an IndexError for every
    # n_pr > 1 -- i.e. for the (n_gt, n_pr) matrix this function documents and
    # returns. Latent because every in-repo caller passes one prediction at a time.
    ks = np.where(missing_gt[:, None, :], 0.0, ks)

    # Compute the OKS.
    n_visible_gt = np.sum(
        (~missing_gt).astype("float32"), axis=-1, keepdims=True
    )  # (n_gt, 1)
    oks = np.sum(ks, axis=-1) / n_visible_gt
    assert oks.shape == (n_gt, n_pr)

    return oks

embedding_full_eval(gallery_emb, gallery_y, query_emb, query_y, k=7)

Combined retrieval + verification + kNN-accuracy metrics dict.

kNN accuracy, like rank-1 and mAP, leaves out the queries whose identity is absent from the gallery.

Source code in sleap_nn/evaluation.py

def embedding_full_eval(gallery_emb, gallery_y, query_emb, query_y, k: int = 7):
    """Combined retrieval + verification + kNN-accuracy metrics dict.

    kNN accuracy, like rank-1 and mAP, leaves out the queries whose identity is
    absent from the gallery.
    """
    qy = np.asarray(query_y)
    has_pos = np.isin(qy, np.asarray(gallery_y))
    if _nonfinite_embeddings(np.asarray(gallery_emb), np.asarray(query_emb)):
        nan = float("nan")
        return {
            "rank1": nan,
            "mAP": nan,
            "n_no_positive_queries": int((~has_pos).sum()),
            "auc": nan,
            "eer": nan,
            "knn_acc": nan,
        }
    out = {}
    out.update(retrieval_metrics(gallery_emb, gallery_y, query_emb, query_y))
    out.update(verification_metrics(gallery_emb, gallery_y, query_emb, query_y))
    pred, _ = knn_classify(gallery_emb, gallery_y, query_emb, k=k)
    out["knn_acc"] = _mean_or_nan((pred == qy)[has_pos])
    return out

embedding_leave_self_out_eval(emb, y, k=7, max_n=5000)

Leave-self-out retrieval/verification/kNN over one labeled embedding set.

Gallery == query == the same set, with each item's self-match excluded (the similarity diagonal is masked to -inf so an item is never retrieved by itself). This is exactly the protocol the per-epoch :class:~sleap_nn.training.callbacks.EmbeddingEvaluationCallback uses for checkpoint selection, so the post-training headline matches the selected metric.

• rank-1, mAP and kNN accuracy are averaged over the queries that have a positive, i.e. whose identity appears at least twice. Singleton identities have nothing to retrieve and are counted in n_no_positive_queries instead (the standard CMC protocol).
• kNN accuracy: the similarity-weighted vote of the k nearest other items, with negative similarities carrying no weight (see :func:_knn_vote).
• AUC / EER score every unordered pair of distinct items once, same identity vs different identity (see :func:_auc_eer).

The similarity is computed once, in blocks of query rows, and every metric reads that one pass, so memory stays at a few (block x N) arrays plus the N^2/2 pair scores.

Parameters:

Name	Type	Description	Default
emb		(N, D) embeddings.	required
y		(N,) identity labels (any comparable dtype).	required
k	int	k for the cosine-kNN accuracy (clamped to [1, N - 1]).	7
max_n	int	Cap on the number of embeddings scored. Larger sets are deterministically subsampled so the per-epoch eval stays bounded (an uncapped set costs O(N^2) every epoch). None disables the cap.	5000

Returns:

Type	Description
	dict with rank1, mAP, auc, eer, knn_acc (rank-1, mAP and kNN are NaN when no query has a positive; AUC/EER when no pair of either kind exists; all five when any embedding is NaN/inf) and n_no_positive_queries.

Source code in sleap_nn/evaluation.py

def embedding_leave_self_out_eval(emb, y, k: int = 7, max_n: int = 5000):
    """Leave-self-out retrieval/verification/kNN over one labeled embedding set.

    Gallery == query == the same set, with each item's self-match excluded (the
    similarity diagonal is masked to -inf so an item is never retrieved by itself).
    This is exactly the protocol the per-epoch
    :class:`~sleap_nn.training.callbacks.EmbeddingEvaluationCallback` uses for
    checkpoint selection, so the post-training headline matches the selected metric.

    - rank-1, mAP and kNN accuracy are averaged over the queries that have a
      positive, i.e. whose identity appears at least twice. Singleton identities
      have nothing to retrieve and are counted in ``n_no_positive_queries`` instead
      (the standard CMC protocol).
    - kNN accuracy: the similarity-weighted vote of the ``k`` nearest other items,
      with negative similarities carrying no weight (see :func:`_knn_vote`).
    - AUC / EER score every unordered pair of distinct items once, same identity vs
      different identity (see :func:`_auc_eer`).

    The similarity is computed once, in blocks of query rows, and every metric reads
    that one pass, so memory stays at a few ``(block x N)`` arrays plus the ``N^2/2``
    pair scores.

    Args:
        emb: ``(N, D)`` embeddings.
        y: ``(N,)`` identity labels (any comparable dtype).
        k: ``k`` for the cosine-kNN accuracy (clamped to ``[1, N - 1]``).
        max_n: Cap on the number of embeddings scored. Larger sets are
            deterministically subsampled so the per-epoch eval stays bounded (an
            uncapped set costs O(N^2) every epoch). ``None`` disables the cap.

    Returns:
        dict with ``rank1``, ``mAP``, ``auc``, ``eer``, ``knn_acc`` (rank-1, mAP and
        kNN are NaN when no query has a positive; AUC/EER when no pair of either kind
        exists; all five when any embedding is NaN/inf) and
        ``n_no_positive_queries``.
    """
    emb = np.asarray(emb, dtype=np.float64)
    y = np.asarray(y)
    if max_n is not None and len(emb) > max_n:
        # Deterministic subsample so the N x N similarity + argsort stay bounded.
        keep = np.sort(np.random.default_rng(0).choice(len(emb), max_n, replace=False))
        emb, y = emb[keep], y[keep]
    n = len(emb)
    _, identity, counts = np.unique(y, return_inverse=True, return_counts=True)
    has_pos = counts[identity.reshape(-1)] > 1
    if _nonfinite_embeddings(emb):
        nan = float("nan")
        return {
            "rank1": nan,
            "mAP": nan,
            "auc": nan,
            "eer": nan,
            "knn_acc": nan,
            "n_no_positive_queries": int((~has_pos).sum()),
        }
    emb = _l2_normalize(emb)

    top1 = np.zeros(n, dtype=bool)
    ap = np.zeros(n)
    knn_hit = np.zeros(n, dtype=bool)
    # Every unordered pair (i < j) once: same-identity scores in `pos`, the rest in
    # `neg`, filled block by block.
    n_pos_pairs = int((counts * (counts - 1) // 2).sum())
    pos = np.empty(n_pos_pairs)
    neg = np.empty(n * (n - 1) // 2 - n_pos_pairs)
    n_pos_filled = n_neg_filled = 0
    cols = np.arange(n)
    kk = max(1, min(k, n - 1))
    for start in range(0, n if n > 1 else 0, _RETRIEVAL_BLOCK_ROWS):
        rows = cols[start : start + _RETRIEVAL_BLOCK_ROWS]
        sim = emb[rows] @ emb.T
        upper = cols[None, :] > rows[:, None]
        same = y[rows, None] == y[None, :]
        block_pos, block_neg = sim[upper & same], sim[upper & ~same]
        pos[n_pos_filled : n_pos_filled + block_pos.size] = block_pos
        neg[n_neg_filled : n_neg_filled + block_neg.size] = block_neg
        n_pos_filled += block_pos.size
        n_neg_filled += block_neg.size

        sim[np.arange(len(rows)), rows] = -np.inf  # leave-self-out
        top1[rows], ap[rows], knn_hit[rows] = _score_queries(
            sim, y[rows], y, kk, drop_self=True
        )

    pos.sort()
    neg.sort()
    auc, eer = _auc_eer(pos, neg)
    return {
        "rank1": _mean_or_nan(top1[has_pos]),
        "mAP": _mean_or_nan(ap[has_pos]),
        "auc": round(auc, 4),
        "eer": round(eer, 4),
        "knn_acc": _mean_or_nan(knn_hit[has_pos]),
        "n_no_positive_queries": int((~has_pos).sum()),
    }

find_frame_pairs(labels_gt, labels_pr, user_labels_only=True, keep_user_centroid_frames=False)

Find corresponding frames across two sets of labels.

This function uses sleap-io's robust video matching API to handle various scenarios including embedded videos, cross-platform paths, and videos with different metadata.

Parameters:

Name	Type	Description	Default
labels_gt	Labels	A sio.Labels instance with ground truth instances.	required
labels_pr	Labels	A sio.Labels instance with predicted instances.	required
keep_user_centroid_frames	bool	If True, a ground-truth frame also survives the user_labels_only filter when it carries user Centroid annotations but no user instances. Set by match_method="centroid": centroid annotations (hand-made, or derived from segmentation masks by data_config.centroids_from_masks) are the ground truth for a centroid model, and a mask-only file has no instances at all -- so the instance-only filter dropped every frame and evaluation died with "Empty Frame Pairs".	False
user_labels_only	bool	If False, frames with predicted instances in labels_gt will also be considered for matching.	True

Returns:

Type	Description
List[Tuple[LabeledFrame, LabeledFrame]]	A list of pairs of sio.LabeledFrames in the form (frame_gt, frame_pr).

Source code in sleap_nn/evaluation.py

def find_frame_pairs(
    labels_gt: sio.Labels,
    labels_pr: sio.Labels,
    user_labels_only: bool = True,
    keep_user_centroid_frames: bool = False,
) -> List[Tuple[sio.LabeledFrame, sio.LabeledFrame]]:
    """Find corresponding frames across two sets of labels.

    This function uses sleap-io's robust video matching API to handle various
    scenarios including embedded videos, cross-platform paths, and videos with
    different metadata.

    Args:
        labels_gt: A `sio.Labels` instance with ground truth instances.
        labels_pr: A `sio.Labels` instance with predicted instances.
        keep_user_centroid_frames: If True, a ground-truth frame also survives the
            ``user_labels_only`` filter when it carries user ``Centroid``
            annotations but no user instances. Set by ``match_method="centroid"``:
            centroid annotations (hand-made, or derived from segmentation masks by
            ``data_config.centroids_from_masks``) are the ground truth for a
            centroid model, and a mask-only file has no instances at all -- so the
            instance-only filter dropped every frame and evaluation died with
            "Empty Frame Pairs".
        user_labels_only: If False, frames with predicted instances in `labels_gt` will
            also be considered for matching.

    Returns:
        A list of pairs of `sio.LabeledFrame`s in the form `(frame_gt, frame_pr)`.
    """
    # Use sleap-io's robust video matching API (added in 0.6.2)
    # The match() method returns a MatchResult with video_map: {pred_video: gt_video}
    #
    # NOTE: sleap-io's AUTO matcher previously shape-rejected candidates before its
    # definitive is_same_file check, so it failed to pair an embedded-subset GT video
    # with its restored-original prediction counterpart (same file, different frame
    # count) -- e.g. post-training eval on an embedded .pkg.slp logged "Empty Frame
    # Pairs". This is resolved by the pinned sleap-io (talmolab/sleap-io#473/#476),
    # whose AUTO matcher resolves effective shape through the source_video chain, so
    # the match here works with no workaround.
    match_result = labels_gt.match(labels_pr)

    frame_pairs = []
    # Iterate over matched video pairs (pred_video -> gt_video mapping)
    for video_pr, video_gt in match_result.video_map.items():
        if video_gt is None:
            # No match found for this prediction video
            continue

        # Find labeled frames in this video.
        labeled_frames_gt = labels_gt.find(video_gt)
        if user_labels_only:
            # Build fresh LabeledFrame copies restricted to user instances,
            # rather than mutating `lf.instances` in place -- `labels_gt.find`
            # returns references into the caller's actual Labels object, so
            # mutating it here permanently discards PredictedInstances from
            # ground truth the caller may reuse afterward (e.g. a second
            # Evaluator call with user_labels_only=False on the same labels_gt).
            labeled_frames_gt = [
                attrs.evolve(lf, instances=lf.user_instances)
                for lf in labeled_frames_gt
                if len(lf.user_instances) > 0
                or (keep_user_centroid_frames and _user_centroids(lf))
            ]

        # Attempt to match each labeled frame in the ground truth.
        for labeled_frame_gt in labeled_frames_gt:
            labeled_frames_pr = labels_pr.find(
                video_pr, frame_idx=labeled_frame_gt.frame_idx
            )

            if not labeled_frames_pr:
                # No match
                continue
            elif len(labeled_frames_pr) == 1:
                # Match!
                frame_pairs.append((labeled_frame_gt, labeled_frames_pr[0]))

    return frame_pairs

get_instances(labeled_frame)

Get a list of instances of type MatchInstance from the Labeled Frame.

Parameters:

Name	Type	Description	Default
labeled_frame	LabeledFrame	Input Labeled frame of type sio.LabeledFrame.	required

Returns:

Type	Description
List[MatchInstance]	List of MatchInstance objects for the given labeled frame.

Source code in sleap_nn/evaluation.py

def get_instances(labeled_frame: sio.LabeledFrame) -> List[MatchInstance]:
    """Get a list of instances of type MatchInstance from the Labeled Frame.

    Args:
        labeled_frame: Input Labeled frame of type sio.LabeledFrame.

    Returns:
        List of MatchInstance objects for the given labeled frame.
    """
    instance_list = []
    frame_idx = labeled_frame.frame_idx
    video_path = _video_key(labeled_frame.video)

    for instance in labeled_frame.instances:
        match_instance = MatchInstance(
            instance=instance, frame_idx=frame_idx, video_path=video_path
        )
        instance_list.append(match_instance)
    return instance_list

identity_metrics(gt_labels, pred_labels, carrier='pose', *, match_threshold=None, mt_threshold=0.8, ml_threshold=0.2, user_labels_only=False, global_identity=False, oks_stddev=0.025)

Score a tracked prediction against tracked ground truth.

Detections are Hungarian-matched to ground truth within each frame (OKS for "pose", pixel distance for single-node pose skeletons, mask IoU for "mask"), then identity is scored over those matches. A detection's track is read off it, or through its pose <-> mask link when unset on it (see :func:_identity_tracks).

What is scored:

• Per video. Trajectories are (video, track name) pairs unless global_identity: trackers name tracks per video, so the same name in two videos is two animals.
• Every ground-truth frame of each video the prediction covers (shares at least one frame with). A frame missing from the prediction scores as misses; a video with no frame in common is left out and named in notes.
• Untracked ground truth is don't-care. It is matched against, and a prediction matched to it counts as neither a hit nor a false positive.
• Untracked predictions are counted (n_pred_untracked) but never matched: they have no identity to get right, so their ground truth scores as missed.
• Corrected predictions are left out of the ground truth. A PredictedInstance (or its mask) that a user label on the same frame replaces -- a pose corrected in the GUI, which keeps the prediction next to the user copy on the same track -- is not ground truth; the user copy is (:func:_superseded_ground_truth). Counted in notes.

Parameters:

Name	Type	Description	Default
gt_labels	Labels	Ground truth with track set on the detections to score.	required
pred_labels	Labels	Prediction with track set by the tracker under test.	required
carrier	str	"pose" (instances) or "mask" (segmentation masks) -- which detections carry identity.	'pose'
match_threshold	Optional[float]	Minimum similarity for a ground-truth/predicted pair to count as matched -- OKS or IoU -- or, for single-node skeletons, the maximum centroid distance in PIXELS. None (default) uses IDENTITY_MATCH_THRESHOLDS: 0.5 for OKS / IoU, 50 px for distance.	None
mt_threshold	float	Coverage at or above which a ground-truth trajectory counts as mostly-tracked.	0.8
ml_threshold	float	Coverage below which a ground-truth trajectory counts as mostly-lost.	0.2
user_labels_only	bool	Drop model output (PredictedInstance / PredictedSegmentationMask) from the GROUND-TRUTH side. Defaults to False, unlike the detection metrics, because tracked ground truth usually is predicted: the standard workflow predicts poses and then assigns or corrects tracks over them, so filtering by type would silently discard the whole ground truth (measured on the re-ID benchmark's GT sessions: 2465 detections to 0). Pass True when the ground truth is user-labeled and the file also carries stale predictions from an earlier run, which would otherwise be scored as extra trajectories. Either way the count is reported in notes, and the prediction side is never filtered. Off, predictions a user label on the same frame replaces are still left out (see above).	False
global_identity	bool	Key trajectories by track name ACROSS videos -- for files whose names are global identities (a multi-class ID model, or a project proofread with one name per animal), where keeping an animal's name from one video to the next is part of the score.	False
oks_stddev	float	Per-keypoint OKS spread (:func:compute_oks) for the "oks" matcher, a finite number > 0. Default 0.025, compute_oks's own.	0.025

Returns:

Name	Type	Description
An	IdentityMetrics	class:IdentityMetrics. When the two files share no frames, the result is empty and notes says so rather than raising -- check n_frames_compared before quoting a number.

Raises:

Type	Description
ValueError	If the carrier is unknown or oks_stddev is not a finite number greater than 0.

Source code in sleap_nn/evaluation.py

def identity_metrics(
    gt_labels: sio.Labels,
    pred_labels: sio.Labels,
    carrier: str = "pose",
    *,
    match_threshold: Optional[float] = None,
    mt_threshold: float = 0.8,
    ml_threshold: float = 0.2,
    user_labels_only: bool = False,
    global_identity: bool = False,
    oks_stddev: float = 0.025,
) -> IdentityMetrics:
    """Score a tracked prediction against tracked ground truth.

    Detections are Hungarian-matched to ground truth within each frame (OKS for
    ``"pose"``, pixel distance for single-node pose skeletons, mask IoU for
    ``"mask"``), then identity is scored over those matches. A detection's track
    is read off it, or through its pose <-> mask link when unset on it (see
    :func:`_identity_tracks`).

    What is scored:

    - **Per video.** Trajectories are ``(video, track name)`` pairs unless
      ``global_identity``: trackers name tracks per video, so the same name in
      two videos is two animals.
    - **Every ground-truth frame** of each video the prediction covers (shares at
      least one frame with). A frame missing from the prediction scores as
      misses; a video with no frame in common is left out and named in
      ``notes``.
    - **Untracked ground truth is don't-care.** It is matched against, and a
      prediction matched to it counts as neither a hit nor a false positive.
    - **Untracked predictions** are counted (``n_pred_untracked``) but never
      matched: they have no identity to get right, so their ground truth scores
      as missed.
    - **Corrected predictions are left out of the ground truth.** A
      ``PredictedInstance`` (or its mask) that a user label on the same frame
      replaces -- a pose corrected in the GUI, which keeps the prediction next
      to the user copy on the same track -- is not ground truth; the user copy
      is (:func:`_superseded_ground_truth`). Counted in ``notes``.

    Args:
        gt_labels: Ground truth with ``track`` set on the detections to score.
        pred_labels: Prediction with ``track`` set by the tracker under test.
        carrier: ``"pose"`` (instances) or ``"mask"`` (segmentation masks) --
            which detections carry identity.
        match_threshold: Minimum similarity for a ground-truth/predicted pair to
            count as matched -- OKS or IoU -- or, for single-node skeletons, the
            maximum centroid distance in PIXELS. ``None`` (default) uses
            ``IDENTITY_MATCH_THRESHOLDS``: 0.5 for OKS / IoU, 50 px for distance.
        mt_threshold: Coverage at or above which a ground-truth trajectory counts
            as mostly-tracked.
        ml_threshold: Coverage below which a ground-truth trajectory counts as
            mostly-lost.
        user_labels_only: Drop model output (``PredictedInstance`` /
            ``PredictedSegmentationMask``) from the GROUND-TRUTH side.
            **Defaults to False, unlike the detection metrics**, because tracked
            ground truth usually *is* predicted: the standard workflow predicts
            poses and then assigns or corrects tracks over them, so filtering by
            type would silently discard the whole ground truth (measured on the
            re-ID benchmark's GT sessions: 2465 detections to 0). Pass True when
            the ground truth is user-labeled and the file also carries stale
            predictions from an earlier run, which would otherwise be scored as
            extra trajectories. Either way the count is reported in ``notes``,
            and the prediction side is never filtered. Off, predictions a user
            label on the same frame replaces are still left out (see above).
        global_identity: Key trajectories by track name ACROSS videos -- for
            files whose names are global identities (a multi-class ID model, or
            a project proofread with one name per animal), where keeping an
            animal's name from one video to the next is part of the score.
        oks_stddev: Per-keypoint OKS spread (:func:`compute_oks`) for the
            ``"oks"`` matcher, a finite number > 0. Default 0.025,
            ``compute_oks``'s own.

    Returns:
        An :class:`IdentityMetrics`. When the two files share no frames, the
        result is empty and ``notes`` says so rather than raising -- check
        ``n_frames_compared`` before quoting a number.

    Raises:
        ValueError: If the carrier is unknown or ``oks_stddev`` is not a finite
            number greater than 0.
    """
    from collections import Counter, defaultdict
    from scipy.optimize import linear_sum_assignment

    carrier = _validate_carrier(carrier)
    oks_stddev = _check_oks_stddev(oks_stddev)
    method = _identity_match_method(gt_labels, pred_labels, carrier)
    threshold = (
        IDENTITY_MATCH_THRESHOLDS[method]
        if match_threshold is None
        else float(match_threshold)
    )
    metrics = IdentityMetrics(
        match_method=method, match_threshold=threshold, global_identity=global_identity
    )

    gt_by_key, pred_by_key, gt_keys, scored_videos, align_notes = (
        _align_identity_frames(gt_labels, pred_labels)
    )
    metrics.notes.extend(align_notes)
    if not scored_videos:
        return metrics
    n_missing = sum(1 for k in gt_keys if k not in pred_by_key)

    def trajectory(video: str, track: sio.Track) -> Tuple[str, str]:
        return ("" if global_identity else video, track.name)

    # gt trajectory -> ordered list of (frame position, matched pred identity)
    timeline: Dict[Tuple[str, str], List[Tuple[int, Optional[Tuple[str, str]]]]] = (
        defaultdict(list)
    )
    co_matched: "Counter[Tuple[Tuple[str, str], Tuple[str, str]]]" = Counter()
    pred_track_dets: "Counter[Tuple[str, str]]" = Counter()
    gt_tracks: set = set()
    pred_tracks: set = set()
    gt_names_by_video: Dict[str, set] = defaultdict(set)
    n_gt_predicted = 0
    n_gt_superseded = 0

    for position, key in enumerate(gt_keys):
        video = key[0]
        gt_frame, pred_frame = gt_by_key[key], pred_by_key.get(key)

        gt_all = _identity_dets(gt_frame, carrier)
        gt_keep, n_superseded = _ground_truth_keep(
            gt_frame, carrier, gt_all, user_labels_only
        )
        n_gt_superseded += n_superseded
        # Predicted GT detections: dropped (user_labels_only) or scored.
        kept = set(gt_keep)
        n_gt_predicted += sum(
            1
            for i, d in enumerate(gt_all)
            if _is_predicted_detection(d) and (user_labels_only or i in kept)
        )
        gt_all_tracks = _identity_tracks(gt_frame, carrier, gt_all)
        gt_trk = [gt_all_tracks[i] for i in gt_keep]
        tracked_gt = [i for i, track in enumerate(gt_trk) if track is not None]

        if pred_frame is None:
            pred_all: List[Any] = []
            pred_all_tracks: List[Optional[sio.Track]] = []
        else:
            pred_all = _identity_dets(pred_frame, carrier)
            pred_all_tracks = _identity_tracks(pred_frame, carrier, pred_all)
        pred_keep = [j for j, track in enumerate(pred_all_tracks) if track is not None]

        metrics.n_gt_dets += len(tracked_gt)
        metrics.n_gt_untracked += len(gt_keep) - len(tracked_gt)
        metrics.n_pred_dets += len(pred_all)
        metrics.n_pred_untracked += len(pred_all) - len(pred_keep)
        for i in tracked_gt:
            gt_tracks.add(trajectory(video, gt_trk[i]))
            gt_names_by_video[video].add(gt_trk[i].name)
        pred_tracks.update(trajectory(video, pred_all_tracks[j]) for j in pred_keep)

        # For the mask carrier, match on decoded image-grid arrays rather than on
        # the mask objects: `_frame_masks` is scale-aware, so a stride-res
        # prediction and a full-res ground-truth mask are compared on a common
        # pixel grid (the class of bug #693/#694 fixed). It decodes `lf.masks` in
        # order, so selecting by the same indices keeps it aligned.
        if carrier == "mask":
            gt_arrays = _frame_masks(gt_frame)
            match_gt = [gt_arrays[i] for i in gt_keep]
            pred_arrays = _frame_masks(pred_frame) if pred_frame is not None else []
            match_pred = [pred_arrays[j] for j in pred_keep]
        else:
            match_gt = [gt_all[i] for i in gt_keep]
            match_pred = [pred_all[j] for j in pred_keep]

        # Match against ALL ground truth, tracked or not: a prediction on an
        # untracked GT animal is don't-care, not a false positive.
        matched_gt: Dict[int, Tuple[str, str]] = {}
        for gt_idx, pred_idx, _score in _match_identity_frame(
            match_gt, match_pred, method, threshold, oks_stddev=oks_stddev
        ):
            gt_track = gt_trk[gt_idx]
            if gt_track is None:
                metrics.n_dont_care += 1
                continue
            gt_key = trajectory(video, gt_track)
            pred_key = trajectory(video, pred_all_tracks[pred_keep[pred_idx]])
            matched_gt[gt_idx] = pred_key
            co_matched[(gt_key, pred_key)] += 1
            pred_track_dets[pred_key] += 1
            metrics.n_matched += 1

        for gt_idx in tracked_gt:
            timeline[trajectory(video, gt_trk[gt_idx])].append(
                (position, matched_gt.get(gt_idx))
            )

    metrics.n_frames_compared = len(gt_keys)
    metrics.n_frames_missing_pred = n_missing
    metrics.n_videos_compared = len(scored_videos)
    metrics.n_gt_tracks = len(gt_tracks)
    metrics.n_pred_tracks = len(pred_tracks)
    if n_gt_predicted:
        if user_labels_only:
            metrics.notes.append(
                f"{n_gt_predicted} predicted detections were dropped from the "
                "ground truth (user_labels_only=True)"
            )
            if not metrics.n_gt_dets:
                metrics.notes.append(
                    "user_labels_only=True left NO ground truth: every tracked "
                    "detection is model output. Tracked GT is usually predicted "
                    "poses with tracks assigned afterwards -- pass "
                    "user_labels_only=False to score it."
                )
        else:
            metrics.notes.append(
                f"{n_gt_predicted} of the ground-truth detections are model "
                "output (tracks assigned over predictions); scored as ground "
                "truth. Pass user_labels_only=True to exclude them."
            )
    if n_gt_superseded:
        metrics.notes.append(_superseded_note(n_gt_superseded))
    if metrics.n_gt_untracked:
        metrics.notes.append(
            f"{metrics.n_gt_untracked} untracked ground-truth detections are "
            f"don't-care; {metrics.n_dont_care} tracked predictions matched them "
            "and are scored as neither hits nor false positives"
        )
    if metrics.n_dont_care and metrics.n_gt_dets and not metrics.n_matched:
        metrics.notes.append(_dont_care_took_every_match_note())
    if not global_identity and len(scored_videos) > 1:
        name_counts = Counter(
            name for names in gt_names_by_video.values() for name in names
        )
        recurring = sorted(name for name, n in name_counts.items() if n > 1)
        if recurring:
            metrics.notes.append(
                f"GT track names recur across videos ({recurring[:5]}"
                f"{' ...' if len(recurring) > 5 else ''}); scored as separate "
                "animals per video. If they are global identities, pass "
                "global_identity=True (--global_identity)."
            )
    if metrics.n_gt_dets and pred_tracks and not metrics.n_matched:
        metrics.notes.append(
            "no tracked prediction matched a tracked ground-truth detection "
            f"({_threshold_phrase(method, threshold)}) -- check the match "
            "threshold and the carrier"
        )

    # --- ID switches, coverage and fragmentation, per ground-truth trajectory ---
    coverages: List[float] = []
    for entries in timeline.values():
        entries.sort(key=lambda entry: entry[0])
        matched = [name for _position, name in entries if name is not None]
        coverage = len(matched) / len(entries) if entries else 0.0
        coverages.append(coverage)

        last: Optional[Tuple[str, str]] = None
        for _position, name in entries:
            if name is None:
                continue
            if last is not None and name != last:
                metrics.id_switches += 1
            last = name

        # Fragmentation: matched -> unmatched -> matched interruptions. Only gaps
        # that are re-matched later count, so a trajectory that simply ends is
        # not a fragmentation.
        matched_seq = [name is not None for _position, name in entries]
        started = False
        for i, is_matched in enumerate(matched_seq):
            if is_matched:
                started = True
            elif started and i > 0 and matched_seq[i - 1] and any(matched_seq[i + 1 :]):
                metrics.fragmentations += 1

        if coverage >= mt_threshold:
            metrics.mostly_tracked += 1
        elif coverage < ml_threshold:
            metrics.mostly_lost += 1
        else:
            metrics.partly_tracked += 1

    metrics.mean_gt_coverage = float(np.mean(coverages)) if coverages else float("nan")

    # --- IDF1: global max-weight ground-truth <-> predicted identity assignment ---
    # Under per-video keying no pair spans two videos, so this one assignment is
    # exactly the per-video assignments, summed.
    gt_names = sorted(gt_tracks)
    pred_names = sorted(pred_tracks)
    if gt_names and pred_names:
        weights = np.zeros((len(gt_names), len(pred_names)))
        gt_index = {name: i for i, name in enumerate(gt_names)}
        pred_index = {name: i for i, name in enumerate(pred_names)}
        for (gt_name, pred_name), count in co_matched.items():
            weights[gt_index[gt_name], pred_index[pred_name]] = count
        rows, cols = linear_sum_assignment(-weights)
        idtp = float(weights[rows, cols].sum())
        idfn = metrics.n_gt_dets - idtp
        idfp = (
            metrics.n_pred_dets - metrics.n_pred_untracked - metrics.n_dont_care
        ) - idtp
        metrics.idp = idtp / (idtp + idfp) if (idtp + idfp) > 0 else float("nan")
        metrics.idr = idtp / (idtp + idfn) if (idtp + idfn) > 0 else float("nan")
        denominator = 2 * idtp + idfp + idfn
        metrics.idf1 = 2 * idtp / denominator if denominator > 0 else float("nan")
        metrics.idf1_matched = (
            idtp / metrics.n_matched if metrics.n_matched else float("nan")
        )

    # --- Track purity, length-weighted over predicted tracks ---
    by_pred: Dict[Tuple[str, str], "Counter[Tuple[str, str]]"] = defaultdict(Counter)
    for (gt_name, pred_name), count in co_matched.items():
        by_pred[pred_name][gt_name] += count
    if by_pred:
        total = sum(pred_track_dets[name] for name in by_pred)
        metrics.mean_track_purity = (
            float(sum(max(counts.values()) for counts in by_pred.values()) / total)
            if total
            else float("nan")
        )

    return metrics

knn_classify(gallery_emb, gallery_y, query_emb, k=7)

Cosine k-NN classification (see :func:_knn_vote). Returns (pred, conf).

Source code in sleap_nn/evaluation.py

def knn_classify(gallery_emb, gallery_y, query_emb, k: int = 7):
    """Cosine k-NN classification (see :func:`_knn_vote`). Returns (pred, conf)."""
    g, q = _l2_normalize(np.asarray(gallery_emb)), _l2_normalize(np.asarray(query_emb))
    gy = np.asarray(gallery_y)
    sim = q @ g.T
    idx = np.argsort(-sim, 1)[:, :k]
    return _knn_vote(gy[idx], np.take_along_axis(sim, idx, 1))

load_metrics(path, split='test', dataset_idx=0)

Load metrics from a model folder or metrics file.

This function supports both the new format (single "metrics" key) and the old format (individual metric keys at top level). It also handles both old and new file naming conventions in model folders.

Parameters:

Name	Type	Description	Default
path	str	Path to a model folder or metrics file (.npz).	required
split	str	Name of the split to load. Must be "train", "val", or "test". Default: "test". If "test" is not found, falls back to "val". Ignored if path points directly to a .npz file.	'test'
dataset_idx	int	Index of the dataset (for multi-dataset training). Default: 0. Ignored if path points directly to a .npz file.	0

Returns:

Type	Description
dict	Dictionary containing metrics with keys: voc_metrics, mOKS, distance_metrics, pck_metrics, visibility_metrics.

Raises:

Type	Description
FileNotFoundError	If no metrics file is found.

Examples:

>>> # Load from model folder (tries test, falls back to val)
>>> metrics = load_metrics("/path/to/model")
>>> print(metrics["mOKS"]["mOKS"])

>>> # Load specific split and dataset
>>> metrics = load_metrics("/path/to/model", split="val", dataset_idx=1)

>>> # Load directly from npz file
>>> metrics = load_metrics("/path/to/metrics.val.0.npz")

Source code in sleap_nn/evaluation.py

def load_metrics(
    path: str,
    split: str = "test",
    dataset_idx: int = 0,
) -> dict:
    """Load metrics from a model folder or metrics file.

    This function supports both the new format (single "metrics" key) and the old
    format (individual metric keys at top level). It also handles both old and new
    file naming conventions in model folders.

    Args:
        path: Path to a model folder or metrics file (.npz).
        split: Name of the split to load. Must be "train", "val", or "test".
            Default: "test". If "test" is not found, falls back to "val".
            Ignored if path points directly to a .npz file.
        dataset_idx: Index of the dataset (for multi-dataset training).
            Default: 0. Ignored if path points directly to a .npz file.

    Returns:
        Dictionary containing metrics with keys: voc_metrics, mOKS,
        distance_metrics, pck_metrics, visibility_metrics.

    Raises:
        FileNotFoundError: If no metrics file is found.

    Examples:
        >>> # Load from model folder (tries test, falls back to val)
        >>> metrics = load_metrics("/path/to/model")
        >>> print(metrics["mOKS"]["mOKS"])

        >>> # Load specific split and dataset
        >>> metrics = load_metrics("/path/to/model", split="val", dataset_idx=1)

        >>> # Load directly from npz file
        >>> metrics = load_metrics("/path/to/metrics.val.0.npz")
    """
    path = Path(path)

    if path.suffix == ".npz":
        metrics_path = path
    else:
        metrics_path = _find_metrics_file(path, split, dataset_idx)

    if not metrics_path.exists():
        raise FileNotFoundError(f"Metrics file not found at {metrics_path}")

    return _load_npz_metrics(metrics_path)

mask_cldice(pred, gt)

Centerline Dice (clDice) between two binary masks.

Shit et al., "clDice — A Novel Topology-Preserving Loss Function for Tubular Structure Segmentation," CVPR 2021 (arXiv:2003.07311). The connectivity-aware F-score of two skeleton-overlap terms:

• Tprec = fraction of the predicted skeleton lying inside the GT mask (is my centerline drawn on a real object?),
• Tsens = fraction of the GT skeleton lying inside the predicted mask (did I cover every real object along its length?),

with clDice = 2·Tprec·Tsens / (Tprec + Tsens). Nearly width-insensitive and connectivity-sensitive, so it is a fairer quality measure than area IoU for thin/tubular structures (roots, vessels, neurites). Uses a hard morphological skeleton (exact, no k to tune).

Two empty masks return 1.0 (matching the _mask_iou "identical -> 1.0" contract). Returns nan when scikit-image is unavailable so callers can drop clDice from the summary without failing.

Source code in sleap_nn/evaluation.py

def mask_cldice(pred: np.ndarray, gt: np.ndarray) -> float:
    """Centerline Dice (clDice) between two binary masks.

    Shit et al., "clDice — A Novel Topology-Preserving Loss Function for Tubular
    Structure Segmentation," CVPR 2021 (arXiv:2003.07311). The connectivity-aware
    F-score of two skeleton-overlap terms:

    * ``Tprec`` = fraction of the *predicted* skeleton lying inside the *GT* mask
      (is my centerline drawn on a real object?),
    * ``Tsens`` = fraction of the *GT* skeleton lying inside the *predicted* mask
      (did I cover every real object along its length?),

    with ``clDice = 2·Tprec·Tsens / (Tprec + Tsens)``. Nearly width-insensitive
    and connectivity-sensitive, so it is a fairer quality measure than area IoU
    for thin/tubular structures (roots, vessels, neurites). Uses a hard
    morphological skeleton (exact, no ``k`` to tune).

    Two empty masks return ``1.0`` (matching the ``_mask_iou`` "identical -> 1.0"
    contract). Returns ``nan`` when scikit-image is unavailable so callers can
    drop clDice from the summary without failing.
    """
    a, b = _align_pair(pred, gt)
    if not a.any() and not b.any():
        return 1.0
    sk_p = _skeletonize(a)
    sk_g = _skeletonize(b)
    if sk_p is None or sk_g is None:
        return float("nan")
    sp, sg = int(sk_p.sum()), int(sk_g.sum())
    if sp == 0 or sg == 0:
        return 0.0
    tprec = int(np.logical_and(sk_p, b).sum()) / sp
    tsens = int(np.logical_and(sk_g, a).sum()) / sg
    if (tprec + tsens) == 0:
        return 0.0
    return float(2.0 * tprec * tsens / (tprec + tsens))

match_centroids(pred_centroids, gt_centroids, max_distance=50.0)

Match predicted centroids to ground truth using Hungarian algorithm.

Parameters:

Name	Type	Description	Default
pred_centroids	ndarray	Predicted centroid locations, shape (n_pred, 2).	required
gt_centroids	ndarray	Ground truth centroid locations, shape (n_gt, 2).	required
max_distance	float	Maximum distance threshold for valid matches (in pixels).	50.0

Returns:

Type	Description
tuple	Tuple of: - matched_pred_indices: Indices of matched predictions - matched_gt_indices: Indices of matched ground truth - unmatched_pred_indices: Indices of unmatched predictions (false positives) - unmatched_gt_indices: Indices of unmatched ground truth (false negatives)

Source code in sleap_nn/evaluation.py

def match_centroids(
    pred_centroids: "np.ndarray",
    gt_centroids: "np.ndarray",
    max_distance: float = 50.0,
) -> tuple:
    """Match predicted centroids to ground truth using Hungarian algorithm.

    Args:
        pred_centroids: Predicted centroid locations, shape (n_pred, 2).
        gt_centroids: Ground truth centroid locations, shape (n_gt, 2).
        max_distance: Maximum distance threshold for valid matches (in pixels).

    Returns:
        Tuple of:
            - matched_pred_indices: Indices of matched predictions
            - matched_gt_indices: Indices of matched ground truth
            - unmatched_pred_indices: Indices of unmatched predictions (false positives)
            - unmatched_gt_indices: Indices of unmatched ground truth (false negatives)
    """
    import numpy as np
    from scipy.optimize import linear_sum_assignment
    from scipy.spatial.distance import cdist

    n_pred = len(pred_centroids)
    n_gt = len(gt_centroids)

    # Handle edge cases
    if n_pred == 0 and n_gt == 0:
        return np.array([]), np.array([]), np.array([]), np.array([])
    if n_pred == 0:
        return np.array([]), np.array([]), np.array([]), np.arange(n_gt)
    if n_gt == 0:
        return np.array([]), np.array([]), np.arange(n_pred), np.array([])

    # Compute pairwise distances
    cost_matrix = cdist(pred_centroids, gt_centroids)

    # Run Hungarian algorithm for optimal matching
    pred_indices, gt_indices = linear_sum_assignment(cost_matrix)

    # Filter matches that exceed max_distance
    matched_pred = []
    matched_gt = []
    for p_idx, g_idx in zip(pred_indices, gt_indices):
        if cost_matrix[p_idx, g_idx] <= max_distance:
            matched_pred.append(p_idx)
            matched_gt.append(g_idx)

    matched_pred = np.array(matched_pred)
    matched_gt = np.array(matched_gt)

    # Find unmatched indices
    all_pred = set(range(n_pred))
    all_gt = set(range(n_gt))
    unmatched_pred = np.array(list(all_pred - set(matched_pred)))
    unmatched_gt = np.array(list(all_gt - set(matched_gt)))

    return matched_pred, matched_gt, unmatched_pred, unmatched_gt

match_frame_pairs(frame_pairs, stddev=0.025, scale=None, threshold=0)

Match all ground truth and predicted instances within each pair of frames.

This is a wrapper for match_instances() but operates on lists of frames.

Parameters:

Name	Type	Description	Default
frame_pairs	List[Tuple[LabeledFrame, LabeledFrame]]	A list of pairs of sleap.LabeledFrames in the form (frame_gt, frame_pr). These can be obtained with find_frame_pairs().	required
stddev	float	The expected spread of coordinates for OKS computation.	0.025
scale	Optional[float]	The scale for normalizing the OKS. If not set, the bounding box area will be used.	None
threshold	float	The minimum OKS between a candidate pair of instances to be considered a match.	0

Returns:

Type	Description
Tuple[List[Tuple[Instance, PredictedInstance, float]], List[Instance]]	A tuple of (positive_pairs, false_negatives). positive_pairs is a list of 3-tuples of the form (instance_gt, instance_pr, oks) containing the matched pair of instances and their OKS. false_negatives is a list of ground truth sio.Instances that could not be matched.

Source code in sleap_nn/evaluation.py

def match_frame_pairs(
    frame_pairs: List[Tuple[sio.LabeledFrame, sio.LabeledFrame]],
    stddev: float = 0.025,
    scale: Optional[float] = None,
    threshold: float = 0,
) -> Tuple[List[Tuple[sio.Instance, sio.PredictedInstance, float]], List[sio.Instance]]:
    """Match all ground truth and predicted instances within each pair of frames.

    This is a wrapper for `match_instances()` but operates on lists of frames.

    Args:
        frame_pairs: A list of pairs of `sleap.LabeledFrame`s in the form
            `(frame_gt, frame_pr)`. These can be obtained with `find_frame_pairs()`.
        stddev: The expected spread of coordinates for OKS computation.
        scale: The scale for normalizing the OKS. If not set, the bounding box area will
            be used.
        threshold: The minimum OKS between a candidate pair of instances to be
            considered a match.

    Returns:
        A tuple of (`positive_pairs`, `false_negatives`).

        `positive_pairs` is a list of 3-tuples of the form
        `(instance_gt, instance_pr, oks)` containing the matched pair of instances and
        their OKS.

        `false_negatives` is a list of ground truth `sio.Instance`s that could not be
        matched.
    """
    positive_pairs = []
    false_negatives = []
    for frame_gt, frame_pr in frame_pairs:
        positive_pairs_frame, false_negatives_frame = match_instances(
            frame_gt,
            frame_pr,
            stddev=stddev,
            scale=scale,
            threshold=threshold,
        )
        positive_pairs.extend(positive_pairs_frame)
        false_negatives.extend(false_negatives_frame)

    return positive_pairs, false_negatives

match_instances(frame_gt, frame_pr, stddev=0.025, scale=None, threshold=0, degenerate_pixel_threshold=50.0)

Match pairs of instances between ground truth and predictions in a frame.

Parameters:

Name	Type	Description	Default
frame_gt	LabeledFrame	A sio.LabeledFrame with ground truth instances.	required
frame_pr	LabeledFrame	A sio.LabeledFrame with predicted instances.	required
stddev	float	The expected spread of coordinates for OKS computation.	0.025
scale	Optional[float]	The scale for normalizing the OKS. If not set, the bounding box area will be used.	None
threshold	float	The minimum OKS between a candidate pair of instances to be considered a match.	0
degenerate_pixel_threshold	float	Pixel distance threshold used to score GT instances whose visible-keypoint bounding box has zero area (see compute_distance_match_score), in place of OKS.	50.0

Returns:

Type	Description
Tuple[List[Tuple[Instance, PredictedInstance, float]], List[Instance]]	A tuple of (positive_pairs, false_negatives). positive_pairs is a list of 3-tuples of the form (instance_gt, instance_pr, oks) containing the matched pair of instances and their OKS. false_negatives is a list of ground truth sleap.Instances that could not be matched.

Notes

This function uses the approach from the PASCAL VOC scoring procedure. Briefly, predictions are sorted descending by their instance-level prediction scores and greedily matched to ground truth instances which are then removed from the pool of available instances.

Ground truth instances that remain unmatched are considered false negatives.

Source code in sleap_nn/evaluation.py

def match_instances(
    frame_gt: sio.LabeledFrame,
    frame_pr: sio.LabeledFrame,
    stddev: float = 0.025,
    scale: Optional[float] = None,
    threshold: float = 0,
    degenerate_pixel_threshold: float = 50.0,
) -> Tuple[List[Tuple[sio.Instance, sio.PredictedInstance, float]], List[sio.Instance]]:
    """Match pairs of instances between ground truth and predictions in a frame.

    Args:
        frame_gt: A `sio.LabeledFrame` with ground truth instances.
        frame_pr: A `sio.LabeledFrame` with predicted instances.
        stddev: The expected spread of coordinates for OKS computation.
        scale: The scale for normalizing the OKS. If not set, the bounding box area will
            be used.
        threshold: The minimum OKS between a candidate pair of instances to be
            considered a match.
        degenerate_pixel_threshold: Pixel distance threshold used to score GT
            instances whose visible-keypoint bounding box has zero area (see
            `compute_distance_match_score`), in place of OKS.

    Returns:
        A tuple of (`positive_pairs`, `false_negatives`).

        `positive_pairs` is a list of 3-tuples of the form
        `(instance_gt, instance_pr, oks)` containing the matched pair of instances and
        their OKS.

        `false_negatives` is a list of ground truth `sleap.Instance`s that could not be
        matched.

    Notes:
        This function uses the approach from the PASCAL VOC scoring procedure. Briefly,
        predictions are sorted descending by their instance-level prediction scores and
        greedily matched to ground truth instances which are then removed from the pool
        of available instances.

        Ground truth instances that remain unmatched are considered false negatives.
    """
    # Sort predicted instances by score.
    frame_pr_match_instances = get_instances(frame_pr)

    scores_pr = np.array(
        [
            m.instance.score
            for m in frame_pr_match_instances
            if hasattr(m.instance, "score")
        ]
    )
    idxs_pr = np.argsort(-scores_pr, kind="mergesort")  # descending
    scores_pr = scores_pr[idxs_pr]

    available_instances_gt = get_instances(frame_gt)
    available_instances_gt_idxs = list(range(len(available_instances_gt)))

    positive_pairs = []
    for idx_pr in idxs_pr:
        # Pull out predicted instance.
        instance_pr = frame_pr_match_instances[idx_pr]

        # Convert instances to point arrays.
        points_pr = np.expand_dims(instance_pr.instance.numpy(), axis=0)
        points_gt = np.stack(
            [
                available_instances_gt[idx].instance.numpy()
                for idx in available_instances_gt_idxs
            ],
            axis=0,
        )

        # Find the best match by computing OKS.
        oks = compute_oks(points_gt, points_pr, stddev=stddev, scale=scale)
        oks = np.squeeze(oks, axis=1)
        assert oks.shape == (len(points_gt),)

        # GT instances with a zero-area visible-keypoint bbox make OKS collapse into a
        # strict equality test (see `_DEGENERATE_AREA_EPS`). Score those against this
        # prediction by pixel distance instead.
        degenerate = compute_instance_area(points_gt) < _DEGENERATE_AREA_EPS
        if degenerate.any():
            distance_scores = compute_distance_match_score(
                points_gt[degenerate],
                points_pr,
                pixel_threshold=degenerate_pixel_threshold,
            )
            oks[degenerate] = np.squeeze(distance_scores, axis=1)

        oks[oks <= threshold] = np.nan
        best_match_gt_idx = np.argsort(-oks, kind="mergesort")[0]
        best_match_oks = oks[best_match_gt_idx]
        if np.isnan(best_match_oks):
            continue

        # Remove matched ground truth instance and add as a positive pair.
        instance_gt_idx = available_instances_gt_idxs.pop(best_match_gt_idx)
        instance_gt = available_instances_gt[instance_gt_idx]
        positive_pairs.append((instance_gt, instance_pr, best_match_oks))

        # Stop matching lower scoring instances if we run out of candidates in the
        # ground truth.
        if not available_instances_gt_idxs:
            break

    # Any remaining ground truth instances are considered false negatives.
    false_negatives = [
        available_instances_gt[idx] for idx in available_instances_gt_idxs
    ]

    return positive_pairs, false_negatives

match_masks(pred_masks, gt_masks, min_iou=0.5)

Match predicted masks to ground-truth masks by IoU (Hungarian).

Parameters:

Name	Type	Description	Default
pred_masks	List[ndarray]	List of boolean arrays, one per predicted instance.	required
gt_masks	List[ndarray]	List of boolean arrays, one per ground-truth instance.	required
min_iou	float	Minimum IoU for a matched pair to count as a true positive.	0.5

Returns:

Type	Description
tuple	Tuple of: - matched_pred_indices: Indices of matched predictions. - matched_gt_indices: Indices of matched ground truth. - unmatched_pred_indices: Unmatched predictions (false positives). - unmatched_gt_indices: Unmatched ground truth (false negatives). - matched_ious: IoU of each matched pair, aligned to matched_pred_indices.

Source code in sleap_nn/evaluation.py

def match_masks(
    pred_masks: List[np.ndarray],
    gt_masks: List[np.ndarray],
    min_iou: float = 0.5,
) -> tuple:
    """Match predicted masks to ground-truth masks by IoU (Hungarian).

    Args:
        pred_masks: List of boolean arrays, one per predicted instance.
        gt_masks: List of boolean arrays, one per ground-truth instance.
        min_iou: Minimum IoU for a matched pair to count as a true positive.

    Returns:
        Tuple of:
            - matched_pred_indices: Indices of matched predictions.
            - matched_gt_indices: Indices of matched ground truth.
            - unmatched_pred_indices: Unmatched predictions (false positives).
            - unmatched_gt_indices: Unmatched ground truth (false negatives).
            - matched_ious: IoU of each matched pair, aligned to
              ``matched_pred_indices``.
    """
    from scipy.optimize import linear_sum_assignment

    n_pred = len(pred_masks)
    n_gt = len(gt_masks)
    empty = np.array([], dtype=int)
    if n_pred == 0 and n_gt == 0:
        return empty, empty, empty, empty, np.array([])
    if n_pred == 0:
        return empty, empty, empty, np.arange(n_gt), np.array([])
    if n_gt == 0:
        return empty, empty, np.arange(n_pred), empty, np.array([])

    iou = _mask_iou_matrix(pred_masks, gt_masks)  # (n_pred, n_gt)
    # Maximize total IoU -> minimize negative IoU.
    pred_indices, gt_indices = linear_sum_assignment(-iou)

    matched_pred, matched_gt, matched_ious = [], [], []
    for p_idx, g_idx in zip(pred_indices, gt_indices):
        if iou[p_idx, g_idx] >= min_iou:
            matched_pred.append(int(p_idx))
            matched_gt.append(int(g_idx))
            matched_ious.append(float(iou[p_idx, g_idx]))

    matched_pred = np.array(matched_pred, dtype=int)
    matched_gt = np.array(matched_gt, dtype=int)
    unmatched_pred = np.array(
        sorted(set(range(n_pred)) - set(matched_pred.tolist())), dtype=int
    )
    unmatched_gt = np.array(
        sorted(set(range(n_gt)) - set(matched_gt.tolist())), dtype=int
    )
    return (
        matched_pred,
        matched_gt,
        unmatched_pred,
        unmatched_gt,
        np.array(matched_ious),
    )

motion_diagnostic(labels, carrier='pose')

Judge whether a labels file is continuous video or temporally sparse samples.

Identity metrics are meaningless on a sparse set, and nothing in the file says so. Embedded .pkg.slp training splits renumber their frames 0..N-1 and record frame_numbers as contiguous, so every index-based contiguity check passes -- while the animal has actually moved across the arena between two "consecutive" frames. Run this before quoting a tracking number on an unfamiliar file.

The decisive quantity is how far the same animal moves per frame relative to its own size. Measured on real files, step_over_size lands near 0.01-0.06 for genuine video and 3-9 for sparse training splits -- and at the high end same-animal consecutive mask IoU is 0.000 for most pairs, so geometric association has no signal to work with and any IoU tracker must fail.

The same animal is followed by its track, so the labels must be tracked (tracked ground truth, or the output of a tracking run). Each video is followed separately, and a step between two detections k frames apart is divided by k: a ground truth labeled every 10^th frame of a continuous video is judged by its per-frame motion, not flagged as sparse, while a .pkg.slp split -- renumbered so every gap reads as 1 -- still is. So median_step_px is the typical per-frame displacement of one animal, which is what --euclidean_scale asks for.

Parameters:

Name	Type	Description	Default
labels	Labels	Tracked labels to inspect.	required
carrier	str	"pose" (instance keypoints) or "mask" (segmentation masks); decides how a detection's center and size are measured.	'pose'

Returns:

Type	Description
Dict[str, Any]	Dict with median_step_px (per frame), median_size_px, step_over_size, median_frame_gap and is_continuous (step_over_size < 0.5). When too few tracked detections are present to judge, step_over_size is NaN, is_continuous is False and a note explains why.

Source code in sleap_nn/evaluation.py

def motion_diagnostic(labels: sio.Labels, carrier: str = "pose") -> Dict[str, Any]:
    """Judge whether a labels file is continuous video or temporally sparse samples.

    **Identity metrics are meaningless on a sparse set, and nothing in the file
    says so.** Embedded ``.pkg.slp`` training splits renumber their frames
    ``0..N-1`` and record ``frame_numbers`` as contiguous, so every index-based
    contiguity check passes -- while the animal has actually moved across the
    arena between two "consecutive" frames. Run this before quoting a tracking
    number on an unfamiliar file.

    The decisive quantity is how far the same animal moves per frame relative to
    its own size. Measured on real files, ``step_over_size`` lands near
    ``0.01-0.06`` for genuine video and ``3-9`` for sparse training splits -- and
    at the high end same-animal consecutive mask IoU is ``0.000`` for most pairs,
    so geometric association has no signal to work with and any IoU tracker must
    fail.

    The same animal is followed by its track, **so the labels must be tracked**
    (tracked ground truth, or the output of a tracking run). Each video is
    followed separately, and a step between two detections ``k`` frames apart is
    divided by ``k``: a ground truth labeled every 10th frame of a continuous
    video is judged by its per-frame motion, not flagged as sparse, while a
    ``.pkg.slp`` split -- renumbered so every gap reads as 1 -- still is. So
    ``median_step_px`` is the typical per-frame displacement of one animal,
    which is what ``--euclidean_scale`` asks for.

    Args:
        labels: Tracked labels to inspect.
        carrier: ``"pose"`` (instance keypoints) or ``"mask"`` (segmentation
            masks); decides how a detection's center and size are measured.

    Returns:
        Dict with ``median_step_px`` (per frame), ``median_size_px``,
        ``step_over_size``, ``median_frame_gap`` and ``is_continuous``
        (``step_over_size < 0.5``). When too few tracked detections are present
        to judge, ``step_over_size`` is NaN, ``is_continuous`` is False and a
        ``note`` explains why.
    """
    carrier = _validate_carrier(carrier)
    # (video, track name) -> (frame_idx, center) of the track's last detection.
    prev: Dict[Tuple[str, str], Tuple[int, np.ndarray]] = {}
    steps: List[float] = []
    gaps: List[int] = []
    sizes: List[float] = []

    frames = sorted(labels, key=lambda lf: (_video_key(lf.video), lf.frame_idx))
    for frame in frames:
        video_key = _video_key(frame.video)
        dets = _identity_dets(frame, carrier)
        tracks = _identity_tracks(frame, carrier, dets)
        items: List[Tuple[str, np.ndarray, float]] = []
        if carrier == "mask":
            for arr, track in zip(_frame_masks(frame), tracks):
                if track is None:
                    continue
                ys, xs = np.nonzero(arr)
                if not len(xs):
                    continue
                items.append(
                    (
                        track.name,
                        np.array([xs.mean(), ys.mean()]),
                        # Equivalent-circle diameter of the mask.
                        2.0 * float(np.sqrt(arr.sum() / np.pi)),
                    )
                )
        else:
            for det, track in zip(dets, tracks):
                if track is None:
                    continue
                pts = np.asarray(det.numpy(), dtype=float)[:, :2]
                pts = pts[~np.isnan(pts).any(axis=1)]
                if not len(pts):
                    continue
                items.append(
                    (track.name, pts.mean(axis=0), float(np.ptp(pts, axis=0).max()))
                )

        for name, center, size in items:
            sizes.append(size)
            key = (video_key, name)
            if key in prev:
                prev_idx, prev_center = prev[key]
                gap = frame.frame_idx - prev_idx
                if gap > 0:
                    steps.append(float(np.linalg.norm(center - prev_center)) / gap)
                    gaps.append(gap)
            prev[key] = (frame.frame_idx, center)

    if not steps or not sizes:
        return {
            "median_step_px": float("nan"),
            "median_size_px": float("nan"),
            "step_over_size": float("nan"),
            "median_frame_gap": float("nan"),
            "is_continuous": False,
            "note": "not enough tracked detections to judge",
        }

    median_step = float(np.median(steps))
    median_size = float(np.median(sizes))
    median_gap = float(np.median(gaps))
    if median_size <= 0.0:
        # A single-node skeleton (a centroid model) or coincident nodes have no
        # measurable extent, so there is nothing to normalize the step against.
        # Report "cannot judge" rather than dividing by ~0 and calling every
        # centroid prediction sparse.
        return {
            "median_step_px": round(median_step, 2),
            "median_size_px": median_size,
            "step_over_size": float("nan"),
            "median_frame_gap": median_gap,
            "is_continuous": False,
            "note": (
                "detections have no measurable extent (single-node skeleton?) -- "
                "cannot judge continuity"
            ),
        }
    ratio = median_step / median_size
    return {
        "median_step_px": round(median_step, 2),
        "median_size_px": round(median_size, 2),
        "step_over_size": round(ratio, 3),
        "median_frame_gap": median_gap,
        "is_continuous": bool(ratio < 0.5),
    }

retrieval_metrics(gallery_emb, gallery_y, query_emb, query_y)

Rank-1 (CMC@1) + mAP of queries against a gallery (cosine similarity).

A query whose identity is absent from the gallery has nothing to retrieve: it is left out of both metrics (NaN when no query has a positive) and counted in n_no_positive_queries.

Source code in sleap_nn/evaluation.py

def retrieval_metrics(gallery_emb, gallery_y, query_emb, query_y):
    """Rank-1 (CMC@1) + mAP of queries against a gallery (cosine similarity).

    A query whose identity is absent from the gallery has nothing to retrieve: it is
    left out of both metrics (NaN when no query has a positive) and counted in
    ``n_no_positive_queries``.
    """
    g, q = np.asarray(gallery_emb), np.asarray(query_emb)
    gy, qy = np.asarray(gallery_y), np.asarray(query_y)
    has_pos = np.isin(qy, gy)
    if _nonfinite_embeddings(g, q):
        nan = float("nan")
        return {
            "rank1": nan,
            "mAP": nan,
            "n_no_positive_queries": int((~has_pos).sum()),
        }
    g, q = _l2_normalize(g), _l2_normalize(q)
    top1, ap, _ = _score_queries(q @ g.T, qy, gy, k=1)
    return {
        "rank1": _mean_or_nan(top1[has_pos]),
        "mAP": _mean_or_nan(ap[has_pos]),
        "n_no_positive_queries": int((~has_pos).sum()),
    }

run_evaluation(ground_truth_path, predicted_path, oks_stddev=0.025, oks_scale=None, match_threshold=0, user_labels_only=True, save_metrics=None, match_method='oks', anchor_part=None, centroid_method=None, centroid_fallback=None, bbox_source='auto', bbox_padding=0.0)

Evaluate SLEAP-NN model predictions against ground truth labels.

Parameters:

Name	Type	Description	Default
ground_truth_path	str	Path to the ground-truth .slp file.	required
predicted_path	str	Path to the predicted .slp file.	required
oks_stddev	float	OKS standard deviation (OKS mode only).	0.025
oks_scale	Optional[float]	OKS scale override (OKS mode only).	None
match_threshold	float	Matching threshold. OKS threshold for OKS mode; PIXEL distance for centroid mode. In centroid mode, if the caller leaves the OKS default of 0.0 it is bumped to 50.0 px.	0
user_labels_only	bool	If False, predicted instances in the GT frame may be matched. For match_method="mask" (default True), this additionally drops masks linked to a PredictedInstance from the ground-truth labels, so stray predicted instances (each of which gets a mask when masks are built per-instance from poses) are not treated as ground truth. Pass False when the GT is intentionally built from predicted poses (pseudo-mask GT). The whole-frame match_method="semantic" union is unaffected either way.	True
save_metrics	Optional[str]	Optional .npz path to save metrics to.	None
match_method	str	"oks", "centroid", "mask", "semantic", or "auto". "mask" matches predicted vs GT segmentation masks by IoU (for bottomup_segmentation models). "semantic" unions each frame's masks into one foreground and scores IoU/clDice/boundary-IoU with NO matching (for whole-frame semantic_segmentation models). "auto" switches to centroid mode when the PREDICTION skeleton is a single-node skeleton (e.g. sio.get_centroid_skeleton()); it never auto-selects "mask" or "semantic" (pass those explicitly). "bbox" scores PredictedBoundingBoxes against the GT's user boxes with COCO box AP (for the bbox_* detectors); a GT file with no boxes has them derived from its poses (or masks) first.	'oks'
anchor_part	Optional[str]	Name of the GT skeleton node used to compute GT centroids (centroid mode). Resolved against the GT skeleton; None (or an absent name) falls back to the mean of visible nodes (#586).	None
centroid_method	Optional[str]	How GT centroids are derived (centroid mode) -- "center_of_mass", "bbox_center", "geometric_median" or "anchor". None (default) infers it from anchor_part. Pass the value the model was TRAINED with (its head_configs.centroid.confmaps.centroid_method), or the distance metric scores predictions against a different centroid than the one they were trained to predict.	None
centroid_fallback	Optional[str]	Reduce method used when the anchor node is not visible.	None
bbox_source	str	"bbox" mode only: where GT boxes come from (auto, bboxes, instances, masks; pass the model's data_config.bbox_source).	'auto'
bbox_padding	float	"bbox" mode only: pixels of padding on GT boxes derived from poses/masks (pass the model's data_config.bbox_padding).	0.0

Returns:

Type	Description
	The metrics dict, or None if the predicted labels have zero frames or contain nothing usable (no instances for "oks"/ "centroid"/"auto", no masks for "mask"/"semantic") -- metric computation is skipped entirely in that case, and no save_metrics file is written.

Source code in sleap_nn/evaluation.py

def run_evaluation(
    ground_truth_path: str,
    predicted_path: str,
    oks_stddev: float = 0.025,
    oks_scale: Optional[float] = None,
    match_threshold: float = 0,
    user_labels_only: bool = True,
    save_metrics: Optional[str] = None,
    match_method: str = "oks",
    anchor_part: Optional[str] = None,
    centroid_method: Optional[str] = None,
    centroid_fallback: Optional[str] = None,
    bbox_source: str = "auto",
    bbox_padding: float = 0.0,
):
    """Evaluate SLEAP-NN model predictions against ground truth labels.

    Args:
        ground_truth_path: Path to the ground-truth ``.slp`` file.
        predicted_path: Path to the predicted ``.slp`` file.
        oks_stddev: OKS standard deviation (OKS mode only).
        oks_scale: OKS scale override (OKS mode only).
        match_threshold: Matching threshold. OKS threshold for OKS mode; PIXEL
            distance for centroid mode. In centroid mode, if the caller leaves
            the OKS default of ``0.0`` it is bumped to ``50.0`` px.
        user_labels_only: If False, predicted instances in the GT frame may be
            matched. For ``match_method="mask"`` (default True), this additionally
            drops masks linked to a ``PredictedInstance`` from the ground-truth
            labels, so stray predicted instances (each of which gets a mask when
            masks are built per-instance from poses) are not treated as ground
            truth. Pass ``False`` when the GT is intentionally built from predicted
            poses (pseudo-mask GT). The whole-frame ``match_method="semantic"`` union is
            unaffected either way.
        save_metrics: Optional ``.npz`` path to save metrics to.
        match_method: ``"oks"``, ``"centroid"``, ``"mask"``, ``"semantic"``, or
            ``"auto"``. ``"mask"`` matches predicted vs GT segmentation masks by
            IoU (for ``bottomup_segmentation`` models). ``"semantic"`` unions each
            frame's masks into one foreground and scores IoU/clDice/boundary-IoU
            with NO matching (for whole-frame ``semantic_segmentation`` models).
            ``"auto"`` switches to centroid mode when the PREDICTION skeleton is a
            single-node skeleton (e.g. ``sio.get_centroid_skeleton()``); it never
            auto-selects ``"mask"`` or ``"semantic"`` (pass those explicitly).
            ``"bbox"`` scores ``PredictedBoundingBox``es against the GT's user
            boxes with COCO box AP (for the ``bbox_*`` detectors); a GT file with
            no boxes has them derived from its poses (or masks) first.
        anchor_part: Name of the GT skeleton node used to compute GT centroids
            (centroid mode). Resolved against the GT skeleton; ``None`` (or an
            absent name) falls back to the mean of visible nodes (#586).
        centroid_method: How GT centroids are derived (centroid mode) --
            ``"center_of_mass"``, ``"bbox_center"``, ``"geometric_median"`` or
            ``"anchor"``. ``None`` (default) infers it from ``anchor_part``. Pass
            the value the model was TRAINED with (its
            ``head_configs.centroid.confmaps.centroid_method``), or the distance
            metric scores predictions against a different centroid than the one
            they were trained to predict.
        centroid_fallback: Reduce method used when the anchor node is not visible.
        bbox_source: ``"bbox"`` mode only: where GT boxes come from (``auto``,
            ``bboxes``, ``instances``, ``masks``; pass the model's
            ``data_config.bbox_source``).
        bbox_padding: ``"bbox"`` mode only: pixels of padding on GT boxes derived
            from poses/masks (pass the model's ``data_config.bbox_padding``).

    Returns:
        The metrics dict, or ``None`` if the predicted labels have zero
        frames or contain nothing usable (no instances for ``"oks"``/
        ``"centroid"``/``"auto"``, no masks for ``"mask"``/``"semantic"``) --
        metric computation is skipped entirely in that case, and no
        ``save_metrics`` file is written.
    """
    logger.info("Loading ground truth labels...")
    ground_truth_instances = sio.load_slp(ground_truth_path)
    logger.info(
        f"  Ground truth: {len(ground_truth_instances.videos)} videos, "
        f"{len(ground_truth_instances.labeled_frames)} frames"
    )

    logger.info("Loading predicted labels...")
    predicted_instances = sio.load_slp(predicted_path)
    logger.info(
        f"  Predictions: {len(predicted_instances.videos)} videos, "
        f"{len(predicted_instances.labeled_frames)} frames"
    )

    if (
        match_method == "auto"
        and not any(len(lf.instances) for lf in predicted_instances)
        and any(b.is_predicted for lf in predicted_instances for b in lf.bboxes)
    ):
        match_method = "bbox"
        logger.info("Auto-detected bbox mode (box-only predictions).")
    if match_method == "bbox":
        return _run_bbox_evaluation(
            ground_truth_instances,
            predicted_instances,
            save_metrics=save_metrics,
            bbox_source=bbox_source,
            bbox_padding=bbox_padding,
        )

    # Detect a fully collapsed prediction set up front and skip the metric
    # math entirely (#719) -- frames may still be present (both predictor
    # pipelines retain empty-detection frames by default), but nothing usable
    # was predicted in any of them, so matching would only produce an
    # all-NaN/all-zero result. ``mask``/``semantic`` predictions live on
    # ``LabeledFrame.masks``, not ``.instances``.
    if match_method in ("mask", "semantic"):
        has_predictions = any(len(lf.masks) for lf in predicted_instances)
    else:
        has_predictions = any(len(lf.instances) for lf in predicted_instances)
    if not len(predicted_instances) or not has_predictions:
        logger.info(
            "0 predicted instances: skipping metric computation (model "
            "likely predicted nothing usable, or training collapsed)."
        )
        return None

    # Auto-detect centroid mode from the PREDICTION skeleton.
    pred_skeleton = (
        predicted_instances.skeletons[0] if predicted_instances.skeletons else None
    )
    if match_method == "auto":
        if _is_single_node_skeleton(pred_skeleton):
            match_method = "centroid"
            logger.info(
                "Auto-detected centroid mode (single-node prediction skeleton)."
            )
        else:
            match_method = "oks"

    # Resolve the anchor node against the GT skeleton (mirror predictor.py).
    gt_skeleton = (
        ground_truth_instances.skeletons[0]
        if ground_truth_instances.skeletons
        else None
    )
    anchor_ind = _resolve_anchor_ind(gt_skeleton, anchor_part)

    # In centroid mode, default the (pixel) match threshold to 50.0 if the
    # caller left the OKS default of 0.0.
    if match_method == "centroid" and match_threshold == 0:
        match_threshold = 50.0

    # In mask mode, default the IoU match threshold to 0.5 if the caller left
    # the OKS default of 0.0.
    if match_method == "mask" and match_threshold == 0:
        match_threshold = 0.5

    # Mask eval matches GT vs predicted MASKS (on ``frame.masks``), independent of
    # whether the frame's keypoint instances are user- or predicted-labeled. The
    # ``user_labels_only`` frame filter (find_frame_pairs) keeps only frames with
    # USER keypoint instances, which silently drops EVERY frame when the GT was
    # built from predicted poses (e.g. pseudo-mask GT from predicted skeletons),
    # raising "Empty Frame Pairs". Mask mode therefore never applies that FRAME
    # filter. The caller's ``user_labels_only`` intent is preserved separately to
    # govern the ground-truth MASK filter: a labels file with stray
    # PredictedInstances gives each a mask (masks are built per-instance), and under
    # user-only labels those must not be treated as ground truth (they would be
    # spurious false negatives that cap recall). Callers evaluating pseudo-mask GT
    # pass ``user_labels_only=False``.
    exclude_predicted_instance_masks = user_labels_only
    if match_method in ("mask", "semantic"):
        user_labels_only = False

    logger.info("Matching videos and frames...")
    # Get match stats before creating evaluator
    match_result = ground_truth_instances.match(predicted_instances)
    logger.info(
        f"  Videos matched: {match_result.n_videos_matched}/{len(match_result.video_map)}"
    )

    logger.info("Matching instances...")
    evaluator = Evaluator(
        ground_truth_instances=ground_truth_instances,
        predicted_instances=predicted_instances,
        oks_stddev=oks_stddev,
        oks_scale=oks_scale,
        match_threshold=match_threshold,
        user_labels_only=user_labels_only,
        match_method=match_method,
        anchor_ind=anchor_ind,
        centroid_method=centroid_method,
        centroid_fallback=centroid_fallback,
        exclude_predicted_instance_masks=exclude_predicted_instance_masks,
    )
    logger.info(
        f"  Frame pairs: {len(evaluator.frame_pairs)}, "
        f"Matched instances: {len(evaluator.positive_pairs)}, "
        f"Unmatched GT: {len(evaluator.false_negatives)}"
    )

    logger.info("Computing evaluation metrics...")
    metrics = evaluator.evaluate()

    if match_method == "centroid":
        # Centroid mode: report detection + distance metrics only (no
        # oks_voc.*/mOKS/PCK/visibility keys exist).
        det = metrics["detection_metrics"]
        dist = metrics["distance_metrics"]
        logger.info("Evaluation Results (centroid mode):")
        logger.info(f"  Precision: {det['precision']:.4f}")
        logger.info(f"  Recall: {det['recall']:.4f}")
        logger.info(f"  F1: {det['f1']:.4f}")
        logger.info(f"  Counts: TP={det['n_tp']}, FP={det['n_fp']}, FN={det['n_fn']}")
        logger.info(f"  Average Distance: {dist['avg']:.2f} px")
        logger.info(f"  dist.p50: {dist['p50']:.2f} px")
        logger.info(f"  dist.p90: {dist['p90']:.2f} px")
        logger.info(f"  dist.p95: {dist['p95']:.2f} px")
        logger.info(f"  dist.p99: {dist['p99']:.2f} px")

        if save_metrics:
            logger.info(f"Saving metrics to {save_metrics}...")
            save_path = Path(save_metrics)
            # Writes the pickled ``.npz`` (back-compat) plus a JSON sibling
            # with the same stem so the app can read metrics without unpickling.
            _write_metrics(save_path, metrics)
            logger.info(f"Metrics saved successfully to {save_path}")

        return metrics

    if match_method == "mask":
        # Mask mode: report detection (IoU-matched) + mask-IoU quality + COCO
        # mask AP/AR (no oks_voc.*/mOKS/PCK/visibility keys exist).
        det = metrics["detection_metrics"]
        mm = metrics["mask_metrics"]
        mvoc = metrics["mask_voc_metrics"]
        logger.info("Evaluation Results (mask mode):")
        logger.info(f"  Precision: {det['precision']:.4f}")
        logger.info(f"  Recall: {det['recall']:.4f}")
        logger.info(f"  F1: {det['f1']:.4f}")
        logger.info(f"  Counts: TP={det['n_tp']}, FP={det['n_fp']}, FN={det['n_fn']}")
        logger.info(f"  Mean mask IoU: {mm['mean_iou']:.4f}")
        logger.info(f"  mask IoU p50: {mm['p50']:.4f}")
        logger.info(f"  mask IoU p25: {mm['p25']:.4f}")
        logger.info(f"  Mean boundary IoU: {mm['mean_boundary_iou']:.4f}")
        logger.info(f"  Mean clDice (centerline): {mm['mean_cldice']:.4f}")
        logger.info(f"  mAP @[.5:.95]: {mvoc['mask_voc.mAP']:.4f}")
        logger.info(
            f"  AP50: {mvoc['mask_voc.AP50']:.4f}  AP75: {mvoc['mask_voc.AP75']:.4f}"
        )
        logger.info(f"  AR @[.5:.95]: {mvoc['mask_voc.AR']:.4f}")
        e0, e1 = mvoc["mask_voc.size_edges"]
        logger.info(
            f"  AP by size [percentile, edges={e0:.0f}/{e1:.0f} px^2]: "
            f"S={mvoc['mask_voc.AP_small']:.4f} "
            f"M={mvoc['mask_voc.AP_medium']:.4f} L={mvoc['mask_voc.AP_large']:.4f} "
            f"(GT S/M/L={mvoc['mask_voc.n_gt_small']}/"
            f"{mvoc['mask_voc.n_gt_medium']}/{mvoc['mask_voc.n_gt_large']})"
        )
        logger.info(
            f"  AP by size [COCO 1024/9216 px^2]: "
            f"S={mvoc['mask_voc.coco.AP_small']:.4f} "
            f"M={mvoc['mask_voc.coco.AP_medium']:.4f} "
            f"L={mvoc['mask_voc.coco.AP_large']:.4f} "
            f"(GT S/M/L={mvoc['mask_voc.coco.n_gt_small']}/"
            f"{mvoc['mask_voc.coco.n_gt_medium']}/{mvoc['mask_voc.coco.n_gt_large']})"
        )
        logger.info(
            f"  Fragmentation: oversegmentation={mm['oversegmentation']}, "
            f"undersegmentation={mm['undersegmentation']}"
        )

        if save_metrics:
            logger.info(f"Saving metrics to {save_metrics}...")
            save_path = Path(save_metrics)
            # Writes the pickled ``.npz`` (back-compat) plus a JSON sibling
            # with the same stem so the app can read metrics without unpickling.
            _write_metrics(save_path, metrics)
            logger.info(f"Metrics saved successfully to {save_path}")

        return metrics

    if match_method == "semantic":
        # Semantic (whole-frame foreground) mode: matching-free IoU / clDice /
        # boundary-IoU only (no detection / mask-AP keys exist).
        sm = metrics["semantic_metrics"]
        logger.info("Evaluation Results (semantic / whole-frame foreground mode):")
        logger.info(f"  Frames scored (non-empty GT fg): {sm['n_frames']}")
        logger.info(f"  Mean foreground IoU: {sm['mean_iou']:.4f}")
        logger.info(f"  Mean clDice (centerline): {sm['mean_cldice']:.4f}")
        logger.info(f"  Mean boundary IoU: {sm['mean_boundary_iou']:.4f}")

        if save_metrics:
            logger.info(f"Saving metrics to {save_metrics}...")
            save_path = Path(save_metrics)
            # Writes the pickled ``.npz`` (back-compat) plus a JSON sibling
            # with the same stem so the app can read metrics without unpickling.
            _write_metrics(save_path, metrics)
            logger.info(f"Metrics saved successfully to {save_path}")

        return metrics

    # Compute PCK at specific thresholds (5 and 10 pixels)
    dists = metrics["distance_metrics"]["dists"]
    dists_clean = np.copy(dists)
    dists_clean[np.isnan(dists_clean)] = np.inf
    # Guard the empty-match case (0 matched instances for the whole split) so
    # this doesn't hit "Mean of empty slice" on top of the evaluate()-level
    # log line already emitted for it.
    pck_5 = float((dists_clean < 5).mean()) if dists_clean.size else np.nan
    pck_10 = float((dists_clean < 10).mean()) if dists_clean.size else np.nan

    # Print key metrics
    logger.info("Evaluation Results:")
    logger.info(f"  mOKS: {metrics['mOKS']['mOKS']:.4f}")
    logger.info(f"  mAP (OKS VOC): {metrics['voc_metrics']['oks_voc.mAP']:.4f}")
    logger.info(f"  mAR (OKS VOC): {metrics['voc_metrics']['oks_voc.mAR']:.4f}")
    logger.info(f"  Average Distance: {metrics['distance_metrics']['avg']:.2f} px")
    logger.info(f"  dist.p50: {metrics['distance_metrics']['p50']:.2f} px")
    logger.info(f"  dist.p95: {metrics['distance_metrics']['p95']:.2f} px")
    logger.info(f"  dist.p99: {metrics['distance_metrics']['p99']:.2f} px")
    logger.info(f"  mPCK: {metrics['pck_metrics']['mPCK']:.4f}")
    logger.info(f"  PCK@5px: {pck_5:.4f}")
    logger.info(f"  PCK@10px: {pck_10:.4f}")
    logger.info(
        f"  Visibility Precision: {metrics['visibility_metrics']['precision']:.4f}"
    )
    logger.info(f"  Visibility Recall: {metrics['visibility_metrics']['recall']:.4f}")

    # Save metrics if path provided
    if save_metrics:
        logger.info(f"Saving metrics to {save_metrics}...")
        save_path = Path(save_metrics)

        # Save metrics in SLEAP 1.4 format (single "metrics" key) plus a JSON
        # sibling (same stem) that the app metrics UI can read without
        # unpickling the numpy object array.
        _write_metrics(save_path, metrics)
        logger.info(f"Metrics saved successfully to {save_path}")

    return metrics

run_identity_evaluation(ground_truth_path, predicted_path, carrier='auto', match_threshold=None, mt_threshold=0.8, ml_threshold=0.2, user_labels_only=False, global_identity=False, save_metrics=None, raise_if_empty=False, oks_stddev=0.025)

Evaluate identity persistence of a tracked prediction against tracked GT.

Computes two blocks over the same matching: sleap-nn's identity metrics (:func:identity_metrics, legacy definitions) and TrackEval-faithful CLEAR-MOT / IDF1 / HOTA (:func:compute_mot_metrics). Where the two name the same quantity (ID switches, IDF1, MT/PT/ML, fragmentation) they can differ; see :class:IdentityMetrics for how. Quote the "mot" block when comparing with published MOT results.

Parameters:

Name	Type	Description	Default
ground_truth_path	str	Path to the ground-truth .slp file, tracked.	required
predicted_path	str	Path to the predicted .slp file, tracked.	required
carrier	str	"pose", "mask", or "auto" -- "auto" picks the carrier the prediction's tracks are on: the one holding more tracked detections (ties go to "pose"), via :func:sleap_nn.inference.tracking.count_carriers.	'auto'
match_threshold	Optional[float]	Minimum OKS (pose) or IoU (mask), or the maximum centroid distance in pixels for single-node skeletons, for a detection pair to count as matched. None: 0.5 / 0.5 / 50 px.	None
mt_threshold	float	Mostly-tracked coverage cut.	0.8
ml_threshold	float	Mostly-lost coverage cut.	0.2
user_labels_only	bool	Drop model output from the ground-truth side; off by default (see :func:identity_metrics for why).	False
global_identity	bool	Key trajectories by track name across videos rather than per video (see :func:identity_metrics).	False
save_metrics	Optional[str]	Optional .json path to write the metrics to.	None
raise_if_empty	bool	Raise :class:NothingToScoreError instead of returning None when nothing is scoreable (the CLI sets it, to exit non-zero).	False
oks_stddev	float	Per-keypoint OKS spread for pose matching, in both blocks; a finite number > 0. Default 0.025.	0.025

Returns:

Type

Description

Optional[Dict[str, Any]]

Dict with the :class:IdentityMetrics fields plus carrier, motion_diagnostic and mot (the :class:MOTMetrics fields, or {"error": message} if the MOT metrics could not be computed -- a track on two detections of one frame), or None if nothing is scoreable: no tracked predicted detection on the carrier, no frame in common with the ground truth, or no tracked ground-truth detection in the frames compared. Nothing is saved in that case.

Raises:

Type	Description
ValueError	If oks_stddev is not a finite number greater than 0 (checked before either file is loaded), or the carrier is unknown.

Source code in sleap_nn/evaluation.py

def run_identity_evaluation(
    ground_truth_path: str,
    predicted_path: str,
    carrier: str = "auto",
    match_threshold: Optional[float] = None,
    mt_threshold: float = 0.8,
    ml_threshold: float = 0.2,
    user_labels_only: bool = False,
    global_identity: bool = False,
    save_metrics: Optional[str] = None,
    raise_if_empty: bool = False,
    oks_stddev: float = 0.025,
) -> Optional[Dict[str, Any]]:
    """Evaluate identity persistence of a tracked prediction against tracked GT.

    Computes two blocks over the same matching: sleap-nn's identity metrics
    (:func:`identity_metrics`, legacy definitions) and TrackEval-faithful
    CLEAR-MOT / IDF1 / HOTA (:func:`compute_mot_metrics`). Where the two name
    the same quantity (ID switches, IDF1, MT/PT/ML, fragmentation) they can
    differ; see :class:`IdentityMetrics` for how. Quote the ``"mot"`` block
    when comparing with published MOT results.

    Args:
        ground_truth_path: Path to the ground-truth ``.slp`` file, tracked.
        predicted_path: Path to the predicted ``.slp`` file, tracked.
        carrier: ``"pose"``, ``"mask"``, or ``"auto"`` -- ``"auto"`` picks the
            carrier the prediction's tracks are on: the one holding more tracked
            detections (ties go to ``"pose"``), via
            :func:`sleap_nn.inference.tracking.count_carriers`.
        match_threshold: Minimum OKS (pose) or IoU (mask), or the maximum
            centroid distance in pixels for single-node skeletons, for a
            detection pair to count as matched. ``None``: 0.5 / 0.5 / 50 px.
        mt_threshold: Mostly-tracked coverage cut.
        ml_threshold: Mostly-lost coverage cut.
        user_labels_only: Drop model output from the ground-truth side; off by
            default (see :func:`identity_metrics` for why).
        global_identity: Key trajectories by track name across videos rather
            than per video (see :func:`identity_metrics`).
        save_metrics: Optional ``.json`` path to write the metrics to.
        raise_if_empty: Raise :class:`NothingToScoreError` instead of returning
            ``None`` when nothing is scoreable (the CLI sets it, to exit
            non-zero).
        oks_stddev: Per-keypoint OKS spread for pose matching, in both blocks;
            a finite number > 0. Default 0.025.

    Returns:
        Dict with the :class:`IdentityMetrics` fields plus ``carrier``,
        ``motion_diagnostic`` and ``mot`` (the :class:`MOTMetrics` fields, or
        ``{"error": message}`` if the MOT metrics could not be computed -- a
        track on two detections of one frame), or ``None`` if nothing is
        scoreable: no tracked
        predicted detection on the carrier, no frame in common with the ground
        truth, or no tracked ground-truth detection in the frames compared.
        Nothing is saved in that case.

    Raises:
        ValueError: If ``oks_stddev`` is not a finite number greater than 0
            (checked before either file is loaded), or the carrier is unknown.
    """
    from sleap_nn.inference.tracking import count_carriers

    oks_stddev = _check_oks_stddev(oks_stddev)

    def nothing_to_score(reason: str) -> None:
        logger.error(reason)
        if raise_if_empty:
            raise NothingToScoreError(reason)
        return None

    logger.info("Loading ground truth labels...")
    gt_labels = sio.load_slp(ground_truth_path)
    logger.info(
        f"  Ground truth: {len(gt_labels.videos)} videos, "
        f"{len(gt_labels.labeled_frames)} frames"
    )

    logger.info("Loading predicted labels...")
    pred_labels = sio.load_slp(predicted_path)
    logger.info(
        f"  Predictions: {len(pred_labels.videos)} videos, "
        f"{len(pred_labels.labeled_frames)} frames"
    )

    def has_track(det: Any) -> bool:
        return getattr(det, "track", None) is not None

    pred_counts = count_carriers(pred_labels, has_track)
    if carrier == "auto":
        # The carrier the tracks are ON, counted -- not "masks and no instances":
        # a pose + mask file tracked on its masks would otherwise be scored on
        # its untracked poses and reported as untracked.
        carrier = pred_counts.dominant
        if carrier is None:
            return nothing_to_score(
                "Nothing to score: the prediction has 0 tracked detections on "
                "either carrier (pose or mask). Identity metrics score tracks -- "
                "run sleap-nn track (or predict with -t) first."
            )
        logger.info(
            f"Auto-detected carrier: {carrier} (tracked detections: "
            f"{pred_counts.n_pose_with} pose, {pred_counts.n_mask_with} mask)."
        )
    carrier = _validate_carrier(carrier)
    other = "mask" if carrier == "pose" else "pose"

    n_tracked = sum(
        1
        for lf in pred_labels
        for track in _identity_tracks(lf, carrier, _identity_dets(lf, carrier))
        if track is not None
    )
    if not n_tracked:
        hint = (
            f"The {other} carrier has {pred_counts.n_with(other)} tracked "
            f"detection(s) -- pass --carrier {other}."
            if pred_counts.n_with(other)
            else "Identity metrics score tracks -- run sleap-nn track (or predict "
            "with -t) first."
        )
        return nothing_to_score(
            f"Nothing to score: the prediction has 0 tracked {carrier} "
            f"detections. {hint}"
        )

    # The sparse-split trap: a `.pkg.slp` training split renumbers its frames
    # contiguously, so it *looks* like video while the animal teleports between
    # "consecutive" frames. Say so loudly rather than reporting a meaningless
    # switch count.
    motion = motion_diagnostic(gt_labels, carrier)
    if np.isnan(motion["step_over_size"]):
        logger.info(f"Continuity check inconclusive: {motion.get('note', '')}")
    elif not motion["is_continuous"]:
        logger.warning(
            "Ground truth does not look like continuous video "
            f"(step/size = {motion['step_over_size']} per frame). Identity metrics "
            "on a temporally sparse set (e.g. a `.pkg.slp` training split) are not "
            "meaningful -- score a real video clip instead."
        )
    if motion["median_frame_gap"] > 1:
        logger.info(
            f"Ground truth is labeled every ~{motion['median_frame_gap']:g} frames; "
            "continuity was judged per frame. The scores assume the prediction was "
            "tracked over the whole video, not only the labeled frames."
        )

    metrics = identity_metrics(
        gt_labels,
        pred_labels,
        carrier,
        match_threshold=match_threshold,
        mt_threshold=mt_threshold,
        ml_threshold=ml_threshold,
        user_labels_only=user_labels_only,
        global_identity=global_identity,
        oks_stddev=oks_stddev,
    )

    logger.info(
        f"Identity (legacy definitions) ({carrier} carrier, matched by "
        f"{_threshold_phrase(metrics.match_method, metrics.match_threshold)}; "
        f"{'global identities' if global_identity else 'per video'}):"
    )
    logger.info(f"  {metrics.summary()}")
    for note in metrics.notes:
        logger.warning(f"  note: {note}")

    if not metrics.n_frames_compared:
        return nothing_to_score(
            "Nothing to score: the prediction shares no frame with the ground "
            "truth (compared by video path and frame index)."
        )
    if not metrics.n_gt_dets:
        gt_counts = count_carriers(gt_labels, has_track)
        hint = (
            f" The ground truth has {gt_counts.n_with(other)} tracked {other} "
            f"detection(s) -- pass --carrier {other}."
            if gt_counts.n_with(other)
            else " Check that the ground truth carries tracks."
        )
        return nothing_to_score(
            f"Nothing to score: no tracked ground-truth {carrier} detections in "
            f"the {metrics.n_frames_compared} frames compared.{hint}"
        )

    # TrackEval-faithful CLEAR-MOT / IDF1 / HOTA over the same frames and
    # matcher. A file TrackEval would refuse (a track on two detections of one
    # frame) still gets the legacy block above; the MOT block says why not.
    mot: Dict[str, Any]
    try:
        mot_scores = compute_mot_metrics(
            gt_labels,
            pred_labels,
            carrier,
            match_threshold=match_threshold,
            oks_stddev=oks_stddev,
            user_labels_only=user_labels_only,
            global_identity=global_identity,
        )
    except ValueError as e:
        logger.error(f"TrackEval-faithful MOT metrics not computed: {e}")
        mot = {"error": str(e)}
    else:
        logger.info(
            "TrackEval-faithful MOT (CLEAR-MOT / IDF1 / HOTA; HOTA averaged over "
            "alpha = 0.05..0.95):"
        )
        logger.info(f"  {mot_scores.summary()}")
        for note in mot_scores.notes:
            if note not in metrics.notes:
                logger.warning(f"  note: {note}")
        mot = mot_scores.as_dict()

    result = metrics.as_dict()
    result["carrier"] = carrier
    result["motion_diagnostic"] = motion
    result["mot"] = mot

    if save_metrics:
        save_path = Path(save_metrics)
        save_path.parent.mkdir(parents=True, exist_ok=True)
        with open(save_path, "w") as f:
            json.dump(_metrics_to_json_safe(result), f, indent=2)
        logger.info(f"Metrics saved successfully to {save_path}")

    return result

verification_metrics(gallery_emb, gallery_y, query_emb, query_y, exclude_diagonal=False)

ROC-AUC + EER over all query x gallery pairs (same vs different identity).

When exclude_diagonal (gallery == query in the same order), the self-pairs on the similarity diagonal are dropped before scoring so a leave-self-out evaluation is not optimistically biased by N perfect same-identity matches at sim=1.0. See :func:_auc_eer for the definitions.

Source code in sleap_nn/evaluation.py

def verification_metrics(
    gallery_emb, gallery_y, query_emb, query_y, exclude_diagonal: bool = False
):
    """ROC-AUC + EER over all query x gallery pairs (same vs different identity).

    When ``exclude_diagonal`` (gallery == query in the same order), the self-pairs on
    the similarity diagonal are dropped before scoring so a leave-self-out evaluation is
    not optimistically biased by ``N`` perfect same-identity matches at sim=1.0.
    See :func:`_auc_eer` for the definitions.
    """
    g, q = np.asarray(gallery_emb), np.asarray(query_emb)
    if _nonfinite_embeddings(g, q):
        return {"auc": float("nan"), "eer": float("nan")}
    g, q = _l2_normalize(g), _l2_normalize(q)
    gy, qy = np.asarray(gallery_y), np.asarray(query_y)
    sim = q @ g.T
    same = qy[:, None] == gy[None, :]
    keep = np.ones_like(same)
    if exclude_diagonal:
        np.fill_diagonal(keep, False)
    pos, neg = np.sort(sim[same & keep]), np.sort(sim[~same & keep])
    auc, eer = _auc_eer(pos, neg)
    return {"auc": round(auc, 4), "eer": round(eer, 4)}