mot_metrics

sleap_nn.tracking.mot_metrics

TrackEval-faithful multi-object tracking metrics: CLEAR-MOT, Identity and HOTA.

A pure numpy/scipy core (no sleap-io): it scores sequences of per-timestep ground-truth ids, predicted ids and a ground-truth x prediction similarity matrix. Turning sio.Labels into such sequences -- matching, don't-care ground truth, frame alignment -- is the job of :func:sleap_nn.evaluation.compute_mot_metrics.

The numbers reproduce TrackEval 1.3.0 (Luiten & Hoffhues; the reference implementation of HOTA, and the MOTChallenge / KITTI evaluation code) exactly, including its documented quirks, so they can be quoted next to published MOT results. Output keys use TrackEval's own field names (MOTA, IDF1, HOTA, CLR_TP, HOTA(0), ...).

Three metric families are computed:

• CLEAR-MOT (Bernardin & Stiefelhagen 2008): per-timestep Hungarian matching at similarity >= threshold that first keeps the previous timestep's matches (the "continuity bonus"), then maximizes similarity. Gives TP / FN / FP, ID switches, MOTA / MOTP / MODA, mostly-tracked / partly-tracked / mostly-lost and fragmentations.
• Identity (Ristani et al. 2016): one global assignment between ground-truth and predicted identities, maximizing the number of timesteps on which the pair could match (similarity >= threshold; every such pair counts, not only a one-to-one per-timestep matching). Gives IDTP / IDFN / IDFP and IDF1 / IDP / IDR.
• HOTA (Luiten et al. 2021): detection, association and localization accuracy over 19 localization thresholds alpha = 0.05, 0.10, ..., 0.95 (:data:HOTA_ALPHAS). Like TrackEval -- and unlike the paper's Eq. 15 -- one alpha-independent Hungarian matching per timestep maximizes global alignment score x similarity, and that single matching is then thresholded at each alpha. Array fields hold one value per alpha; the headline number quoted in papers is their mean.

TrackEval behaviors reproduced on purpose (each one changes a number):

• Continuity bonus looks one timestep back; ID switches look any distance back. Matching prefers the prediction matched to the same ground truth at the immediately previous timestep, while a switch is scored against the last prediction a ground truth was matched to, however long ago.
• Fragmentation (Frag) is, per ground-truth id, the number of tracked segments minus one. A timestep with no prediction at all does NOT end a segment (TrackEval skips such timesteps without resetting the continuity state), while a timestep with only unrelated predictions does: a one-frame gap with nothing predicted gives Frag = 0, the same gap holding a single false positive gives Frag = 1.
• Mostly tracked is strictly above 80 % of a trajectory's timesteps; mostly lost is below 20 %.
• LocA is 1 where there are no true positives (TrackEval divides max(1e-10, sum) / max(1e-10, TP)).
• HOTA(0) / LocA(0) are the values at alpha = 0.05, the first threshold -- not at alpha = 0.
• Empty sequences short-circuit. A sequence with no predictions (or no ground truth) returns only the counts -- all ground truth missed (or all predictions false) -- with every derived field left at 0 except MLR = 1 and LocA = 1, and CLR_Frames = 0. :func:combine_sequences recomputes the derived fields from the summed counts, as TrackEval's COMBINED_SEQ row does.

Ids must be unique within a timestep. They need not be contiguous: every metric relabels them to 0..n-1 first (:func:relabel_contiguous).

Classes:

Name	Description
MOTSequence	One scored sequence: per-timestep ids and similarities.

Functions:

Name	Description
clear_metrics	Compute the CLEAR-MOT metrics of one sequence, as TrackEval does.
combine_sequences	Combine per-sequence results into one, as TrackEval's COMBINED_SEQ.
hota_metrics	Compute the HOTA metrics of one sequence, as TrackEval does.
id_metrics	Compute the identity metrics (IDF1, Ristani et al.) of one sequence.
mot_metrics	Compute CLEAR-MOT, Identity and HOTA for one sequence.
relabel_contiguous	Relabel ground-truth and predicted ids to contiguous 0..n-1.

MOTSequence

One scored sequence: per-timestep ids and similarities.

A sequence is normally one video. Under global identities several videos are concatenated into one sequence, with resets marking where each new video starts.

Construction normalizes and validates every timestep: ids become 1-D int64 arrays, similarities (n_gt, n_pred) float arrays (an empty similarity may be given in any shape), and resets a sorted tuple of unique ints.

Attributes:

Name	Type	Description
gt_ids	List[ndarray]	One integer array of ground-truth ids per timestep. Ids are arbitrary integers, unique within a timestep.
pred_ids	List[ndarray]	One integer array of predicted ids per timestep, same rules.
similarity	List[ndarray]	One (len(gt_ids[t]), len(pred_ids[t])) array per timestep: the similarity of every ground-truth/predicted pair, in [0, 1] (OKS, IoU, ...). Must be finite.
resets	Tuple[int, ...]	Timestep indices at which CLEAR-MOT's frame-to-frame state is reset -- video boundaries when several videos are scored as one sequence. At a reset the continuity bonus and the ID-switch memory are cleared (a ground truth's first match in the new video is never a switch), and fragmentation is counted per side of the boundary (a trajectory's first segment after a reset is not a fragment). Identity and HOTA do not depend on timestep order and ignore it.

Raises:

Type	Description
ValueError	If the three lists differ in length, a similarity matrix has the wrong shape or a non-finite value, ids are not integers, an id occurs twice in one timestep, or a reset is out of range.

Methods:

Name	Description
__attrs_post_init__	Normalize and validate every timestep.

Source code in sleap_nn/tracking/mot_metrics.py

@attrs.define
class MOTSequence:
    """One scored sequence: per-timestep ids and similarities.

    A sequence is normally one video. Under global identities several videos
    are concatenated into one sequence, with ``resets`` marking where each new
    video starts.

    Construction normalizes and validates every timestep: ids become 1-D int64
    arrays, similarities ``(n_gt, n_pred)`` float arrays (an empty similarity
    may be given in any shape), and ``resets`` a sorted tuple of unique ints.

    Attributes:
        gt_ids: One integer array of ground-truth ids per timestep. Ids are
            arbitrary integers, unique within a timestep.
        pred_ids: One integer array of predicted ids per timestep, same rules.
        similarity: One ``(len(gt_ids[t]), len(pred_ids[t]))`` array per
            timestep: the similarity of every ground-truth/predicted pair, in
            ``[0, 1]`` (OKS, IoU, ...). Must be finite.
        resets: Timestep indices at which CLEAR-MOT's frame-to-frame state is
            reset -- video boundaries when several videos are scored as one
            sequence. At a reset the continuity bonus and the ID-switch memory
            are cleared (a ground truth's first match in the new video is never
            a switch), and fragmentation is counted per side of the boundary
            (a trajectory's first segment after a reset is not a fragment).
            Identity and HOTA do not depend on timestep order and ignore it.

    Raises:
        ValueError: If the three lists differ in length, a similarity matrix
            has the wrong shape or a non-finite value, ids are not integers, an
            id occurs twice in one timestep, or a reset is out of range.
    """

    gt_ids: List[np.ndarray]
    pred_ids: List[np.ndarray]
    similarity: List[np.ndarray]
    resets: Tuple[int, ...] = ()

    def __attrs_post_init__(self):
        """Normalize and validate every timestep."""
        n = len(self.gt_ids)
        if len(self.pred_ids) != n or len(self.similarity) != n:
            raise ValueError(
                "gt_ids, pred_ids and similarity must have one entry per timestep, "
                f"got {n}, {len(self.pred_ids)} and {len(self.similarity)}."
            )
        gt_ids, pred_ids, similarity = [], [], []
        for t in range(n):
            g = _ids_array(self.gt_ids[t], "ground-truth", t)
            p = _ids_array(self.pred_ids[t], "predicted", t)
            _check_unique(g, "ground-truth", t)
            _check_unique(p, "predicted", t)
            s = np.asarray(self.similarity[t], dtype=float)
            if s.size == 0 and len(g) * len(p) == 0:
                s = np.zeros((len(g), len(p)))
            elif s.shape != (len(g), len(p)):
                raise ValueError(
                    f"timestep {t}: similarity has shape {s.shape}, expected "
                    f"({len(g)}, {len(p)}) for {len(g)} ground-truth and "
                    f"{len(p)} predicted ids."
                )
            if not np.isfinite(s).all():
                raise ValueError(
                    f"timestep {t}: similarity has non-finite values; map NaN to 0 "
                    "before scoring."
                )
            gt_ids.append(g)
            pred_ids.append(p)
            similarity.append(s)
        resets = tuple(sorted({int(r) for r in self.resets}))
        if resets and (resets[0] < 0 or resets[-1] >= max(n, 1)):
            raise ValueError(
                f"resets must be timestep indices in [0, {n}), got {list(resets)}."
            )
        self.gt_ids = gt_ids
        self.pred_ids = pred_ids
        self.similarity = similarity
        self.resets = resets

__attrs_post_init__()

Normalize and validate every timestep.

Source code in sleap_nn/tracking/mot_metrics.py

def __attrs_post_init__(self):
    """Normalize and validate every timestep."""
    n = len(self.gt_ids)
    if len(self.pred_ids) != n or len(self.similarity) != n:
        raise ValueError(
            "gt_ids, pred_ids and similarity must have one entry per timestep, "
            f"got {n}, {len(self.pred_ids)} and {len(self.similarity)}."
        )
    gt_ids, pred_ids, similarity = [], [], []
    for t in range(n):
        g = _ids_array(self.gt_ids[t], "ground-truth", t)
        p = _ids_array(self.pred_ids[t], "predicted", t)
        _check_unique(g, "ground-truth", t)
        _check_unique(p, "predicted", t)
        s = np.asarray(self.similarity[t], dtype=float)
        if s.size == 0 and len(g) * len(p) == 0:
            s = np.zeros((len(g), len(p)))
        elif s.shape != (len(g), len(p)):
            raise ValueError(
                f"timestep {t}: similarity has shape {s.shape}, expected "
                f"({len(g)}, {len(p)}) for {len(g)} ground-truth and "
                f"{len(p)} predicted ids."
            )
        if not np.isfinite(s).all():
            raise ValueError(
                f"timestep {t}: similarity has non-finite values; map NaN to 0 "
                "before scoring."
            )
        gt_ids.append(g)
        pred_ids.append(p)
        similarity.append(s)
    resets = tuple(sorted({int(r) for r in self.resets}))
    if resets and (resets[0] < 0 or resets[-1] >= max(n, 1)):
        raise ValueError(
            f"resets must be timestep indices in [0, {n}), got {list(resets)}."
        )
    self.gt_ids = gt_ids
    self.pred_ids = pred_ids
    self.similarity = similarity
    self.resets = resets

clear_metrics(seq, threshold=0.5)

Compute the CLEAR-MOT metrics of one sequence, as TrackEval does.

Parameters:

Name	Type	Description	Default
seq	MOTSequence	The sequence to score. Ids are relabeled contiguously first.	required
threshold	float	Minimum similarity for a ground-truth/predicted pair to match (TrackEval's THRESHOLD, applied with one float epsilon of slack).	0.5

Returns:

Type	Description
Dict[str, Any]	Dict keyed by TrackEval's field names: the integer counts CLR_TP, CLR_FN, CLR_FP, IDSW, MT, PT, ML, Frag, CLR_Frames and the floats MOTA, MOTP, MODA, sMOTA, CLR_Re, CLR_Pr, CLR_F1, MTR, PTR, MLR, FP_per_frame, MOTAL, MOTP_sum.

Source code in sleap_nn/tracking/mot_metrics.py

def clear_metrics(seq: MOTSequence, threshold: float = 0.5) -> Dict[str, Any]:
    """Compute the CLEAR-MOT metrics of one sequence, as TrackEval does.

    Args:
        seq: The sequence to score. Ids are relabeled contiguously first.
        threshold: Minimum similarity for a ground-truth/predicted pair to
            match (TrackEval's ``THRESHOLD``, applied with one float epsilon of
            slack).

    Returns:
        Dict keyed by TrackEval's field names: the integer counts
        ``CLR_TP``, ``CLR_FN``, ``CLR_FP``, ``IDSW``, ``MT``, ``PT``, ``ML``,
        ``Frag``, ``CLR_Frames`` and the floats ``MOTA``, ``MOTP``, ``MODA``,
        ``sMOTA``, ``CLR_Re``, ``CLR_Pr``, ``CLR_F1``, ``MTR``, ``PTR``,
        ``MLR``, ``FP_per_frame``, ``MOTAL``, ``MOTP_sum``.
    """
    return _clear(relabel_contiguous(seq), float(threshold))

combine_sequences(per_seq)

Combine per-sequence results into one, as TrackEval's COMBINED_SEQ.

Counts are summed and every derived field recomputed from the sums -- so the combined MOTA / IDF1 / DetA are detection-weighted, not means of the per-sequence values. HOTA's association and localization scores (AssA, AssRe, AssPr, LocA) are averages weighted by each sequence's HOTA_TP. Combining one sequence recomputes its derived fields, which differs from the per-sequence result only for an empty sequence (e.g. its MOTA becomes -FP instead of 0).

Parameters:

Name	Type	Description	Default
per_seq	Sequence[Dict[str, Any]]	Results of :func:mot_metrics (or of any of :func:clear_metrics, :func:id_metrics, :func:hota_metrics; every result must hold the same families).	required

Returns:

Type	Description
Dict[str, Any]	One dict with the same keys.

Raises:

Type	Description
ValueError	If per_seq is empty, the results hold different metric families, or their HOTA arrays differ in length.

Source code in sleap_nn/tracking/mot_metrics.py

def combine_sequences(per_seq: Sequence[Dict[str, Any]]) -> Dict[str, Any]:
    """Combine per-sequence results into one, as TrackEval's ``COMBINED_SEQ``.

    Counts are summed and every derived field recomputed from the sums -- so
    the combined MOTA / IDF1 / DetA are detection-weighted, not means of the
    per-sequence values. HOTA's association and localization scores (``AssA``,
    ``AssRe``, ``AssPr``, ``LocA``) are averages weighted by each sequence's
    ``HOTA_TP``. Combining one sequence recomputes its derived fields, which
    differs from the per-sequence result only for an empty sequence (e.g. its
    MOTA becomes ``-FP`` instead of 0).

    Args:
        per_seq: Results of :func:`mot_metrics` (or of any of
            :func:`clear_metrics`, :func:`id_metrics`, :func:`hota_metrics`;
            every result must hold the same families).

    Returns:
        One dict with the same keys.

    Raises:
        ValueError: If ``per_seq`` is empty, the results hold different metric
            families, or their HOTA arrays differ in length.
    """
    per_seq = list(per_seq)
    if not per_seq:
        raise ValueError("combine_sequences needs at least one result.")
    out: Dict[str, Any] = {}

    if _has_block(per_seq, "CLR_TP"):
        res: Dict[str, Any] = {
            f: int(sum(int(r[f]) for r in per_seq)) for f in CLEAR_INTEGER_FIELDS
        }
        res["MOTP_sum"] = float(sum(float(r["MOTP_sum"]) for r in per_seq))
        out.update(_clear_final_fields(res))

    if _has_block(per_seq, "IDTP"):
        res = {f: int(sum(int(r[f]) for r in per_seq)) for f in ID_INTEGER_FIELDS}
        out.update(_id_final_fields(res))

    if _has_block(per_seq, "HOTA_TP"):
        lengths = {len(np.asarray(r["HOTA_TP"])) for r in per_seq}
        if len(lengths) != 1:
            raise ValueError(
                f"HOTA results were computed over different alphas (lengths "
                f"{sorted(lengths)}); they cannot be combined."
            )
        res = {
            f: np.sum([np.asarray(r[f], dtype=float) for r in per_seq], axis=0)
            for f in HOTA_COUNT_FIELDS
        }
        weight = np.maximum(1.0, res["HOTA_TP"])
        for f in ("AssRe", "AssPr", "AssA"):
            res[f] = (
                np.sum(
                    [
                        np.asarray(r[f], dtype=float) * np.asarray(r["HOTA_TP"])
                        for r in per_seq
                    ],
                    axis=0,
                )
                / weight
            )
        loc_weighted = np.sum(
            [
                np.asarray(r["LocA"], dtype=float) * np.asarray(r["HOTA_TP"])
                for r in per_seq
            ],
            axis=0,
        )
        res["LocA"] = np.maximum(1e-10, loc_weighted) / np.maximum(
            1e-10, res["HOTA_TP"]
        )
        out.update(_hota_final_fields(res))

    return out

hota_metrics(seq, alphas=HOTA_ALPHAS)

Compute the HOTA metrics of one sequence, as TrackEval does.

Parameters:

Name	Type	Description	Default
seq	MOTSequence	The sequence to score. Ids are relabeled contiguously first.	required
alphas	Any	Localization thresholds. Default :data:HOTA_ALPHAS, the 19 values 0.05, 0.10, ..., 0.95 every published HOTA uses.	HOTA_ALPHAS

Returns:

Type	Description
Dict[str, Any]	Dict with one value per alpha (float arrays) for HOTA, DetA, AssA, DetRe, DetPr, AssRe, AssPr, LocA, OWTA and the counts HOTA_TP, HOTA_FN, HOTA_FP; plus the floats HOTA(0), LocA(0) (the values at the FIRST alpha, 0.05 by default) and HOTALocA(0) (their product). Quote the mean over alphas of an array field as the headline value.

Source code in sleap_nn/tracking/mot_metrics.py

def hota_metrics(seq: MOTSequence, alphas: Any = HOTA_ALPHAS) -> Dict[str, Any]:
    """Compute the HOTA metrics of one sequence, as TrackEval does.

    Args:
        seq: The sequence to score. Ids are relabeled contiguously first.
        alphas: Localization thresholds. Default :data:`HOTA_ALPHAS`, the 19
            values ``0.05, 0.10, ..., 0.95`` every published HOTA uses.

    Returns:
        Dict with one value per alpha (float arrays) for ``HOTA``, ``DetA``,
        ``AssA``, ``DetRe``, ``DetPr``, ``AssRe``, ``AssPr``, ``LocA``,
        ``OWTA`` and the counts ``HOTA_TP``, ``HOTA_FN``, ``HOTA_FP``; plus the
        floats ``HOTA(0)``, ``LocA(0)`` (the values at the FIRST alpha, 0.05 by
        default) and ``HOTALocA(0)`` (their product). Quote the mean over
        alphas of an array field as the headline value.
    """
    return _hota(relabel_contiguous(seq), _check_alphas(alphas))

id_metrics(seq, threshold=0.5)

Compute the identity metrics (IDF1, Ristani et al.) of one sequence.

Named to avoid clashing with :func:sleap_nn.evaluation.identity_metrics, the older sleap-nn definitions (which differ from TrackEval, see :class:sleap_nn.evaluation.IdentityMetrics).

Parameters:

Name	Type	Description	Default
seq	MOTSequence	The sequence to score. Ids are relabeled contiguously first.	required
threshold	float	Minimum similarity for a pair to count as a potential match.	0.5

Returns:

Type	Description
Dict[str, Any]	Dict with the integer counts IDTP, IDFN, IDFP and the floats IDF1, IDP, IDR.

Source code in sleap_nn/tracking/mot_metrics.py

def id_metrics(seq: MOTSequence, threshold: float = 0.5) -> Dict[str, Any]:
    """Compute the identity metrics (IDF1, Ristani et al.) of one sequence.

    Named to avoid clashing with :func:`sleap_nn.evaluation.identity_metrics`,
    the older sleap-nn definitions (which differ from TrackEval, see
    :class:`sleap_nn.evaluation.IdentityMetrics`).

    Args:
        seq: The sequence to score. Ids are relabeled contiguously first.
        threshold: Minimum similarity for a pair to count as a potential match.

    Returns:
        Dict with the integer counts ``IDTP``, ``IDFN``, ``IDFP`` and the floats
        ``IDF1``, ``IDP``, ``IDR``.
    """
    return _identity(relabel_contiguous(seq), float(threshold))

mot_metrics(seq, threshold=0.5)

Compute CLEAR-MOT, Identity and HOTA for one sequence.

Parameters:

Name	Type	Description	Default
seq	MOTSequence	The sequence to score.	required
threshold	float	Match threshold for CLEAR-MOT and Identity. HOTA has its own thresholds (:data:HOTA_ALPHAS).	0.5

Returns:

Type	Description
Dict[str, Any]	The union of :func:clear_metrics, :func:id_metrics and :func:hota_metrics.

Source code in sleap_nn/tracking/mot_metrics.py

def mot_metrics(seq: MOTSequence, threshold: float = 0.5) -> Dict[str, Any]:
    """Compute CLEAR-MOT, Identity and HOTA for one sequence.

    Args:
        seq: The sequence to score.
        threshold: Match threshold for CLEAR-MOT and Identity. HOTA has its
            own thresholds (:data:`HOTA_ALPHAS`).

    Returns:
        The union of :func:`clear_metrics`, :func:`id_metrics` and
        :func:`hota_metrics`.
    """
    seq = relabel_contiguous(seq)
    threshold = float(threshold)
    res = _clear(seq, threshold)
    res.update(_identity(seq, threshold))
    res.update(_hota(seq, HOTA_ALPHAS))
    return res

relabel_contiguous(seq)

Relabel ground-truth and predicted ids to contiguous 0..n-1.

Each side is relabeled independently, in sorted order of its original ids. Every metric needs this (TrackEval's datasets do it while loading): the per-id tables are indexed by id, so a gap in the ids would create a phantom trajectory -- counted as mostly lost, and a 0/0 in HOTA's alignment score.

Parameters:

Name	Type	Description	Default
seq	MOTSequence	The sequence to relabel.	required

Returns:

Type	Description
MOTSequence	A new :class:MOTSequence with the same similarities and resets (the similarity arrays are shared, not copied).

Source code in sleap_nn/tracking/mot_metrics.py

def relabel_contiguous(seq: MOTSequence) -> MOTSequence:
    """Relabel ground-truth and predicted ids to contiguous ``0..n-1``.

    Each side is relabeled independently, in sorted order of its original ids.
    Every metric needs this (TrackEval's datasets do it while loading): the
    per-id tables are indexed by id, so a gap in the ids would create a phantom
    trajectory -- counted as mostly lost, and a 0/0 in HOTA's alignment score.

    Args:
        seq: The sequence to relabel.

    Returns:
        A new :class:`MOTSequence` with the same similarities and resets (the
        similarity arrays are shared, not copied).
    """
    empty = np.zeros(0, dtype=np.int64)
    gt_unique = np.unique(np.concatenate(list(seq.gt_ids) + [empty]))
    pred_unique = np.unique(np.concatenate(list(seq.pred_ids) + [empty]))
    return MOTSequence(
        gt_ids=[np.searchsorted(gt_unique, g) for g in seq.gt_ids],
        pred_ids=[np.searchsorted(pred_unique, p) for p in seq.pred_ids],
        similarity=list(seq.similarity),
        resets=seq.resets,
    )