Skip to content

Legacy Warning

Content in this section reflects outdated practices or deprecated features. It's recommended to avoid using these in new developments.

While existing implementations using these features will continue to receive support, we strongly advise adopting the latest standards and tools for new projects to ensure optimal performance and compatibility. For more information and up-to-date practices, please refer to our newest documentation at docs.kolena.io.

kolena._experimental.workflow.ThresholdedMetrics#

Experimental Feature

This pre-built workflow is an experimental feature. Experimental features are under active development and may occasionally undergo API-breaking changes.

Thresholded Metrics play a crucial role in specific contexts of model evaluation, particularly when the performance of models is assessed based on varying threshold values. In such scenarios, these metrics are vital for accurately interpreting the effectiveness of the models, as different thresholds can lead to markedly different performance outcomes.

ThresholdedMetrics(threshold) dataclass #

Bases: TypedDataObject[_MetricsType]

Represents metrics tied to a specific threshold.

List[ThresholdedMetrics] should be used as a field type within MetricsTestSample from the kolena.workflow module. This list is meant to hold metric values associated with distinct thresholds. These metrics are expected to be uniform across TestSample instances within a single test execution.

ThresholdedMetrics prohibits the use of dictionary objects as field values and guarantees that the threshold values remain immutable once set. For application within a particular workflow, subclassing is required to define relevant metrics fields.

Usage example:

from kolena.workflow import MetricsTestSample
from kolena._experimental.workflow import ThresholdedMetrics

@dataclass(frozen=True)
class ClassThresholdedMetrics(ThresholdedMetrics):
    precision: float
    recall: float
    f1: float

@dataclass(frozen=True)
class TestSampleMetrics(MetricsTestSample):
    car: List[ClassThresholdedMetrics]
    pedestrian: List[ClassThresholdedMetrics]

# Creating an instance of metrics
metric = TestSampleMetrics(
    car=[
        ClassThresholdedMetrics(threshold=0.3, precision=0.5, recall=0.8, f1=0.615),
        ClassThresholdedMetrics(threshold=0.4, precision=0.6, recall=0.6, f1=0.6),
        ClassThresholdedMetrics(threshold=0.5, precision=0.8, recall=0.4, f1=0.533),
        # ...
    ],
    pedestrian=[
        ClassThresholdedMetrics(threshold=0.3, precision=0.6, recall=0.9, f1=0.72),
        ClassThresholdedMetrics(threshold=0.4, precision=0.7, recall=0.7, f1=0.7),
        ClassThresholdedMetrics(threshold=0.5, precision=0.8, recall=0.6, f1=0.686),
        # ...
    ],
)

Raises: TypeError: If any of the field values is a dictionary.