kolena.workflow.TestCase

Legacy Warning

Content in this section reflects outdated practices or deprecated features. It's recommended to avoid using these in new developments.

While existing implementations using these features will continue to receive support, we strongly advise adopting the latest standards and tools for new projects to ensure optimal performance and compatibility. For more information and up-to-date practices, please refer to our newest documentation at docs.kolena.io.

TestCase(name, version=None, description=None, test_samples=None, reset=False)

Bases: Frozen, WithTelemetry

A test case holds a list of test samples paired with ground truths representing a testing dataset or a slice of a testing dataset.

Rather than importing this class directly, use the TestCase type definition returned from define_workflow.

workflow: Workflow instance-attribute

The workflow of this test case. Automatically populated when constructing via test case type returned from define_workflow.

name: str instance-attribute

The unique name of this test case. Cannot be changed after creation.

version: int instance-attribute

The version of this test case. A test case's version is automatically incremented whenever it is edited via TestCase.edit.

description: str instance-attribute

Free-form, human-readable description of this test case. Can be edited at any time via TestCase.edit.

Editor(description, reset)

description(description)

Update the description of the test case.

add(test_sample, ground_truth)

Add a test sample to the test case. When the test sample already exists in the test case, its ground truth is overwritten with the ground truth provided here.

Parameters:

Name	Type	Description	Default
test_sample	TestSample	The test sample to add.	required
ground_truth	GroundTruth	The ground truth for the test sample.	required

remove(test_sample)

Remove a test sample from the test case. Does nothing if the test sample is not in the test case.

Parameters:

Name	Type	Description	Default
test_sample	TestSample	The test sample to remove.	required

create(name, description=None, test_samples=None) classmethod

Create a new test case with the provided name.

Parameters:

Name	Type	Description	Default
name	str	The name of the new test case to create.	required
description	Optional[str]	Optional free-form description of the test case to create.	None
test_samples	Optional[List[Tuple[TestSample, GroundTruth]]]	Optionally specify a set of test samples and ground truths to populate the test case.	None

Returns:

Type	Description
TestCase	The newly created test case.

load(name, version=None) classmethod

Load an existing test case with the provided name.

Parameters:

Name	Type	Description	Default
name	str	The name of the test case to load.	required
version	Optional[int]	Optionally specify a particular version of the test case to load. Defaults to the latest version when unset.	None

Returns:

Type	Description
TestCase	The loaded test case.

load_test_samples()

Load all TestSamples and GroundTruths contained in this test case.

Returns:

Type	Description
List[Tuple[TestSample, GroundTruth]]	A list of each test sample, paired with its ground truth, in this test case.

iter_test_samples()

Iterate through all TestSamples and GroundTruths contained in this test case.

Returns:

Type	Description
Iterator[Tuple[TestSample, GroundTruth]]	An iterator yielding each test sample, paired with its ground truth, in this test case.

edit(reset=False)

Edit this test case in a context:

with test_case.edit() as editor:
    # perform as many editing actions as desired
    editor.add(...)
    editor.remove(...)

Changes are committed to the Kolena platform when the context is exited.

Parameters:

Name	Type	Description	Default
reset	bool	Clear all existing test samples in the test case.	False

init_many(data, reset=False) classmethod

Experimental

This function is considered experimental, so beware that it is subject to changes even in patch releases.

Create, load or edit multiple test cases.

test_cases = TestCase.init_many([
    ("test-case 1", [(test_sample_1, ground_truth_1), ...]),
    ("test-case 2", [(test_sample_2, ground_truth_2), ...])
])

test_suite = TestSuite("my test suite", test_cases=test_cases)

Changes are committed to the Kolena platform together. If there is an error, none of the edits would take effect.

Parameters:

Name	Type	Description	Default
data	List[Tuple[str, List[Tuple[TestSample, GroundTruth]]]]	A list of tuples where each tuple is a test case name and a set of test samples and ground truths tuples for the test case.	required
reset	bool	If a test case of the same name already exists, overwrite with the provided test_samples.	False

Returns:

Type	Description
List[TestCase]	The test cases.