kolena.workflow.TestSuite

Legacy Warning

Content in this section reflects outdated practices or deprecated features. It's recommended to avoid using these in new developments.

While existing implementations using these features will continue to receive support, we strongly advise adopting the latest standards and tools for new projects to ensure optimal performance and compatibility. For more information and up-to-date practices, please refer to our newest documentation at docs.kolena.io.

TestSuite(name, version=None, description=None, test_cases=None, reset=False, tags=None)

Bases: Frozen, WithTelemetry

A test suite groups together one or more test cases. Typically a test suite represents a benchmark test dataset, with test cases representing different meaningful subsets, or slices, or this benchmark.

Rather than importing this class directly, use the TestSuite type definition returned from define_workflow.

workflow: Workflow instance-attribute

The workflow of this test suite. Automatically populated when constructing via test suite type returned from define_workflow.

name: str instance-attribute

The unique name of this test suite.

version: int instance-attribute

The version of this test suite. A test suite's version is automatically incremented whenever it is edited via TestSuite.edit.

description: str instance-attribute

Free-form, human-readable description of this test suite. Can be edited at any time via TestSuite.edit.

test_cases: List[TestCase] instance-attribute

The TestCase objects belonging to this test suite.

tags: Set[str] instance-attribute

The tags associated with this test suite.

Editor(test_cases, description, tags, reset)

description(description)

Update the description of the test suite.

add(test_case)

Add a test case to this test suite. If a different version of the test case already exists in this test suite, it is replaced.

Parameters:

Name	Type	Description	Default
test_case	TestCase	The test case to add to the test suite.	required

remove(test_case)

Remove a test case from this test suite. Does nothing if the test case is not in the test suite.

Parameters:

Name	Type	Description	Default
test_case	TestCase	The test case to remove.	required

create(name, description=None, test_cases=None, tags=None) classmethod

Create a new test suite with the provided name.

Parameters:

Name	Type	Description	Default
name	str	The name of the new test suite to create.	required
description	Optional[str]	Optional free-form description of the test suite to create.	None
test_cases	Optional[List[TestCase]]	Optionally specify a list of test cases to populate the test suite.	None
tags	Optional[Set[str]]	Optionally specify a set of tags to attach to the test suite.	None

Returns:

Type	Description
TestSuite	The newly created test suite.

load(name, version=None) classmethod

Load an existing test suite with the provided name.

Parameters:

Name	Type	Description	Default
name	str	The name of the test suite to load.	required
version	Optional[int]	Optionally specify a particular version of the test suite to load. Defaults to the latest version when unset.	None

Returns:

Type	Description
TestSuite	The loaded test suite.

load_all(*, tags=None) classmethod

Load the latest version of all test suites with this workflow.

Parameters:

Name	Type	Description	Default
tags	Optional[Set[str]]	Optionally specify a set of tags to apply as a filter. The loaded test suites will include only test suites with tags matching each of these specified tags, i.e. test_suite.tags.intersection(tags) == tags.	None

Returns:

Type	Description
List[TestSuite]	The latest version of all test suites, with matching tags when specified.

edit(reset=False)

Edit this test suite in a context:

with test_suite.edit() as editor:
    # perform as many editing actions as desired
    editor.add(...)
    editor.remove(...)

Changes are committed to the Kolena platform when the context is exited.

Parameters:

Name	Type	Description	Default
reset	bool	Clear all existing test cases in the test suite.	False

load_test_samples()

Load test samples for all test cases within this test suite.

Returns:

Type	Description
List[Tuple[TestCase, List[TestSample]]]	A list of TestCases, each paired with the list of TestSamples it contains.