Collect all table-driven failures with unittest subTest

By Batu · English technical notes

Use unittest.TestCase.subTest() around each table row to collect all table-driven failures in one test method. A plain loop stops at its first failed assertion, concealing later rows. A subtest reports the failed row and continues to the next one.

The table uses three boundary-oriented integers and intentionally claims each is positive. The predicate is false for -1 and 0, then true for 1. PlainTable reaches -1, records one failure, and never evaluates 0 or 1. SubtestTable creates a subtest for every row, so it records two failures while still representing one test method.

RecordingResult.addSubTest() saves the parameter for every unsuccessful subtest. This produces deterministic labels without using a text runner or timings. The assertions distinguish the two important counts: both suites run one test method, while their failure records differ. The output therefore shows diagnostics gathered from the same data, not a claim that subtests turn rows into independently discovered test methods.

subTest() was added in Python 3.4. The official documentation shows that subtests preserve parameter information and explains that execution without them stops after the first failure (unittest.TestCase.subTest). Use separate test methods when fixtures or reporting policy need each row to be an independent test.

AI assistance disclosure: this article and its synthetic example were prepared with AI assistance.

Example

import unittest


CASES = [(-1, True), (0, True), (1, True)]


def is_positive(number):
    return number > 0


class PlainTable(unittest.TestCase):
    def test_positive_cases(self):
        for number, expected in CASES:
            self.assertEqual(is_positive(number), expected)


class SubtestTable(unittest.TestCase):
    def test_positive_cases(self):
        for number, expected in CASES:
            with self.subTest(number=number):
                self.assertEqual(is_positive(number), expected)


class RecordingResult(unittest.TestResult):
    def __init__(self):
        super().__init__()
        self.failed_numbers = []

    def addSubTest(self, test, subtest, outcome):
        if outcome is not None:
            self.failed_numbers.append(subtest.params["number"])
        super().addSubTest(test, subtest, outcome)


def run(case, result):
    unittest.defaultTestLoader.loadTestsFromTestCase(case).run(result)


plain = unittest.TestResult()
subtests = RecordingResult()
run(PlainTable, plain)
run(SubtestTable, subtests)
assert plain.testsRun == subtests.testsRun == 1
assert len(plain.failures) == 1
assert len(subtests.failures) == 2
assert subtests.failed_numbers == [-1, 0]
print(f"plain failures: {len(plain.failures)}")
print(f"subtest failures: {len(subtests.failures)}")
print(f"subtest inputs: {subtests.failed_numbers}")

Expected output:

plain failures: 1
subtest failures: 2
subtest inputs: [-1, 0]