Report DUT-related errors in fixtures as fail instead of error?

[Slarag]

Pytest has the really great approach that unhandeled exceptions or a call to pytest.fail in a fixture reports the test as error instead of failed.

We're testing devices (DUT, device under test) with a pytest-based framework. And before each test, we use fixtures to ensure that the device is in a certain state. If somehow the DUT fails to enter the correct state due to a DUT-related problem (which will raise a specific exception in the fixture) the test analysts want to see the according test marked as failed and not as error as the root-cause is DUT-related and not environment-related.

So far, I could not find an easy solution for this. Can I do this via a hook function? If yes, which hook function would be the correct one?

I've tried pytest_runtest_makereport as there I have acces to the call object after the test has run (or failed), but there I cannot alter call.result and calling pytest.fail inside this hook will raise an internal error and caues pytest to exit.

Another approach was to just catch the exception in a try/except block, append some error infomation to the request.node object and continue fixture setup. Then we would evaluate that information in the pytest_runtest_call hook and call pytest.fail from there. This works, but the big downside here is that other fixtures that might depend on the failed fixture are still being setup. And as the device is already in a bad state, those fixture might have undefined behavior. In this case I would need an option to avoid setting up other fixtures (without needing to add checks in each and every other fixture).

[Labeeb2339]

The clean way is to change the reporting category for only the known DUT failure; do not call pytest.fail() from pytest_runtest_makereport() or try to mutate call.result. Leave the exception in setup: pytest will then skip setup of fixtures that depend on the failed fixture.

import pytest

dut_failure = pytest.StashKey[bool]()

@pytest.fixture
def dut(request):
    try:
        ensure_dut_state()
    except DutError as exc:
        request.node.stash[dut_failure] = True
        pytest.fail(str(exc), pytrace=False)
    return device

@pytest.hookimpl(wrapper=True, tryfirst=True)
def pytest_runtest_makereport(item, call):
    report = yield
    if report.when == "setup" and item.stash.get(dut_failure, False):
        report.dut_failure = True
    return report

def pytest_report_teststatus(report, config):
    if getattr(report, "dut_failure", False):
        return "failed", "F", "FAILED"

pytest_report_teststatus is the public hook that maps a TestReport to the terminal category/letter, so this displays the selected setup failure as FAILED while preserving pytest's fixture dependency short-circuit. Other reporters may still use the original when="setup" phase; if they must also see a call-phase failure, that needs a more invasive report transformation.