Before you skip or mark a flaky test, know what the marker hides

Skip, xfail and quarantine are different promises

A team that is tired of a red build often marks the flaky test and moves on. In pytest the markers mean different things. A skipped test is not run at all. An xfail test is run and expected to fail; a failure is reported as XFAIL without a traceback, and a pass is reported as XPASS. By default neither outcome fails the suite. Marking a test strict makes an XPASS fail the suite. A separate quarantine suite, as Martin Fowler describes, moves non-deterministic tests out of the main run so the healthy tests keep giving trustworthy feedback. Each choice leaves a different record and takes a different amount of protection away.

• Skip: the test gives no signal in either direction while the marker stays.
• xfail without strict: a flaky test will sometimes XPASS and sometimes XFAIL, and neither changes the exit status.
• xfail with strict: a flaky test that happens to pass will fail the suite, which makes the marker noisy.
• Quarantine suite: the test still runs somewhere, but not in the run that gates a merge.

What each marker hides

The pytest documentation provides the controls that make a marker less blind. The option raises= restricts an xfail to one exception type, so a different failure is reported as an ordinary failure instead of being absorbed. The option run=False reports the test without executing it, which suits a test that crashes the interpreter. The report flag -rxXs lists xfailed, xpassed and skipped tests, and --runxfail runs the marked tests as though they had no marker. Used together, these let a team see what it has set aside. Unused, a marker is a permanent hole that looks like a passing suite.

• Put a reason in every marker, including who owns it and the date to revisit.
• Add the xfail and skip report to the CI summary so the count is visible each run.
• Run the marked tests unmarked now and then, to see whether they still fail.

A quarantine needs an owner and an end date

Fowler points out that quarantined tests give no regression protection, so they should be fixed quickly rather than forgotten. He suggests limiting how many tests may sit in quarantine, or for how long, giving eight tests and a week as examples. Treat the quarantine as a list with limits. For each test record the reason, the person or team who owns it, the date it will be reviewed and what protection is lost meanwhile. When the date arrives, fix the cause, extend the quarantine with a stated reason or delete the test. A quarantine with no dates is a graveyard.

• Set a maximum number of quarantined tests and a maximum age, and fail a build or raise a task when either is exceeded.
• Do not quarantine a test that fails every time. That test is reporting something repeatable.

Retries are not a quarantine, and how a paid job is accepted

A retry setting reruns a failing test instead of setting it aside. In Playwright's runner, retries are off unless you enable them, and a test that fails and then passes is reported as flaky, in a category of its own. Someone has to read that category, or the problem is merely renamed. For a named set of flaky tests, the one-off job here reproduces each failure by repeated runs and classifies the cause, then fixes the cause or quarantines the test with a visible marker, a reason, an owner and a review date. Acceptance requires each quarantined test to be marked and reported as excluded in the suite summary, and each fixed test to pass the agreed run count under the baseline conditions. We do not add retry-until-green settings as a repair.