Flaky tests

What's a flaky test?

It's a test that sometimes fails, but if you retry it enough times, it passes, eventually.

Quarantined tests

When a test frequently fails in master, a ~"master:broken" issue should be created. If the test cannot be fixed in a timely fashion, there is an impact on the productivity of all the developers, so it should be placed in quarantine by assigning the :quarantine metadata.

This means it will be skipped unless run with --tag quarantine:

bin/rspec --tag quarantine

Before putting a test in quarantine, you should make sure that a ~"master:broken" issue exists for it so it won't stay in quarantine forever.

Once a test is in quarantine, there are 3 choices:

Quarantine tests on the CI

Quarantined tests are run on the CI in dedicated jobs that are allowed to fail:

Automatic retries and flaky tests detection

On our CI, we use rspec-retry to automatically retry a failing example a few times (see [spec/spec_helper.rb] for the precise retries count).

We also use a home-made RspecFlaky::Listener listener which records flaky examples in a JSON report file on master (retrieve-tests-metadata and update-tests-metadata jobs), and warns when a new flaky example is detected in any other branch (flaky-examples-check job). In the future, the flaky-examples-check job will not be allowed to fail.

This was originally implemented in: https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/13021.

Problems we had in the past at GitLab

Time-sensitive flaky tests

Array order expectation

Feature tests

Capybara expectation times out

Resources

Return to Testing documentation