Test-only changes

The test-only changes job does not run the full set of PHPUnit tests, but instead it just runs the tests that have been modified in a Merge Request. Additionally all of the non-test files are reverted back to their unchanged state.

The purpose of this job is to demonstrate that the changes in the tests provide test coverage for the feature/bug/problem which is being worked on. It is equivalent to the "test-only will fail" patches that have been used on DrupalCI for contrib projects and for core over the past years. Drupal Core already has a 'test-only changes' GitLab pipeline job, but this feature is new for contrib projects.

This job is only created in Merge Request pipelines, it will not appear in branch or scheduled pipelines. It will only be created if the merge request actually contains changes to PHPUnit test files.

It is manual, which means it is optional and will not start automatically, but requires you to click the [>] 'play' button.

The log has extra information showing which files have been changed in the merge request and which files have been reverted before running the tests. The job is intended to fail with an amber warning, as this demonstrates that the modified tests do provide test coverage for the problem being fixed.

If this job ends with a green pass it means that the bug or feature does not have test coverage, because the tests pass even with the unmodified source files.

It can also be used to demonstrate new test coverage for existing source code which is not being changed. To do this, make a temporary change to the source (to simulate a bug) which will cause the new test to fail in the regular phpunit job. However the test-only changes job should pass green. This is the reverse outcome compared the usual scenario described above, but it does indicate that the new test provides coverage as intended.

Parallel option

If the phpunit job uses parallel, those options will be inherited by the test-only changes job. The parallel options could be a number, to split a job into several ones, or could be a matrix of options, each one passed to a different parallel job. This means that in some cases, keeping this value makes sense but in others it does not.

If you want the test-only changes job to not have parallel options, you can do this in your .gitlab-ci.yml:

test-only changes:
  # Remove the parallel definition that is defined in the customized phpunit job.
  parallel: []