Update the test removal procedure
Tests that target and API which does not provide a version discovery
mechanism, and which introduces backward incompatibilites cannot
live in Tempest due to its branchless nature.
With this commit I propose to extend the list of exceptions from
the standard removal procedure to include this group of tests.
Example: I84626976ca729f65b0ee0ea6afe5c9a6a408eecc
Change-Id: I209b0b3d470bd284949a77d9c266d78e7c5a5c13
diff --git a/doc/source/test-removal.rst b/doc/source/test-removal.rst
index 79a5846..4757dc4 100644
--- a/doc/source/test-removal.rst
+++ b/doc/source/test-removal.rst
@@ -38,8 +38,10 @@
#. The test proposed for removal has a failure rate < 0.50% in the gate over
the past release (the value and interval will likely be adjusted in the
future)
- #. There must not be an external user/consumer of tempest that depends on the
- test proposed for removal
+
+ .. _`prong #3`:
+ #. There must not be an external user/consumer of tempest
+ that depends on the test proposed for removal
The answers to 1 and 2 are easy to verify. For 1 just provide a link to the new
test location. If you are linking to the tempest removal patch please also put
@@ -133,6 +135,10 @@
#. A revert for a patch which added a broken test, or testing which didn't
actually run in the gate (basically any revert for something which
shouldn't have been added)
+ #. Tests that would become out of scope as a consequence of an API change,
+ as described in `API Compatibility`_.
+ Such tests cannot live in Tempest because of the branchless nature of
+ Tempest. Such test must still honor `prong #3`_.
For the first exception type the only types of testing in tree which have been
declared out of scope at this point are:
@@ -149,7 +155,7 @@
Tempest Scope
^^^^^^^^^^^^^
-Also starting in the liberty cycle tempest has defined a set of projects which
+Starting in the liberty cycle tempest has defined a set of projects which
are defined as in scope for direct testing in tempest. As of today that list
is:
@@ -166,3 +172,17 @@
to maintain continuity after migrating the tests out of tempest.
.. _tempest plugin mechanism: http://docs.openstack.org/developer/tempest/plugin.html
+
+API Compatibility
+"""""""""""""""""
+
+If an API introduces a non-discoverable, backward incompatible change, and
+such change is not backported to all versions supported by Tempest, tests for
+that API cannot live in Tempest anymore.
+This is because tests would not be able to know or control which API response
+to expect, and thus would not be able to enforce a specific behavior.
+
+If a test exists in Tempest that would meet this criteria as consequence of a
+change, the test must be removed according to the procedure discussed into
+this document. The API change should not be merged until all conditions
+required for test removal can be met.
\ No newline at end of file