| ================================= |
| Microversion Testing With Tempest |
| ================================= |
| |
| Many OpenStack Services provide their APIs with `microversion`_ |
| support and want to test them in Tempest. |
| |
| .. _microversion: http://specs.openstack.org/openstack/api-wg/guidelines/microversion_specification.html |
| |
| This document covers how to test microversions for each project and |
| whether tests should live in Tempest or on project side. |
| |
| Tempest Scope For Microversion Testing |
| """""""""""""""""""""""""""""""""""""" |
| APIs microversions for any OpenStack service grow rapidly and |
| testing each and every microversion in Tempest is not feasible and |
| efficient way. |
| Also not every API microversion changes the complete system behavior |
| and many of them only change the API or DB layer to accept and return more |
| data on API. |
| |
| Tempest is an integration test suite, but not all API microversion testing fall under this category. |
| As a result, Tempest mainly covers integration test cases for microversions, Other testing coverage |
| for microversion should be hosted on project side as functional tests or via Tempest plugin as per |
| project guidelines. |
| |
| .. note:: Integration tests are those tests which involve more than one service to |
| verify the expected behavior by single or combination of API requests. |
| If a test is just to verify the API behavior as success and failure cases |
| or verify its expected response object, then it does not fall under integration |
| tests. |
| |
| Tempest will cover only integration testing of applicable microversions with |
| below exceptions: |
| |
| #. Test covers a feature which is important for interoperability. This covers tests requirement |
| from Defcore. |
| #. Test needed to fill Schema gaps. |
| Tempest validates API responses with defined JSON schema. API responses can be different on |
| each microversion and the JSON schemas need to be defined separately for the microversion. |
| While implementing new integration tests for a specific microversion, there |
| may be a gap in the JSON schemas (caused by previous microversions) implemented |
| in Tempest. |
| Filling that gap while implementing the new integration test cases is not efficient due to |
| many reasons: |
| |
| * Hard to review |
| * Sync between multiple integration tests patches which try to fill the same schema gap at same |
| time |
| * Might delay the microversion change on project side where project team wants Tempest |
| tests to verify the results. |
| |
| Tempest will allow to fill the schema gaps at the end of each cycle, or more |
| often if required. |
| Schema gap can be filled with testing those with a minimal set of tests. Those |
| tests might not be integration tests and might be already covered on project |
| side also. |
| This exception is needed because: |
| |
| * Allow to create microversion response schema in Tempest at the same time that projects are |
| implementing their API microversions. This will make implementation easier for adding |
| required tests before a new microversion change can be merged in the corresponding project |
| and hence accelerate the development of microversions. |
| * New schema must be verified by at least one test case which exercises such schema. |
| |
| For example: |
| If any projects implemented 4 API microversion say- v2.3, v2.4, v2.5, v2.6 |
| Assume microversion v2.3, v2.4, v2.6 change the API Response which means Tempest |
| needs to add JSON schema for v2.3, v2.4, v2.6. |
| In that case if only 1 or 2 tests can verify all new schemas then we do not need |
| separate tests for each new schemas. In worst case, we have to add 3 separate tests. |
| #. Test covers service behavior at large scale with involvement of more deep layer like hypervisor |
| etc not just API/DB layer. This type of tests will be added case by case basis and |
| with project team consultation about why it cannot be covered on project side and worth to test |
| in Tempest. |
| |
| Project Scope For Microversion Testing |
| """""""""""""""""""""""""""""""""""""" |
| All microversions testing which are not covered under Tempest as per above section, should be |
| tested on project side as functional tests or as Tempest plugin as per project decision. |
| |
| |
| Configuration options for Microversion |
| """""""""""""""""""""""""""""""""""""" |
| |
| * Add configuration options for specifying test target Microversions. |
| We need to specify test target Microversions because the supported |
| Microversions may be different between OpenStack clouds. For operating |
| multiple Microversion tests in a single Tempest operation, configuration |
| options should represent the range of test target Microversions. |
| New configuration options are: |
| |
| * min_microversion |
| * max_microversion |
| |
| Those should be defined under respective section of each service. |
| For example: |
| |
| .. code-block:: ini |
| |
| [compute] |
| min_microversion = None |
| max_microversion = latest |
| |
| |
| How To Implement Microversion Tests |
| """"""""""""""""""""""""""""""""""" |
| |
| Tempest provides stable interfaces to test API Microversion. |
| For Details, see: `API Microversion testing Framework`_ |
| This document explains how to implement Microversion tests using those |
| interfaces. |
| |
| .. _API Microversion testing Framework: https://docs.openstack.org/tempest/latest/library/api_microversion_testing.html |
| |
| |
| Step1: Add skip logic based on configured Microversion range |
| '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' |
| |
| Add logic to skip the tests based on Tests class and configured Microversion |
| range. |
| api_version_utils.check_skip_with_microversion function can be used |
| to automatically skip the tests which do not fall under configured |
| Microversion range. |
| For example: |
| |
| .. code-block:: python |
| |
| class BaseTestCase1(api_version_utils.BaseMicroversionTest): |
| |
| [..] |
| @classmethod |
| def skip_checks(cls): |
| super(BaseTestCase1, cls).skip_checks() |
| api_version_utils.check_skip_with_microversion(cls.min_microversion, |
| cls.max_microversion, |
| CONF.compute.min_microversion, |
| CONF.compute.max_microversion) |
| |
| Skip logic can be added in tests base class or any specific test class depends on |
| tests class structure. |
| |
| Step2: Selected API request microversion |
| '''''''''''''''''''''''''''''''''''''''' |
| |
| Select appropriate Microversion which needs to be used |
| to send with API request. |
| api_version_utils.select_request_microversion function can be used |
| to select the appropriate Microversion which will be used for API request. |
| For example: |
| |
| .. code-block:: python |
| |
| @classmethod |
| def resource_setup(cls): |
| super(BaseTestCase1, cls).resource_setup() |
| cls.request_microversion = ( |
| api_version_utils.select_request_microversion( |
| cls.min_microversion, |
| CONF.compute.min_microversion)) |
| |
| |
| Step3: Set Microversion on Service Clients |
| '''''''''''''''''''''''''''''''''''''''''' |
| |
| Microversion selected by Test Class in previous step needs to be set on |
| service clients so that APIs can be requested with selected Microversion. |
| |
| Microversion can be defined as global variable on service clients which |
| can be set using fixture. |
| Also Microversion header name needs to be defined on service clients which |
| should be constant because it is not supposed to be changed by project |
| as per API contract. |
| For example: |
| |
| .. code-block:: python |
| |
| COMPUTE_MICROVERSION = None |
| |
| class BaseClient1(rest_client.RestClient): |
| api_microversion_header_name = 'X-OpenStack-Nova-API-Version' |
| |
| Now test class can set the selected Microversion on required service clients |
| using fixture which can take care of resetting the same once tests is completed. |
| For example: |
| |
| .. code-block:: python |
| |
| def setUp(self): |
| super(BaseTestCase1, self).setUp() |
| self.useFixture(api_microversion_fixture.APIMicroversionFixture( |
| self.request_microversion)) |
| |
| Service clients needs to add set Microversion in API request header which |
| can be done by overriding the get_headers() method of rest_client. |
| For example: |
| |
| .. code-block:: python |
| |
| COMPUTE_MICROVERSION = None |
| |
| class BaseClient1(rest_client.RestClient): |
| api_microversion_header_name = 'X-OpenStack-Nova-API-Version' |
| |
| def get_headers(self): |
| headers = super(BaseClient1, self).get_headers() |
| if COMPUTE_MICROVERSION: |
| headers[self.api_microversion_header_name] = COMPUTE_MICROVERSION |
| return headers |
| |
| |
| Step4: Separate Test classes for each Microversion |
| '''''''''''''''''''''''''''''''''''''''''''''''''' |
| |
| This is last step to implement Microversion test class. |
| |
| For any Microversion tests, basically we need to implement a |
| separate test class. In addition, each test class defines its |
| Microversion range with class variable like min_microversion |
| and max_microversion. Tests will be valid for that defined range. |
| If that range is out of configured Microversion range then, test |
| will be skipped. |
| |
| .. note:: Microversion testing is supported at test class level not at |
| individual test case level. |
| |
| For example: |
| |
| Below test is applicable for Microversion from 2.2 till 2.9: |
| |
| .. code-block:: python |
| |
| class BaseTestCase1(api_version_utils.BaseMicroversionTest, |
| tempest.test.BaseTestCase): |
| |
| [..] |
| |
| |
| class Test1(BaseTestCase1): |
| min_microversion = '2.2' |
| max_microversion = '2.9' |
| |
| [..] |
| |
| Below test is applicable for Microversion from 2.10 till latest: |
| |
| .. code-block:: python |
| |
| class Test2(BaseTestCase1): |
| min_microversion = '2.10' |
| max_microversion = 'latest' |
| |
| [..] |
| |
| |
| Notes about Compute Microversion Tests |
| """""""""""""""""""""""""""""""""""""" |
| |
| Some of the compute Microversion tests have been already implemented |
| with the Microversion testing framework. So for further tests only |
| step 4 is needed. |
| |
| Along with that JSON response schema might need versioning if needed. |
| |
| Compute service clients strictly validate the response against defined JSON |
| schema and does not allow additional elements in response. |
| So if that Microversion changed the API response then schema needs to be versioned. |
| New JSON schema file needs to be defined with new response attributes and service |
| client methods will select the schema based on requested microversion. |
| |
| If Microversion tests are implemented randomly meaning not |
| in sequence order(v2.20 tests added and previous Microversion tests are not yet added) |
| then, still schema might need to be version for older Microversion if they changed |
| the response. |
| This is because Nova Microversion includes all the previous Microversions behavior. |
| |
| For Example: |
| Implementing the v2.20 Microversion tests before v2.9 and 2.19- |
| v2.20 API request will respond as latest behavior of Nova till v2.20, |
| and in v2.9 and 2.19, server response has been changed so response schema needs |
| to be versioned accordingly. |
| |
| That can be done by using the get_schema method in below module: |
| |
| The base_compute_client module |
| '''''''''''''''''''''''''''''' |
| |
| .. automodule:: tempest.lib.services.compute.base_compute_client |
| :members: |
| |
| |
| Microversion tests implemented in Tempest |
| """"""""""""""""""""""""""""""""""""""""" |
| |
| * Compute |
| |
| * `2.1`_ |
| |
| .. _2.1: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id1 |
| |
| * `2.2`_ |
| |
| .. _2.2: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id2 |
| |
| * `2.3`_ |
| |
| .. _2.3: http://docs.openstack.org/developer/nova/api_microversion_history.html#maximum-in-kilo |
| |
| * `2.6`_ |
| |
| .. _2.6: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id5 |
| |
| * `2.8`_ |
| |
| .. _2.8: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id7 |
| |
| * `2.9`_ |
| |
| .. _2.9: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id8 |
| |
| * `2.10`_ |
| |
| .. _2.10: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id9 |
| |
| * `2.19`_ |
| |
| .. _2.19: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id17 |
| |
| * `2.20`_ |
| |
| .. _2.20: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id18 |
| |
| * `2.21`_ |
| |
| .. _2.21: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id19 |
| |
| * `2.25`_ |
| |
| .. _2.25: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#maximum-in-mitaka |
| |
| * `2.26`_ |
| |
| .. _2.26: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id23 |
| |
| * `2.28`_ |
| |
| .. _2.28: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id25 |
| |
| * `2.32`_ |
| |
| .. _2.32: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id29 |
| |
| * `2.33`_ |
| |
| .. _2.33: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id30 |
| |
| * `2.36`_ |
| |
| .. _2.36: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#microversion |
| |
| * `2.37`_ |
| |
| .. _2.37: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id35 |
| |
| * `2.39`_ |
| |
| .. _2.39: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id36 |
| |
| * `2.41`_ |
| |
| .. _2.41: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id38 |
| |
| * `2.42`_ |
| |
| .. _2.42: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#maximum-in-ocata |
| |
| * `2.45`_ |
| |
| .. _2.45: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id41 |
| |
| * `2.47`_ |
| |
| .. _2.47: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id43 |
| |
| * `2.48`_ |
| |
| .. _2.48: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id44 |
| |
| * `2.49`_ |
| |
| .. _2.49: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id45 |
| |
| * `2.50`_ |
| |
| .. _2.50: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id46 |
| |
| * `2.53`_ |
| |
| .. _2.53: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#maximum-in-pike |
| |
| * `2.54`_ |
| |
| .. _2.54: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id50 |
| |
| * `2.55`_ |
| |
| .. _2.55: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id51 |
| |
| * `2.57`_ |
| |
| .. _2.57: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id53 |
| |
| * `2.59`_ |
| |
| .. _2.59: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id55 |
| |
| * `2.60`_ |
| |
| .. _2.60: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#maximum-in-queens |
| |
| * `2.61`_ |
| |
| .. _2.61: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id56 |
| |
| * `2.63`_ |
| |
| .. _2.63: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id58 |
| |
| * `2.64`_ |
| |
| .. _2.64: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id59 |
| |
| * `2.70`_ |
| |
| .. _2.70: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id64 |
| |
| * `2.71`_ |
| |
| .. _2.71: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id65 |
| |
| * `2.73`_ |
| |
| .. _2.73: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id66 |
| |
| * `2.75`_ |
| |
| .. _2.75: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id68 |
| |
| * `2.79`_ |
| |
| .. _2.79: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#maximum-in-train |
| |
| * `2.86`_ |
| |
| .. _2.86: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id79 |
| |
| * Volume |
| |
| * `3.3`_ |
| |
| .. _3.3: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id3 |
| |
| * `3.9`_ |
| |
| .. _3.9: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id9 |
| |
| * `3.11`_ |
| |
| .. _3.11: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id11 |
| |
| * `3.12`_ |
| |
| .. _3.12: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id12 |
| |
| * `3.13`_ |
| |
| .. _3.13: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id13 |
| |
| * `3.14`_ |
| |
| .. _3.14: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id14 |
| |
| * `3.19`_ |
| |
| .. _3.19: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id18 |
| |
| * `3.20`_ |
| |
| .. _3.20: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id19 |
| |
| * `3.55`_ |
| |
| .. _3.55: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#maximum-in-rocky |