Merge "Move wait_for_resource_status()"
diff --git a/doc/source/configuration.rst b/doc/source/configuration.rst
index 524c0fa..367be41 100644
--- a/doc/source/configuration.rst
+++ b/doc/source/configuration.rst
@@ -246,6 +246,21 @@
run. This section covers the different methods of configuring Tempest to provide
a network when creating servers.
+The ``validation`` group gathers all the connection options to remotely access the
+created servers.
+
+To enable remote access to servers, at least the three following options need to be
+set:
+
+* The ``run_validation`` option needs be set to ``true``.
+
+* The ``connect_method`` option. Two connect methods are available: ``fixed`` and
+ ``floating``, the later being set by default.
+
+* The ``auth_method`` option. Currently, only authentication by keypair is
+ available.
+
+
Fixed Network Name
""""""""""""""""""
This is the simplest method of specifying how networks should be used. You can
@@ -296,7 +311,7 @@
With Dynamic Credentials
""""""""""""""""""""""""
With dynamic credentials enabled and using nova-network, your only option for
-configuration is to either set a fixed network name or not. However, in most
+configuration is to either set a fixed network name or not. However, in most
cases it shouldn't matter because nova-network should have no problem booting a
server with multiple networks. If this is not the case for your cloud then using
an accounts file is recommended because it provides the necessary flexibility to
diff --git a/doc/source/index.rst b/doc/source/index.rst
index 17def1c..5f357b2 100644
--- a/doc/source/index.rst
+++ b/doc/source/index.rst
@@ -12,6 +12,7 @@
REVIEWING
plugin
library
+ microversion_testing
------------
Field Guides
diff --git a/doc/source/microversion_testing.rst b/doc/source/microversion_testing.rst
new file mode 100644
index 0000000..bb48b3f
--- /dev/null
+++ b/doc/source/microversion_testing.rst
@@ -0,0 +1,206 @@
+===================================
+How To Implement Microversion Tests
+===================================
+
+Tempest provides stable interfaces to test API microversion.
+For Details, see: `API Microversion testing Framework`_
+This document explains how to implement microversion tests using those
+interfaces.
+
+.. _API Microversion testing Framework: http://docs.openstack.org/developer/tempest/library/api_microversion_testing.html
+
+
+Configuration options for Microversion
+""""""""""""""""""""""""""""""""""""""
+
+* Add configuration options for specifying test target Microversions.
+ We need to specify test target Microversions because the supported
+ microversions may be different between OpenStack clouds. For operating
+ multiple Microversion tests in a single Tempest operation, configuration
+ options should represent the range of test target Microversions.
+ New configuration options are:
+ - min_microversion
+ - max_microversion
+ Those should be defined under respective section of each service.
+ For Example::
+ [compute]
+ min_microversion = None
+ max_microversion = latest
+
+
+How To Implement Microversion Tests
+"""""""""""""""""""""""""""""""""""
+
+Step1: Add skip logic based on configured microversion range
+''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
+
+Add logic to skip the tests based on Tests class and configured Microversion
+range.
+api_version_utils.check_skip_with_microversion function can be used
+to automatically skip the tests which does not fall under configured
+Microversion range.
+For example::
+
+ class BaseTestCase1(api_version_utils.BaseMicroversionTest):
+
+ [..]
+ @classmethod
+ def skip_checks(cls):
+ super(BaseTestCase1, cls).skip_checks()
+ api_version_utils.check_skip_with_microversion(cls.min_microversion,
+ cls.max_microversion,
+ CONF.compute.min_microversion,
+ CONF.compute.max_microversion)
+
+Skip logic can be added in tests base class or any specific test class depends on
+tests class structure.
+
+Step2: Selected API request microversion
+''''''''''''''''''''''''''''''''''''''''
+
+Select appropriate Microversion which needs to be used
+to send with API request.
+api_version_utils.select_request_microversion function can be used
+to select the appropriate microversion which will be used for API request.
+For example::
+
+ @classmethod
+ def resource_setup(cls):
+ super(BaseTestCase1, cls).resource_setup()
+ cls.request_microversion = (
+ api_version_utils.select_request_microversion(
+ cls.min_microversion,
+ CONF.compute.min_microversion))
+
+
+Step3: Set Microversion on Service Clients
+''''''''''''''''''''''''''''''''''''''''''
+
+Microversion selected by Test Class in previous step needs to be set on
+service clients so that APIs can be requested with selected microversion.
+
+Microversion can be defined as global variable on service clients which
+can be set using fixture.
+Also microversion header name needs to be defined on service clients which
+should be constant because it is not supposed to be changed by project
+as per API contract.
+For example::
+
+ COMPUTE_MICROVERSION = None
+
+ class BaseClient1(rest_client.RestClient):
+ api_microversion_header_name = 'X-OpenStack-Nova-API-Version'
+
+Now test class can set the selected microversion on required service clients
+using fixture which can take care of reseting the same once tests is completed.
+For example::
+
+ def setUp(self):
+ super(BaseTestCase1,, self).setUp()
+ self.useFixture(api_microversion_fixture.APIMicroversionFixture(
+ self.request_microversion))
+
+Service clients needs to add set microversion in API request header which
+can be done by overriding the get_headers() method of rest_client.
+For example::
+
+ COMPUTE_MICROVERSION = None
+
+ class BaseClient1(rest_client.RestClient):
+ api_microversion_header_name = 'X-OpenStack-Nova-API-Version'
+
+ def get_headers(self):
+ headers = super(BaseClient1, self).get_headers()
+ if COMPUTE_MICROVERSION:
+ headers[self.api_microversion_header_name] = COMPUTE_MICROVERSION
+ return headers
+
+
+Step4: Separate Test classes for each Microversion
+''''''''''''''''''''''''''''''''''''''''''''''''''
+
+This is last step to implement microversion test class.
+
+For any microversion tests, basically we need to implement a
+separate test class. In addition, each test class defines its
+microversion range with class variable like min_microversion
+and max_microversion. Tests will be valid for that defined range.
+If that range is out of configured microversion range then, test
+will be skipped.
+
+*NOTE: Microversion testing is supported at test class level not at individual
+test case level.*
+For example:
+
+Below test is applicable for microversion from 2.2 till 2.9::
+
+ class BaseTestCase1(api_version_utils.BaseMicroversionTest,
+ tempest.test.BaseTestCase):
+
+ [..]
+
+
+ class Test1(BaseTestCase1):
+ min_microversion = '2.2'
+ max_microversion = '2.9'
+
+ [..]
+
+Below test is applicable for microversion from 2.10 till latest::
+
+ class Test2(BaseTestCase1):
+ min_microversion = '2.10'
+ max_microversion = 'latest'
+
+ [..]
+
+
+
+
+Notes about Compute Microversion Tests
+"""""""""""""""""""""""""""""""""""
+Some of the compute Microversion tests have been already implemented
+with the microversion testing framework. So for further tests only
+step 4 is needed.
+
+Along with that JSON response schema might need versioning if needed.
+
+Compute service clients strictly validate the response against defined JSON
+schema and does not allow additional elements in response.
+So if that microversion changed the API response then schema needs to be version.
+New JSON schema file needs be defined with new response attributes and service
+client methods will select the schema based on requested microversion.
+
+If microversion tests are implemented randomly means not
+in sequence order(v2.20 tests added and previous microversion tests are not yet added)
+then, still schema might needs to be version for older microversion if they changed
+the response.
+This is because Nova microversion includes all the previous microversions behvaior.
+
+For Example:
+ Implementing the v2.20 microversion tests before v2.9 and 2.19-
+ v2.20 API request will respond as latest behavior of Nova till v2.20,
+ and in v2.9 and 2.19, server response has been changed so response schema needs
+ to be version accordingly.
+
+That can be done by using the get_schema method in below module:
+
+The base_compute_client module
+''''''''''''''''''''''''''''''
+
+.. automodule:: tempest.lib.services.compute.base_compute_client
+ :members:
+
+
+Microversion tests implemented in Tempest
+"""""""""""""""""""""""""""""""""""""""""
+
+* Compute
+
+ * `2.1`_
+
+ .. _2.1: http://docs.openstack.org/developer/nova/api_microversion_history.html#id1
+
+ * `2.2`_
+
+ .. _2.2: http://docs.openstack.org/developer/nova/api_microversion_history.html#id2
diff --git a/tempest/api/compute/floating_ips/test_floating_ips_actions.py b/tempest/api/compute/floating_ips/test_floating_ips_actions.py
index 85fd4ab..3d7f4f8 100644
--- a/tempest/api/compute/floating_ips/test_floating_ips_actions.py
+++ b/tempest/api/compute/floating_ips/test_floating_ips_actions.py
@@ -16,9 +16,12 @@
from tempest.api.compute.floating_ips import base
from tempest.common.utils import data_utils
from tempest.common import waiters
+from tempest import config
from tempest.lib import exceptions as lib_exc
from tempest import test
+CONF = config.CONF
+
class FloatingIPsTestJSON(base.BaseFloatingIPsTest):
server_id = None
@@ -38,7 +41,8 @@
server = cls.create_test_server(wait_until='ACTIVE')
cls.server_id = server['id']
# Floating IP creation
- body = cls.client.create_floating_ip()['floating_ip']
+ body = cls.client.create_floating_ip(
+ pool=CONF.network.floating_network_name)['floating_ip']
cls.floating_ip_id = body['id']
cls.floating_ip = body['ip']
@@ -62,7 +66,8 @@
def test_allocate_floating_ip(self):
# Positive test:Allocation of a new floating IP to a project
# should be successful
- body = self.client.create_floating_ip()['floating_ip']
+ body = self.client.create_floating_ip(
+ pool=CONF.network.floating_network_name)['floating_ip']
floating_ip_id_allocated = body['id']
self.addCleanup(self.client.delete_floating_ip,
floating_ip_id_allocated)
@@ -78,7 +83,8 @@
# Positive test:Deletion of valid floating IP from project
# should be successful
# Creating the floating IP that is to be deleted in this method
- floating_ip_body = self.client.create_floating_ip()['floating_ip']
+ floating_ip_body = self.client.create_floating_ip(
+ pool=CONF.network.floating_network_name)['floating_ip']
self.addCleanup(self._try_delete_floating_ip, floating_ip_body['id'])
# Deleting the floating IP from the project
self.client.delete_floating_ip(floating_ip_body['id'])
diff --git a/tempest/api/compute/floating_ips/test_list_floating_ips.py b/tempest/api/compute/floating_ips/test_list_floating_ips.py
index d003967..5738293 100644
--- a/tempest/api/compute/floating_ips/test_list_floating_ips.py
+++ b/tempest/api/compute/floating_ips/test_list_floating_ips.py
@@ -14,8 +14,11 @@
# under the License.
from tempest.api.compute import base
+from tempest import config
from tempest import test
+CONF = config.CONF
+
class FloatingIPDetailsTestJSON(base.BaseV2ComputeTest):
@@ -31,7 +34,8 @@
cls.floating_ip = []
cls.floating_ip_id = []
for i in range(3):
- body = cls.client.create_floating_ip()['floating_ip']
+ body = cls.client.create_floating_ip(
+ pool=CONF.network.floating_network_name)['floating_ip']
cls.floating_ip.append(body)
cls.floating_ip_id.append(body['id'])
@@ -57,7 +61,8 @@
def test_get_floating_ip_details(self):
# Positive test:Should be able to GET the details of floatingIP
# Creating a floating IP for which details are to be checked
- body = self.client.create_floating_ip()['floating_ip']
+ body = self.client.create_floating_ip(
+ pool=CONF.network.floating_network_name)['floating_ip']
floating_ip_id = body['id']
self.addCleanup(self.client.delete_floating_ip,
floating_ip_id)
diff --git a/tempest/api/compute/servers/test_server_rescue.py b/tempest/api/compute/servers/test_server_rescue.py
index 12b824f..32faf7d 100644
--- a/tempest/api/compute/servers/test_server_rescue.py
+++ b/tempest/api/compute/servers/test_server_rescue.py
@@ -41,7 +41,8 @@
super(ServerRescueTestJSON, cls).resource_setup()
# Floating IP creation
- body = cls.floating_ips_client.create_floating_ip()['floating_ip']
+ body = cls.floating_ips_client.create_floating_ip(
+ pool=CONF.network.floating_network_name)['floating_ip']
cls.floating_ip_id = str(body['id']).strip()
cls.floating_ip = str(body['ip']).strip()
diff --git a/tempest/common/validation_resources.py b/tempest/common/validation_resources.py
index 84d0143..c3c9a41 100644
--- a/tempest/common/validation_resources.py
+++ b/tempest/common/validation_resources.py
@@ -59,7 +59,9 @@
create_ssh_security_group(os, add_rule)
if validation_resources['floating_ip']:
floating_client = os.compute_floating_ips_client
- validation_data.update(floating_client.create_floating_ip())
+ validation_data.update(
+ floating_client.create_floating_ip(
+ pool=CONF.network.floating_network_name))
return validation_data
diff --git a/tempest/scenario/manager.py b/tempest/scenario/manager.py
index 8ba5f9a..c9ac0a4 100644
--- a/tempest/scenario/manager.py
+++ b/tempest/scenario/manager.py
@@ -600,6 +600,8 @@
def create_floating_ip(self, thing, pool_name=None):
"""Create a floating IP and associates to a server on Nova"""
+ if not pool_name:
+ pool_name = CONF.network.floating_network_name
floating_ip = (self.compute_floating_ips_client.
create_floating_ip(pool=pool_name)['floating_ip'])
self.addCleanup(self.delete_wrapper,
diff --git a/tempest/services/orchestration/json/orchestration_client.py b/tempest/services/orchestration/json/orchestration_client.py
index 6019cf5..ea5dbe5 100644
--- a/tempest/services/orchestration/json/orchestration_client.py
+++ b/tempest/services/orchestration/json/orchestration_client.py
@@ -151,42 +151,6 @@
self.expected_success(204, resp.status)
return rest_client.ResponseBody(resp)
- def wait_for_resource_status(self, stack_identifier, resource_name,
- status, failure_pattern='^.*_FAILED$'):
- """Waits for a Resource to reach a given status."""
- start = int(time.time())
- fail_regexp = re.compile(failure_pattern)
-
- while True:
- try:
- body = self.show_resource(
- stack_identifier, resource_name)['resource']
- except lib_exc.NotFound:
- # ignore this, as the resource may not have
- # been created yet
- pass
- else:
- resource_name = body['resource_name']
- resource_status = body['resource_status']
- if resource_status == status:
- return
- if fail_regexp.search(resource_status):
- raise exceptions.StackResourceBuildErrorException(
- resource_name=resource_name,
- stack_identifier=stack_identifier,
- resource_status=resource_status,
- resource_status_reason=body['resource_status_reason'])
-
- if int(time.time()) - start >= self.build_timeout:
- message = ('Resource %s failed to reach %s status '
- '(current %s) within the required time (%s s).' %
- (resource_name,
- status,
- resource_status,
- self.build_timeout))
- raise exceptions.TimeoutException(message)
- time.sleep(self.build_interval)
-
def wait_for_stack_status(self, stack_identifier, status,
failure_pattern='^.*_FAILED$'):
"""Waits for a Stack to reach a given status."""
diff --git a/tempest/test.py b/tempest/test.py
index fe3c770..6ba4962 100644
--- a/tempest/test.py
+++ b/tempest/test.py
@@ -611,13 +611,25 @@
'dhcp': dhcp}
@classmethod
- def get_tenant_network(cls):
+ def get_tenant_network(cls, credentials_type='primary'):
"""Get the network to be used in testing
+ :param credentials_type: The type of credentials for which to get the
+ tenant network
+
:return: network dict including 'id' and 'name'
"""
+ # Get a manager for the given credentials_type, but at least
+ # always fall back on getting the manager for primary credentials
+ if isinstance(credentials_type, six.string_types):
+ manager = cls.get_client_manager(credential_type=credentials_type)
+ elif isinstance(credentials_type, list):
+ manager = cls.get_client_manager(roles=credentials_type[1:])
+ else:
+ manager = cls.get_client_manager()
+
# Make sure cred_provider exists and get a network client
- networks_client = cls.get_client_manager().compute_networks_client
+ networks_client = manager.compute_networks_client
cred_provider = cls._get_credentials_provider()
# In case of nova network, isolated tenants are not able to list the
# network configured in fixed_network_name, even if they can use it
diff --git a/tempest/tests/test_base_test.py b/tempest/tests/test_base_test.py
new file mode 100644
index 0000000..dc355b4
--- /dev/null
+++ b/tempest/tests/test_base_test.py
@@ -0,0 +1,110 @@
+# Copyright 2016 IBM Corp.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+
+import mock
+
+from tempest import clients
+from tempest.common import credentials_factory as credentials
+from tempest.common import fixed_network
+from tempest import config
+from tempest import test
+from tempest.tests import base
+from tempest.tests import fake_config
+
+
+class TestBaseTestCase(base.TestCase):
+ def setUp(self):
+ super(TestBaseTestCase, self).setUp()
+ self.useFixture(fake_config.ConfigFixture())
+ self.fixed_network_name = 'fixed-net'
+ config.CONF.compute.fixed_network_name = self.fixed_network_name
+ config.CONF.service_available.neutron = True
+
+ @mock.patch.object(test.BaseTestCase, 'get_client_manager')
+ @mock.patch.object(test.BaseTestCase, '_get_credentials_provider')
+ @mock.patch.object(fixed_network, 'get_tenant_network')
+ def test_get_tenant_network(self, mock_gtn, mock_gprov, mock_gcm):
+ net_client = mock.Mock()
+ mock_prov = mock.Mock()
+ mock_gcm.return_value.compute_networks_client = net_client
+ mock_gprov.return_value = mock_prov
+
+ test.BaseTestCase.get_tenant_network()
+
+ mock_gcm.assert_called_once_with(credential_type='primary')
+ mock_gprov.assert_called_once_with()
+ mock_gtn.assert_called_once_with(mock_prov, net_client,
+ self.fixed_network_name)
+
+ @mock.patch.object(test.BaseTestCase, 'get_client_manager')
+ @mock.patch.object(test.BaseTestCase, '_get_credentials_provider')
+ @mock.patch.object(fixed_network, 'get_tenant_network')
+ @mock.patch.object(test.BaseTestCase, 'get_identity_version')
+ @mock.patch.object(credentials, 'is_admin_available')
+ @mock.patch.object(clients, 'Manager')
+ def test_get_tenant_network_with_nova_net(self, mock_man, mock_iaa,
+ mock_giv, mock_gtn, mock_gcp,
+ mock_gcm):
+ config.CONF.service_available.neutron = False
+ mock_prov = mock.Mock()
+ mock_admin_man = mock.Mock()
+ mock_iaa.return_value = True
+ mock_gcp.return_value = mock_prov
+ mock_man.return_value = mock_admin_man
+
+ test.BaseTestCase.get_tenant_network()
+
+ mock_man.assert_called_once_with(
+ mock_prov.get_admin_creds.return_value)
+ mock_iaa.assert_called_once_with(
+ identity_version=mock_giv.return_value)
+ mock_gcp.assert_called_once_with()
+ mock_gtn.assert_called_once_with(
+ mock_prov, mock_admin_man.compute_networks_client,
+ self.fixed_network_name)
+
+ @mock.patch.object(test.BaseTestCase, 'get_client_manager')
+ @mock.patch.object(test.BaseTestCase, '_get_credentials_provider')
+ @mock.patch.object(fixed_network, 'get_tenant_network')
+ def test_get_tenant_network_with_alt_creds(self, mock_gtn, mock_gprov,
+ mock_gcm):
+ net_client = mock.Mock()
+ mock_prov = mock.Mock()
+ mock_gcm.return_value.compute_networks_client = net_client
+ mock_gprov.return_value = mock_prov
+
+ test.BaseTestCase.get_tenant_network(credentials_type='alt')
+
+ mock_gcm.assert_called_once_with(credential_type='alt')
+ mock_gprov.assert_called_once_with()
+ mock_gtn.assert_called_once_with(mock_prov, net_client,
+ self.fixed_network_name)
+
+ @mock.patch.object(test.BaseTestCase, 'get_client_manager')
+ @mock.patch.object(test.BaseTestCase, '_get_credentials_provider')
+ @mock.patch.object(fixed_network, 'get_tenant_network')
+ def test_get_tenant_network_with_role_creds(self, mock_gtn, mock_gprov,
+ mock_gcm):
+ net_client = mock.Mock()
+ mock_prov = mock.Mock()
+ mock_gcm.return_value.compute_networks_client = net_client
+ mock_gprov.return_value = mock_prov
+ creds = ['foo_type', 'role1']
+
+ test.BaseTestCase.get_tenant_network(credentials_type=creds)
+
+ mock_gcm.assert_called_once_with(roles=['role1'])
+ mock_gprov.assert_called_once_with()
+ mock_gtn.assert_called_once_with(mock_prov, net_client,
+ self.fixed_network_name)