General doc updates
Added missing doc, for example for serial tests, fixed 404 links,
spellings, formatting and etc
Change-Id: I07a8cf9508d3cba33bd04afdc301bb7145153d59
diff --git a/HACKING.rst b/HACKING.rst
index caf954b..23bc61b 100644
--- a/HACKING.rst
+++ b/HACKING.rst
@@ -184,7 +184,7 @@
the ``slow`` attribute is leveraged to run slow tests on a selective basis,
to keep total `Zuul`_ job runtime down to a reasonable time frame.
-.. _Zuul: https://docs.openstack.org/infra/zuul/
+.. _Zuul: https://zuul-ci.org/docs/zuul/latest/
Smoke Attribute
^^^^^^^^^^^^^^^
@@ -488,7 +488,7 @@
Otherwise the bug fix won't be able to land in the project.
Handily, `Zuul's cross-repository dependencies
-<https://docs.openstack.org/infra/zuul/user/gating.html#cross-project-dependencies>`_.
+<https://zuul-ci.org/docs/zuul/latest/gating.html#cross-project-dependencies>`_.
can be leveraged to do without step 2 and to have steps 3 and 4 happen
"atomically". To do that, make the patch written in step 1 to depend (refer to
Zuul's documentation above) on the patch written in step 4. The commit message
diff --git a/README.rst b/README.rst
index 3cde2bf..7880357 100644
--- a/README.rst
+++ b/README.rst
@@ -3,7 +3,6 @@
========================
.. image:: https://governance.openstack.org/tc/badges/tempest.svg
- :target: https://governance.openstack.org/tc/reference/tags/index.html
.. Change things from this point on
diff --git a/doc/source/contributor/contributing.rst b/doc/source/contributor/contributing.rst
index 139f0b7..81a1874 100644
--- a/doc/source/contributor/contributing.rst
+++ b/doc/source/contributor/contributing.rst
@@ -14,8 +14,9 @@
Communication
~~~~~~~~~~~~~
* IRC channel ``#openstack-qa`` at OFTC
-* Mailing list (prefix subjects with ``[qa]`` for faster responses)
- http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss
+* `Mailing list <https://lists.openstack.org/mailman3/lists/openstack-discuss.lists.openstack.org/>`_
+ (prefix subjects with ``[qa]`` for faster responses)
+
Contacting the Core Team
~~~~~~~~~~~~~~~~~~~~~~~~
@@ -26,25 +27,29 @@
~~~~~~~~~~~~~~~~~~~~
If you want to propose a new feature please read `Feature Proposal Process`_
Tempest features are tracked on `Launchpad BP <https://blueprints.launchpad.net/tempest>`_.
+It also helps to bring the feature up during the next PTG and contact
+`the current PTL of QA project <https://governance.openstack.org/tc/reference/projects/>`_.
+Information about PTG is always posted on `the Mailing list
+<https://lists.openstack.org/mailman3/lists/openstack-discuss.lists.openstack.org/>`_.
Task Tracking
~~~~~~~~~~~~~
We track our tasks in `Launchpad <https://bugs.launchpad.net/tempest>`_.
-If you're looking for some smaller, easier work item to pick up and get started
+If you're looking for some smaller, easier work items to pick up and get started
on, search for the 'low-hanging-fruit' tag.
Reporting a Bug
~~~~~~~~~~~~~~~
-You found an issue and want to make sure we are aware of it? You can do so on
-`Launchpad <https://bugs.launchpad.net/tempest/+filebug>`__.
+Have you found an issue and want to make sure we are aware of it? You can do so
+on `Launchpad <https://bugs.launchpad.net/tempest/+filebug>`__.
More info about Launchpad usage can be found on `OpenStack docs page
<https://docs.openstack.org/contributors/common/task-tracking.html#launchpad>`_
Getting Your Patch Merged
~~~~~~~~~~~~~~~~~~~~~~~~~
-All changes proposed to the Tempest require single ``Code-Review +2`` votes from
-Tempest core reviewers by giving ``Workflow +1`` vote. More detailed guidelines
+All changes proposed to the Tempest require a single ``Code-Review +2`` vote
+from a Tempest core followed by a ``Workflow +1`` vote. More detailed guidelines
for reviewers are available at :doc:`../REVIEWING`.
Project Team Lead Duties
diff --git a/doc/source/index.rst b/doc/source/index.rst
index 2f29cf2..0340f8d 100644
--- a/doc/source/index.rst
+++ b/doc/source/index.rst
@@ -83,6 +83,7 @@
:maxdepth: 2
HACKING
+ serial_tests
REVIEWING
microversion_testing
test_removal
diff --git a/doc/source/library/api_microversion_testing.rst b/doc/source/library/api_microversion_testing.rst
index 8be924d..e979683 100644
--- a/doc/source/library/api_microversion_testing.rst
+++ b/doc/source/library/api_microversion_testing.rst
@@ -9,9 +9,9 @@
Many of the OpenStack components have implemented API microversions.
It is important to test those microversions in Tempest or external plugins.
-Tempest now provides stable interfaces to support to test the API microversions.
+Tempest now provides stable interfaces to support testing the API microversions.
Based on the microversion range coming from the combination of both configuration
-and each test case, APIs request will be made with selected microversion.
+and each test case, APIs requests will be made with the selected microversion.
This document explains the interfaces needed for microversion testing.
diff --git a/doc/source/library/clients.rst b/doc/source/library/clients.rst
index 0f4ba4c..fe9f4ca 100644
--- a/doc/source/library/clients.rst
+++ b/doc/source/library/clients.rst
@@ -6,11 +6,11 @@
Tests make requests against APIs using service clients. Service clients are
specializations of the ``RestClient`` class. The service clients that cover the
APIs exposed by a service should be grouped in a service clients module.
-A service clients module is python module where all service clients are
+A service clients module is a Python module where all service clients are
defined. If major API versions are available, submodules should be defined,
one for each version.
-The ``ClientsFactory`` class helps initializing all clients of a specific
+The ``ClientsFactory`` class helps to initialize all clients of a specific
service client module from a set of shared parameters.
The ``ServiceClients`` class provides a convenient way to get access to all
diff --git a/doc/source/library/credential_providers.rst b/doc/source/library/credential_providers.rst
index d25f85c..8c9a16a 100644
--- a/doc/source/library/credential_providers.rst
+++ b/doc/source/library/credential_providers.rst
@@ -4,12 +4,12 @@
====================
These library interfaces are used to deal with allocating credentials on demand
-either dynamically by calling keystone to allocate new credentials, or from
+either dynamically by calling Keystone to allocate new credentials, or from
a list of preprovisioned credentials. These 2 modules are implementations of
the same abstract credential providers class and can be used interchangeably.
However, each implementation has some additional parameters that are used to
influence the behavior of the modules. The API reference at the bottom of this
-doc shows the interface definitions for both modules, however that may be a bit
+doc shows the interface definitions for both modules, however, that may be a bit
opaque. You can see some examples of how to leverage this interface below.
Initialization Example
@@ -30,7 +30,7 @@
# If a test requires a new account to work, it can have it via forcing
# dynamic credentials. A new account will be produced only for that test.
# In case admin credentials are not available for the account creation,
- # the test should be skipped else it would fail.
+ # the test should be skipped else it will fail.
identity_version = identity_version or CONF.identity.auth_version
if CONF.auth.use_dynamic_credentials or force_tenant_isolation:
admin_creds = get_configured_admin_credentials(
@@ -81,12 +81,12 @@
Once you have a credential provider object created the access patterns for
allocating and removing credentials are the same across both the dynamic
and preprovisioned credentials. These are defined in the abstract
-CredentialProvider class. At a high level the credentials provider enables
-you to get 3 basic types of credentials at once (per object): a primary, alt,
+CredentialProvider class. At a high level, the credentials provider enables
+you to get 3 basic types of credentials at once (per object): primary, alt,
and admin. You're also able to allocate a credential by role. These credentials
-are tracked by the provider object and delete must manually be called otherwise
-the created resources will not be deleted (or returned to the pool in the case
-of preprovisioned creds)
+are tracked by the provider object and delete must be called manually,
+otherwise, the created resources will not be deleted (or returned to the pool
+in the case of preprovisioned creds).
Examples
''''''''
diff --git a/doc/source/plugins/plugin.rst b/doc/source/plugins/plugin.rst
index 0771318..31aa134 100644
--- a/doc/source/plugins/plugin.rst
+++ b/doc/source/plugins/plugin.rst
@@ -80,30 +80,30 @@
Since all that's required for a plugin to be detected by Tempest is a valid
setuptools entry point in the proper namespace there is no difference from the
-Tempest perspective on either creating a separate python package to
-house the plugin or adding the code to an existing python project. However,
+Tempest perspective on either creating a separate Python package to
+house the plugin or adding the code to an existing Python project. However,
there are tradeoffs to consider when deciding which approach to take when
creating a new plugin.
-If you create a separate python project for your plugin this makes a lot of
+If you create a separate Python project for your plugin this makes a lot of
things much easier. Firstly it makes packaging and versioning much simpler, you
can easily decouple the requirements for the plugin from the requirements for
the other project. It lets you version the plugin independently and maintain a
single version of the test code across project release boundaries (see the
`Branchless Tempest Spec`_ for more details on this). It also greatly
simplifies the install time story for external users. Instead of having to
-install the right version of a project in the same python namespace as Tempest
+install the right version of a project in the same Python namespace as Tempest
they simply need to pip install the plugin in that namespace. It also means
that users don't have to worry about inadvertently installing a Tempest plugin
when they install another package.
.. _Branchless Tempest Spec: https://specs.openstack.org/openstack/qa-specs/specs/tempest/implemented/branchless-tempest.html
-The sole advantage to integrating a plugin into an existing python project is
+The sole advantage of integrating a plugin into an existing Python project is
that it enables you to land code changes at the same time you land test changes
in the plugin. This reduces some of the burden on contributors by not having
-to land 2 changes to add a new API feature and then test it and doing it as a
-single combined commit.
+to land 2 changes to add a new API feature and then test it, and do it as a
+single combined commit instead.
Plugin Class
@@ -122,7 +122,7 @@
class MyPlugin(plugins.TempestPlugin):
Then you need to ensure you locally define all of the mandatory methods in the
-abstract class, you can refer to the api doc below for a reference of what that
+abstract class, you can refer to the API doc below for a reference of what that
entails.
Abstract Plugin Class
@@ -135,7 +135,7 @@
================
While there are no hard and fast rules for the structure of a plugin, there are
basically no constraints on what the plugin looks like as long as the 2 steps
-above are done. However, there are some recommended patterns to follow to make
+above are done. However, there are some recommended patterns to follow to make
it easy for people to contribute and work with your plugin. For example, if you
create a directory structure with something like::
@@ -214,7 +214,7 @@
Parameters:
* **name**: Name of the attribute used to access the ``ClientsFactory`` from
- the ``ServiceClients`` instance. See example below.
+ the ``ServiceClients`` instance. See the example below.
* **service_version**: Tempest enforces a single implementation for each
service client. Available service clients are held in a ``ClientsRegistry``
singleton, and registered with ``service_version``, which means that
@@ -229,7 +229,7 @@
.. code-block:: python
- # my_creds is instance of tempest.lib.auth.Credentials
+ # my_creds is an instance of tempest.lib.auth.Credentials
# identity_uri is v2 or v3 depending on the configuration
from tempest.lib.services import clients
@@ -241,13 +241,13 @@
constraints on the structure of the configuration options exposed by the
plugin.
-First ``service_version`` should be in the format `service_config[.version]`.
+Firstly, ``service_version`` should be in the format `service_config[.version]`.
The `.version` part is optional, and should only be used if there are multiple
versions of the same API available. The `service_config` must match the name of
a configuration options group defined by the plugin. Different versions of one
API must share the same configuration group.
-Second the configuration options group `service_config` must contain the
+Secondly, the configuration options group `service_config` must contain the
following options:
* `catalog_type`: corresponds to `service` in the catalog
@@ -257,10 +257,10 @@
as they do not necessarily apply to all service clients.
* `region`: default to identity.region
-* `build_timeout` : default to compute.build_timeout
+* `build_timeout`: default to compute.build_timeout
* `build_interval`: default to compute.build_interval
-Third the service client classes should inherit from ``RestClient``, should
+Thirdly, the service client classes should inherit from ``RestClient``, should
accept generic keyword arguments, and should pass those arguments to the
``__init__`` method of ``RestClient``. Extra arguments can be added. For
instance:
@@ -276,7 +276,7 @@
self.my_arg = my_arg
self.my_args2 = my_arg
-Finally the service client should be structured in a python module, so that all
+Finally, the service client should be structured in a Python module, so that all
service client classes are importable from it. Each major API version should
have its own module.
@@ -299,7 +299,7 @@
__all__ = ['API1Client', 'API2Client']
The following folder and module structure is recommended for multiple major
-API version::
+API versions::
plugin_dir/
services/
@@ -325,14 +325,14 @@
=============
Tempest will automatically discover any installed plugins when it is run. So by
-just installing the python packages which contain your plugin you'll be using
+just installing the Python packages, which contain your plugin, you'll be using
them with Tempest, nothing else is really required.
-However, you should take care when installing plugins. By their very nature
+However, you should take care when installing plugins. By their very nature,
there are no guarantees when running Tempest with plugins enabled about the
quality of the plugin. Additionally, while there is no limitation on running
with multiple plugins, it's worth noting that poorly written plugins might not
-properly isolate their tests which could cause unexpected cross interactions
+properly isolate their tests which could cause unexpected cross-interactions
between plugins.
Notes for using plugins with virtualenvs
diff --git a/doc/source/serial_tests.rst b/doc/source/serial_tests.rst
new file mode 120000
index 0000000..6709115
--- /dev/null
+++ b/doc/source/serial_tests.rst
@@ -0,0 +1 @@
+../../tempest/serial_tests/README.rst
\ No newline at end of file
diff --git a/doc/source/tests/modules.rst b/doc/source/tests/modules.rst
index 026a7a5..697b011 100644
--- a/doc/source/tests/modules.rst
+++ b/doc/source/tests/modules.rst
@@ -19,3 +19,10 @@
network/modules
object_storage/modules
volume/modules
+
+Serial Tests
+------------
+.. toctree::
+ :maxdepth: 2
+
+ serial_tests/modules
diff --git a/tempest/README.rst b/tempest/README.rst
index b345032..b300dcb 100644
--- a/tempest/README.rst
+++ b/tempest/README.rst
@@ -8,7 +8,7 @@
implementations for both correctness, as well as a burn in tool for
OpenStack clouds.
-As such Tempest tests come in many flavors, each with their own rules
+As such Tempest tests come in many flavors, each with its own rules
and guidelines. Below is the overview of the Tempest repository structure
to make this clear.
@@ -17,6 +17,7 @@
tempest/
api/ - API tests
scenario/ - complex scenario tests
+ serial_tests/ - tests that run always in the serial mode
tests/ - unit tests for Tempest internals
Each of these directories contains different types of tests. What
@@ -41,13 +42,20 @@
---------------------------
Scenario tests are complex "through path" tests for OpenStack
-functionality. They are typically a series of steps where complicated
+functionality. They are typically a series of steps where a complicated
state requiring multiple services is set up exercised, and torn down.
Scenario tests should not use the existing Python clients for OpenStack,
but should instead use the Tempest implementations of clients.
+:ref:`serial_tests_guide`
+--------------------------------
+
+Tests within this category will always be executed serially from the rest of
+the test cases.
+
+
:ref:`unit_tests_field_guide`
-----------------------------
diff --git a/tempest/api/README.rst b/tempest/api/README.rst
index a796922..7051230 100644
--- a/tempest/api/README.rst
+++ b/tempest/api/README.rst
@@ -7,20 +7,20 @@
What are these tests?
---------------------
-One of Tempest's prime function is to ensure that your OpenStack cloud
+One of Tempest's prime functions is to ensure that your OpenStack cloud
works with the OpenStack API as documented. The current largest
portion of Tempest code is devoted to test cases that do exactly this.
It's also important to test not only the expected positive path on
APIs, but also to provide them with invalid data to ensure they fail
in expected and documented ways. The latter type of tests is called
-``negative tests`` in Tempest source code. Over the course of the OpenStack
-project Tempest has discovered many fundamental bugs by doing just
+``negative tests`` in Tempest source code. Throughout the OpenStack
+project, Tempest has discovered many fundamental bugs by doing just
this.
In order for some APIs to return meaningful results, there must be
enough data in the system. This means these tests might start by
-spinning up a server, image, etc, then operating on it.
+spinning up a server, image, etc., and then operating on it.
Why are these tests in Tempest?
@@ -32,7 +32,7 @@
It could be argued that some of the negative testing could be done
back in the projects themselves, and we might evolve there over time,
-but currently in the OpenStack gate this is a fundamentally important
+but currently, in the OpenStack gate, this is a fundamentally important
place to keep things.
@@ -43,7 +43,7 @@
OpenStack API, as we want to ensure that bugs aren't hidden by the
official clients.
-They should test specific API calls, and can build up complex state if
+They should test specific API calls and can build up complex states if
it's needed for the API call to be meaningful.
They should send not only good data, but bad data at the API and look
diff --git a/tempest/scenario/README.rst b/tempest/scenario/README.rst
index efcd139..6c51f22 100644
--- a/tempest/scenario/README.rst
+++ b/tempest/scenario/README.rst
@@ -7,14 +7,14 @@
What are these tests?
---------------------
-Scenario tests are "through path" tests of OpenStack
-function. Complicated setups where one part might depend on completion
+Scenario tests are "through path" tests of OpenStack function.
+Complicated setups where one part might depend on the completion
of a previous part. They ideally involve the integration between
multiple OpenStack services to exercise the touch points between them.
Any scenario test should have a real-life use case. An example would be:
-- "As operator I want to start with a blank environment":
+- "As an operator, I want to start with a blank environment":
1. upload a glance image
2. deploy a vm from it
@@ -24,12 +24,14 @@
Why are these tests in Tempest?
-------------------------------
+
This is one of Tempest's core purposes, testing the integration between
projects.
Scope of these tests
--------------------
+
Scenario tests should always use the Tempest implementation of the
OpenStack API, as we want to ensure that bugs aren't hidden by the
official clients.
@@ -40,6 +42,7 @@
Example of a good test
----------------------
+
While we are looking for interaction of 2 or more services, be
specific in your interactions. A giant "this is my data center" smoke
test is hard to debug when it goes wrong.
diff --git a/tempest/serial_tests/README.rst b/tempest/serial_tests/README.rst
new file mode 100644
index 0000000..40a50f4
--- /dev/null
+++ b/tempest/serial_tests/README.rst
@@ -0,0 +1,61 @@
+.. _serial_tests_guide:
+
+Tempest Field Guide to Serial tests
+===================================
+
+
+What are these tests?
+---------------------
+
+Tempest can run tests serially as well as in parallel, depending on the
+configuration that is fully up to the user. However, sometimes you need to
+make sure that tests are not interfering with each other via OpenStack
+resources with the other tests running in parallel. Tempest creates separate
+projects for each test class to separate project based resources between test
+cases.
+
+If your tests use resources outside of projects, e.g. host aggregates then
+you might need to explicitly separate interfering test cases. If you only need
+to separate a small set of test cases from each other then you can use the
+``LockFixture``.
+
+However, in some cases, a small set of tests needs to be run serially. For
+example, some of the host aggregate and availability zone testing needs
+compute nodes without any running nova server to be able to move compute hosts
+between availability zones. But many tempest tests start one or more nova
+servers.
+
+
+Why are these tests in Tempest?
+-------------------------------
+
+This is one of Tempest's core purposes, testing the integration between
+projects.
+
+
+Scope of these tests
+--------------------
+
+The tests should always use the Tempest implementation of the OpenStack API,
+as we want to ensure that bugs aren't hidden by the official clients.
+
+Tests should be tagged with which services they exercise, as
+determined by which client libraries are used directly by the test.
+
+
+Example of a good test
+----------------------
+
+While we are looking for interaction of 2 or more services, be specific in
+your interactions. A giant "this is my data center" smoke test is hard to
+debug when it goes wrong.
+
+The tests that need to be run serially need to be marked with the
+``@serial`` class decorator. This will make sure that even if tempest is
+configured to run the tests in parallel, these tests will always be executed
+separately from the rest of the test cases.
+
+Please note that due to test ordering optimization reasons test cases marked
+for ``@serial`` execution need to be put under ``tempest/serial_tests``
+directory. This will ensure that the serial tests will block the parallel tests
+in the least amount of time.
diff --git a/tempest/tests/README.rst b/tempest/tests/README.rst
index 0587e7b..081dd07 100644
--- a/tempest/tests/README.rst
+++ b/tempest/tests/README.rst
@@ -14,6 +14,7 @@
Why are these tests in Tempest?
-------------------------------
+
These tests exist to make sure that the mechanisms that we use inside of
Tempest are valid and remain functional. They are only here for self
validation of Tempest.
@@ -21,6 +22,7 @@
Scope of these tests
--------------------
+
Unit tests should not require an external service to be running or any extra
configuration to run. Any state that is required for a test should either be
mocked out or created in a temporary test directory. (see test_wrappers.py for
diff --git a/tox.ini b/tox.ini
index de81707..7c50a3b 100644
--- a/tox.ini
+++ b/tox.ini
@@ -359,8 +359,9 @@
sphinx-apidoc -f -o doc/source/tests/image tempest/api/image
sphinx-apidoc -f -o doc/source/tests/network tempest/api/network
sphinx-apidoc -f -o doc/source/tests/object_storage tempest/api/object_storage
- sphinx-apidoc -f -o doc/source/tests/scenario tempest/scenario
sphinx-apidoc -f -o doc/source/tests/volume tempest/api/volume
+ sphinx-apidoc -f -o doc/source/tests/scenario tempest/scenario
+ sphinx-apidoc -f -o doc/source/tests/serial_tests tempest/serial_tests
rm -rf doc/build
sphinx-build -W -b html doc/source doc/build/html
allowlist_externals =
@@ -377,8 +378,9 @@
sphinx-apidoc -f -o doc/source/tests/image tempest/api/image
sphinx-apidoc -f -o doc/source/tests/network tempest/api/network
sphinx-apidoc -f -o doc/source/tests/object_storage tempest/api/object_storage
- sphinx-apidoc -f -o doc/source/tests/scenario tempest/scenario
sphinx-apidoc -f -o doc/source/tests/volume tempest/api/volume
+ sphinx-apidoc -f -o doc/source/tests/scenario tempest/scenario
+ sphinx-apidoc -f -o doc/source/tests/serial_tests tempest/serial_tests
sphinx-build -W -b latex doc/source doc/build/pdf
make -C doc/build/pdf