Update the Tempest README for new run workflow
This commit updates the tempest README to document the new workflow
for running and interacting with tempest. The quickstart section is
used for this because that is where we were previously explaining how
to run tempest. The previous instructions for running tempest will
still work (and should always continue to work, we do not want to
break that usage model) so instead of being removed a new section at
the bottom of the README is added to document this, despite not being
the currently recommended workflow. Right now tempest run is excluded
from this doc, since more work needs to be done to figure out how
that will implemented.
Partially Implements: bp tempest-cli-improvements
Change-Id: Iced98b74b7ff651423fd1efa205dc0a4aaa28d5f
diff --git a/README.rst b/README.rst
index 431be7c..1b2e474 100644
--- a/README.rst
+++ b/README.rst
@@ -37,57 +37,61 @@
Quickstart
----------
-To run Tempest, you first need to create a configuration file that
-will tell Tempest where to find the various OpenStack services and
-other testing behavior switches.
+To run Tempest, you first need to create a configuration file that will tell
+Tempest where to find the various OpenStack services and other testing behavior
+switches. Where the configuration file lives and how you interact with it
+depends on how you'll be running Tempest. There are 2 methods of using Tempest.
+The first, which is a newer and recommended workflow treats Tempest as a system
+installed program. The second older method is to run Tempest assuming your
+working dir is the actually Tempest source repo, and there are a number of
+assumptions related to that. For this section we'll only cover the newer method
+as it is simpler, and quicker to work with.
-The easiest way to create a configuration file is to generate a sample
-in the ``etc/`` directory ::
+#. You first need to install Tempest this is done with pip, after you check out
+ the Tempest repo you simply run something like::
- $> cd $TEMPEST_ROOT_DIR
- $> oslo-config-generator --config-file \
- tools/config/config-generator.tempest.conf \
- --output-file etc/tempest.conf
+ $ pip install tempest
-After that, open up the ``etc/tempest.conf`` file and edit the
-configuration variables to match valid data in your environment.
-This includes your Keystone endpoint, a valid user and credentials,
-and reference data to be used in testing.
+ This can be done within a venv, but the assumption for this guide is that
+ the Tempest cli entry point will be in your shell's PATH.
-.. note::
+#. Installing Tempest will create a /etc/tempest dir which will contain the
+ sample config file packaged with Tempest. The contents of /etc/tempest will
+ be copied to all local working dirs, so if there is any common configuration
+ you'd like to be shared between anyone setting up local Tempest working dirs
+ it's recommended that you copy or rename tempest.conf.sample to tempest.conf
+ and make those changes to that file in /etc/tempest
- If you have a running devstack environment, Tempest will be
- automatically configured and placed in ``/opt/stack/tempest``. It
- will have a configuration file already set up to work with your
- devstack installation.
+#. Setup a local working Tempest dir. This is done using the tempest init
+ command::
-Tempest is not tied to any single test runner, but `testr`_ is the most commonly
-used tool. Also, the nosetests test runner is **not** recommended to run Tempest.
+ tempest init cloud-01
-After setting up your configuration file, you can execute the set of Tempest
-tests by using ``testr`` ::
+ works the same as::
- $> testr run --parallel
+ mkdir cloud-01 && cd cloud-01 && tempest init
-.. _testr: http://testrepository.readthedocs.org/en/latest/MANUAL.html
+ This will create a new directory for running a single Tempest configuration.
+ If you'd like to run Tempest against multiple OpenStack deployments the idea
+ is that you'll create a new working directory for each to maintain separate
+ configuration files and local artifact storage for each.
-To run one single test serially ::
+#. Then cd into the newly created working dir and also modify the local
+ config files located in the etc/ subdir created by the ``tempest init``
+ command. Tempest is expecting a tempest.conf file in etc/ so if only a
+ sample exists you must rename or copy it to tempest.conf before making
+ any changes to it otherwise Tempest will not know how to load it.
- $> testr run tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_reboot_non_existent_server
+#. Once the configuration is done you're now ready to run Tempest. This can
+ be done with testr directly or any `testr`_ based test runner, like
+ `ostestr`_. For example, from the working dir running::
-Alternatively, you can use the run_tempest.sh script which will create a venv
-and run the tests or use tox to do the same. Tox also contains several existing
-job configurations. For example::
+ $ ostestr --regex '(?!.*\[.*\bslow\b.*\])(^tempest\.(api|scenario|thirdparty))'
- $> tox -efull
+ will run the same set of tests as the default gate jobs.
-which will run the same set of tests as the OpenStack gate. (it's exactly how
-the gate invokes Tempest) Or::
-
- $> tox -esmoke
-
-to run the tests tagged as smoke.
-
+.. _testr: https://testrepository.readthedocs.org/en/latest/MANUAL.html
+.. _ostestr: http://docs.openstack.org/developer/os-testr/
Configuration
-------------
@@ -146,3 +150,62 @@
there might be uncaught issues running on Python 3.4. So until there is a gating
job which does a full Tempest run using Python 3.4 there isn't any guarantee
that running Tempest under Python 3.4 is bug free.
+
+Legacy run method
+-----------------
+
+The legacy method of running Tempest is to just treat the Tempest source code
+as a python unittest repository and run directly from the source repo. When
+running in this way you still start with a Tempest config file and the steps
+are basically the same except that it expects you know where the Tempest code
+lives on your system and requires a bit more manual interaction to get Tempest
+running. For example, when running Tempest this way things like a lock file
+directory do not get generated automatically and the burden is on the user to
+create and configure that.
+
+To start you need to create a configuration file. The easiest way to create a
+configuration file is to generate a sample in the ``etc/`` directory ::
+
+ $> cd $TEMPEST_ROOT_DIR
+ $> oslo-config-generator --config-file \
+ tools/config/config-generator.tempest.conf \
+ --output-file etc/tempest.conf
+
+After that, open up the ``etc/tempest.conf`` file and edit the
+configuration variables to match valid data in your environment.
+This includes your Keystone endpoint, a valid user and credentials,
+and reference data to be used in testing.
+
+.. note::
+
+ If you have a running devstack environment, Tempest will be
+ automatically configured and placed in ``/opt/stack/tempest``. It
+ will have a configuration file already set up to work with your
+ devstack installation.
+
+Tempest is not tied to any single test runner, but `testr`_ is the most commonly
+used tool. Also, the nosetests test runner is **not** recommended to run Tempest.
+
+After setting up your configuration file, you can execute the set of Tempest
+tests by using ``testr`` ::
+
+ $> testr run --parallel
+
+.. _testr: http://testrepository.readthedocs.org/en/latest/MANUAL.html
+
+To run one single test serially ::
+
+ $> testr run tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_reboot_non_existent_server
+
+Alternatively, you can use the run_tempest.sh script which will create a venv
+and run the tests or use tox to do the same. Tox also contains several existing
+job configurations. For example::
+
+ $> tox -efull
+
+which will run the same set of tests as the OpenStack gate. (it's exactly how
+the gate invokes Tempest) Or::
+
+ $> tox -esmoke
+
+to run the tests tagged as smoke.