doc/source/plugin.rst

=============================
Tempest Test Plugin Interface
=============================

Tempest has an external test plugin interface which enables anyone to integrate
an external test suite as part of a tempest run. This will let any project
leverage being run with the rest of the tempest suite while not requiring the
tests live in the tempest tree.

Creating a plugin
=================

Creating a plugin is fairly straightforward and doesn't require much additional
effort on top of creating a test suite using tempest.lib. One thing to note with
doing this is that the interfaces exposed by tempest are not considered stable
(with the exception of configuration variables which ever effort goes into
ensuring backwards compatibility). You should not need to import anything from
tempest itself except where explicitly noted.

Stable Tempest APIs plugins may use
-----------------------------------

As noted above, several tempest APIs are acceptable to use from plugins, while
others are not. A list of stable APIs available to plugins is provided below:

* tempest.lib.*
* tempest.config
* tempest.test_discover.plugins

If there is an interface from tempest that you need to rely on in your plugin
which is not listed above, it likely needs to be migrated to tempest.lib. In
that situation, file a bug, push a migration patch, etc. to expedite providing
the interface in a reliable manner.

Plugin Cookiecutter
-------------------

In order to create the basic structure with base classes and test directories
you can use the tempest-plugin-cookiecutter project::

  > pip install -U cookiecutter && cookiecutter https://git.openstack.org/openstack/tempest-plugin-cookiecutter

  Cloning into 'tempest-plugin-cookiecutter'...
  remote: Counting objects: 17, done.
  remote: Compressing objects: 100% (13/13), done.
  remote: Total 17 (delta 1), reused 14 (delta 1)
  Unpacking objects: 100% (17/17), done.
  Checking connectivity... done.
  project (default is "sample")? foo
  testclass (default is "SampleTempestPlugin")? FooTempestPlugin

This would create a folder called ``foo_tempest_plugin/`` with all necessary
basic classes. You only need to move/create your test in
``foo_tempest_plugin/tests``.

Entry Point
-----------

Once you've created your plugin class you need to add an entry point to your
project to enable tempest to find the plugin. The entry point must be added
to the "tempest.test_plugins" namespace.

If you are using pbr this is fairly straightforward, in the setup.cfg just add
something like the following::

  [entry_points]
  tempest.test_plugins =
      plugin_name = module.path:PluginClass

Standalone Plugin vs In-repo Plugin
-----------------------------------

Since all that's required for a plugin to be detected by tempest is a valid
setuptools entry point in the proper namespace there is no difference from the
tempest perspective on either creating a separate python package to
house the plugin or adding the code to an existing python project. However,
there are tradeoffs to consider when deciding which approach to take when
creating a new plugin.

If you create a separate python project for your plugin this makes a lot of
things much easier. Firstly it makes packaging and versioning much simpler, you
can easily decouple the requirements for the plugin from the requirements for
the other project. It lets you version the plugin independently and maintain a
single version of the test code across project release boundaries (see the
`Branchless Tempest Spec`_ for more details on this). It also greatly
simplifies the install time story for external users. Instead of having to
install the right version of a project in the same python namespace as tempest
they simply need to pip install the plugin in that namespace. It also means
that users don't have to worry about inadvertently installing a tempest plugin
when they install another package.

.. _Branchless Tempest Spec: http://specs.openstack.org/openstack/qa-specs/specs/tempest/implemented/branchless-tempest.html

The sole advantage to integrating a plugin into an existing python project is
that it enables you to land code changes at the same time you land test changes
in the plugin. This reduces some of the burden on contributors by not having
to land 2 changes to add a new API feature and then test it and doing it as a
single combined commit.


Plugin Class
============

To provide tempest with all the required information it needs to be able to run
your plugin you need to create a plugin class which tempest will load and call
to get information when it needs. To simplify creating this tempest provides an
abstract class that should be used as the parent for your plugin. To use this
you would do something like the following::

  from tempest.test_discover import plugins

  class MyPlugin(plugins.TempestPlugin):

Then you need to ensure you locally define all of the methods in the abstract
class, you can refer to the api doc below for a reference of what that entails.

Abstract Plugin Class
---------------------

.. autoclass:: tempest.test_discover.plugins.TempestPlugin
   :members:

Plugin Structure
================
While there are no hard and fast rules for the structure a plugin, there are
basically no constraints on what the plugin looks like as long as the 2 steps
above are done. However,  there are some recommended patterns to follow to make
it easy for people to contribute and work with your plugin. For example, if you
create a directory structure with something like::

    plugin_dir/
      config.py
      plugin.py
      tests/
        api/
        scenario/
      services/
        client.py

That will mirror what people expect from tempest. The file

* **config.py**: contains any plugin specific configuration variables
* **plugin.py**: contains the plugin class used for the entry point
* **tests**: the directory where test discovery will be run, all tests should
             be under this dir
* **services**: where the plugin specific service clients are

Additionally, when you're creating the plugin you likely want to follow all
of the tempest developer and reviewer documentation to ensure that the tests
being added in the plugin act and behave like the rest of tempest.

Dealing with configuration options
----------------------------------

Historically Tempest didn't provide external guarantees on its configuration
options. However, with the introduction of the plugin interface this is no
longer the case. An external plugin can rely on using any configuration option
coming from Tempest, there will be at least a full deprecation cycle for any
option before it's removed. However, just the options provided by Tempest
may not be sufficient for the plugin. If you need to add any plugin specific
configuration options you should use the ``register_opts`` and
``get_opt_lists`` methods to pass them to Tempest when the plugin is loaded.
When adding configuration options the ``register_opts`` method gets passed the
CONF object from tempest. This enables the plugin to add options to both
existing sections and also create new configuration sections for new options.

Using Plugins
=============

Tempest will automatically discover any installed plugins when it is run. So by
just installing the python packages which contain your plugin you'll be using
them with tempest, nothing else is really required.

However, you should take care when installing plugins. By their very nature
there are no guarantees when running tempest with plugins enabled about the
quality of the plugin. Additionally, while there is no limitation on running
with multiple plugins it's worth noting that poorly written plugins might not
properly isolate their tests which could cause unexpected cross interactions
between plugins.

Notes for using plugins with virtualenvs
----------------------------------------

When using a tempest inside a virtualenv (like when running under tox) you have
to ensure that the package that contains your plugin is either installed in the
venv too or that you have system site-packages enabled. The virtualenv will
isolate the tempest install from the rest of your system so just installing the
plugin package on your system and then running tempest inside a venv will not
work.

Tempest also exposes a tox job, all-plugin, which will setup a tox virtualenv
with system site-packages enabled. This will let you leverage tox without
requiring to manually install plugins in the tox venv before running tests.