docs: Add Patrole overview documentation
This patchset adds a Patrole overview documentation
section which is sorely needed. It combines the previous
usage documentation into it.
Change-Id: Ia7412f2e99f33fbdfd2e60ba54ffdba757d1f886
diff --git a/README.rst b/README.rst
index 0c786b9..fb8976f 100644
--- a/README.rst
+++ b/README.rst
@@ -33,10 +33,22 @@
* *Atomicity*. Patrole tests should be atomic: they should test policies in
isolation. Unlike Tempest, a Patrole test strives to only call a single
endpoint at a time.
-* *Holistic coverage*. Patrole strives for complete coverage of the OpenStack
- API. Additionally, Patrole strives to test the API-to-policy mapping
- contained in each project's policy in code documentation.
-* *Self-contained*. Patrole should attempt to clean up after itself; whenever
+* *Complete coverage*. Patrole should validate all policy in code defaults. For
+ testing, Patrole uses the API-to-policy mapping contained in each project's
+ `policy in code`_ documentation where applicable.
+
+ For example, Nova's policy in code documentation is located in the
+ `Nova repository`_ under ``nova/policies``. Likewise, Keystone's policy in
+ code documentation is located in the `Keystone repository`_ under
+ ``keystone/common/policies``. The other OpenStack services follow the same
+ directory layout pattern with respect to policy in code.
+
+ .. note::
+
+ Realistically this is not always possible because some services have
+ not yet moved to policy in code.
+
+* *Self-cleaning*. Patrole should attempt to clean up after itself; whenever
possible we should tear down resources when done.
.. note::
@@ -45,7 +57,11 @@
pre-provisioned credentials. Work is currently underway to clean up
modifications made to pre-provisioned credentials.
-* *Self-tested*. Patrole should be self-tested.
+* *Self-testing*. Patrole should be self-testing.
+
+.. _policy in code: https://specs.openstack.org/openstack/oslo-specs/specs/newton/policy-in-code.html
+.. _Nova repository: https://github.com/openstack/nova/tree/master/nova/policies
+.. _Keystone repository: https://github.com/openstack/keystone/tree/master/keystone/common/policies
Features
--------
diff --git a/doc/source/HACKING.rst b/doc/source/HACKING.rst
index 1847447..8777875 100644
--- a/doc/source/HACKING.rst
+++ b/doc/source/HACKING.rst
@@ -1,4 +1,5 @@
-=======
-Hacking
-=======
+====================
+Patrole Coding Guide
+====================
+
.. include:: ../../HACKING.rst
diff --git a/doc/source/framework/rbac_utils.rst b/doc/source/framework/rbac_utils.rst
index 69ba045..7143928 100644
--- a/doc/source/framework/rbac_utils.rst
+++ b/doc/source/framework/rbac_utils.rst
@@ -23,8 +23,19 @@
and test execution, respectively. This is especially true when considering
custom policy rule definitions, which can be arbitrarily complex.
-Patrole, therefore, implicitly splits up each test into 3 stages: set up,
-test execution, and teardown.
+.. _role-overriding:
+
+Role Overriding
+^^^^^^^^^^^^^^^
+
+Role overriding is the way Patrole is able to create resources and delete
+resources -- including those that require admin credentials -- while still
+being able to exercise the same set of Tempest credentials to perform the API
+action that authorizes the policy under test, by manipulating the role of
+the Tempest credentials.
+
+Patrole implicitly splits up each test into 3 stages: set up, test execution,
+and teardown.
The role workflow is as follows:
@@ -43,7 +54,7 @@
Test Setup
----------
-Automatic role switch in background.
+Automatic role override in background.
Resources can be set up inside the ``resource_setup`` class method that Tempest
provides. These resources are typically reserved for "expensive" resources
@@ -59,7 +70,7 @@
Test Execution
--------------
-Manual role switch required.
+Manual role override required.
"Test execution" here means calling the API endpoint that enforces the policy
action expected by the ``rbac_rule_validation`` decorator. Test execution
@@ -152,7 +163,7 @@
Test Cleanup
------------
-Automatic role switch in background.
+Automatic role override in background.
After the test -- no matter whether it ended successfully or in failure --
the credentials are overridden with the admin role by the Patrole framework,
diff --git a/doc/source/index.rst b/doc/source/index.rst
index 8368262..255fd9a 100644
--- a/doc/source/index.rst
+++ b/doc/source/index.rst
@@ -2,6 +2,14 @@
Patrole: Tempest Plugin for RBAC Testing
========================================
+Overview
+========
+
+.. toctree::
+ :maxdepth: 2
+
+ overview
+
User's Guide
============
@@ -12,7 +20,6 @@
:maxdepth: 2
configuration
- usage
Patrole Installation Guide
--------------------------
diff --git a/doc/source/overview.rst b/doc/source/overview.rst
new file mode 100644
index 0000000..795359e
--- /dev/null
+++ b/doc/source/overview.rst
@@ -0,0 +1,178 @@
+========================
+Team and repository tags
+========================
+
+.. image:: https://governance.openstack.org/tc/badges/patrole.svg
+ :target: https://governance.openstack.org/tc/reference/tags/index.html
+
+Patrole - The OpenStack RBAC Validation Test Suite
+==================================================
+
+The documentation for Patrole is officially hosted at:
+https://docs.openstack.org/patrole/latest/
+
+This is a set of integration tests to be run against a live OpenStack
+cluster. Patrole has a battery of tests dedicated to validating the correctness
+and security of the cloud's RBAC implementation.
+
+Design Principles
+-----------------
+
+As a `Tempest plugin`_, Patrole borrows some `design principles`_ from Tempest,
+but not all, as its testing scope is confined to policies.
+
+* Patrole uses OpenStack public interfaces. Tests in Patrole should only touch
+ public OpenStack APIs.
+* Patrole tests should be atomic: they should test policies in isolation.
+ Unlike Tempest, a Patrole test strives to only call a single endpoint at a
+ time. This is because it is important to validate each policy is authorized
+ correctly and the best way to do that is to validate the policy alone.
+* Patrole should validate all policy in code defaults. For testing, Patrole
+ uses the API-to-policy mapping contained in each project's `policy in code`_
+ documentation where applicable.
+
+ For example, Nova's policy in code documentation is located in the
+ `Nova repository`_ under ``nova/policies``. Likewise, Keystone's policy in
+ code documentation is located in the `Keystone repository`_ under
+ ``keystone/common/policies``. The other OpenStack services follow the same
+ directory layout pattern with respect to policy in code.
+
+ .. note::
+
+ Realistically this is not always possible because some services have
+ not yet moved to policy in code.
+
+* Patrole should attempt to clean up after itself; whenever possible it should
+ tear down resources when done.
+
+ .. note::
+
+ Patrole modifies roles dynamically in the background, which affects
+ pre-provisioned credentials. Work is currently underway to clean up
+ modifications made to pre-provisioned credentials.
+
+* Patrole should be self-testing.
+
+.. _Tempest plugin: https://docs.openstack.org/tempest/latest/plugin.html
+.. _design principles: https://docs.openstack.org/tempest/latest/overview.html#design-principles
+.. _policy in code: https://specs.openstack.org/openstack/oslo-specs/specs/newton/policy-in-code.html
+.. _Nova repository: https://github.com/openstack/nova/tree/master/nova/policies
+.. _Keystone repository: https://github.com/openstack/keystone/tree/master/keystone/common/policies
+
+Quickstart
+----------
+
+To run Patrole, you must first have `Tempest`_ installed and configured
+properly. Please reference Tempest's `Quickstart`_ guide to do so. Follow all
+the steps outlined therein. Afterward, proceed with the steps below.
+
+#. You first need to install Patrole. This is done with pip after you check out
+ the Patrole repo::
+
+ $ git clone https://git.openstack.org/openstack/patrole
+ $ pip install patrole/
+
+ This can be done within a venv.
+
+ .. note::
+
+ You may also install Patrole from source code by running::
+
+ pip install -e patrole/
+
+#. Next you must properly configure Patrole, which is relatively
+ straightforward. For details on configuring Patrole refer to the
+ :ref:`patrole-configuration`.
+
+#. Once the configuration is done you're now ready to run Patrole. This can
+ be done using the `tempest_run`_ command. This can be done by running::
+
+ $ tempest run --regex '^patrole_tempest_plugin\.tests\.api'
+
+ There is also the option to use testr directly, or any `testr`_ based test
+ runner, like `ostestr`_. For example, from the workspace dir run::
+
+ $ stestr --regex '(?!.*\[.*\bslow\b.*\])(^patrole_tempest_plugin\.tests\.api))'
+
+ will run the same set of tests as the default gate jobs.
+
+ You can also run Patrole tests using `tox`_. To do so, ``cd`` into the
+ **Tempest** directory and run::
+
+ $ tox -eall-plugin -- patrole_tempest_plugin.tests.api
+
+ .. note::
+
+ It is possible to run Patrole via ``tox -eall`` in order to run Patrole
+ isolated from other plugins. This can be accomplished by including the
+ installation of services that currently use policy in code -- for example,
+ Nova and Keystone. For example::
+
+ $ tox -evenv-tempest -- pip install /opt/stack/patrole /opt/stack/keystone /opt/stack/nova
+ $ tox -eall -- patrole_tempest_plugin.tests.api
+
+#. Log information from tests is captured in ``tempest.log`` under the Tempest
+ repository. Some Patrole debugging information is captured in that log
+ related to expected test results and :ref:`role-overriding`.
+
+ More detailed RBAC testing log output is emitted to ``patrole.log`` under
+ the Patrole repository. To configure Patrole's logging, see the
+ :ref:`patrole-configuration` guide.
+
+.. _Tempest: https://github.com/openstack/tempest
+.. _Quickstart: https://docs.openstack.org/tempest/latest/overview.html#quickstart
+.. _tempest_run: https://docs.openstack.org/tempest/latest/run.html
+.. _testr: https://testrepository.readthedocs.org/en/latest/MANUAL.html
+.. _ostestr: https://docs.openstack.org/os-testr/latest/
+.. _tox: https://tox.readthedocs.io/en/latest/
+
+RBAC Tests
+----------
+
+To change the role that the patrole tests are being run as, edit
+``rbac_test_role`` in the ``patrole`` section of tempest.conf: ::
+
+ [patrole]
+ rbac_test_role = member
+ ...
+
+.. note::
+
+ The ``rbac_test_role`` is service-specific. member, for example,
+ is an arbitrary role, but by convention is used to designate the default
+ non-admin role in the system. Most Patrole tests should be run with
+ **admin** and **member** roles. However, other services may use entirely
+ different roles.
+
+For more information about the member role and its nomenclature,
+please see: `<https://ask.openstack.org/en/question/4759/member-vs-_member_/>`__.
+
+Unit Tests
+----------
+
+Patrole also has a set of unit tests which test the Patrole code itself. These
+tests can be run by specifying the test discovery path::
+
+ $ stestr --test-path ./patrole_tempest_plugin/tests/unit run
+
+By setting ``--test-path`` option to ``./patrole_tempest_plugin/tests/unit``
+it specifies that test discovery should only be run on the unit test directory.
+
+Alternatively, there are the py27 and py35 tox jobs which will run the unit
+tests with the corresponding version of Python.
+
+One common activity is to just run a single test; you can do this with tox
+simply by specifying to just run py27 or py35 tests against a single test::
+
+ $ tox -e py27 -- -n patrole_tempest_plugin.tests.unit.test_rbac_utils.RBACUtilsTest.test_override_role_with_missing_admin_role
+
+Or all tests in the test_rbac_utils.py file::
+
+ $ tox -e py27 -- -n patrole_tempest_plugin.tests.unit.test_rbac_utils
+
+You may also use regular expressions to run any matching tests::
+
+ $ tox -e py27 -- test_rbac_utils
+
+For more information on these options and details about stestr, please see the
+`stestr documentation <http://stestr.readthedocs.io/en/latest/MANUAL.html>`_.
diff --git a/doc/source/usage.rst b/doc/source/usage.rst
deleted file mode 100644
index 14c2cc7..0000000
--- a/doc/source/usage.rst
+++ /dev/null
@@ -1,61 +0,0 @@
-.. _patrole-usage:
-
-========
-Usage
-========
-
-Patrole (API) Tests
-===================
-
-If Patrole is installed correctly, then the RBAC tests can be executed
-from inside the tempest root directory as follows::
-
- $ tox -eall-plugin -- patrole_tempest_plugin.tests.api
-
-To execute patrole tests for a specific module, run::
-
- $ tox -eall-plugin -- patrole_tempest_plugin.tests.api.compute
-
-.. note::
-
- It is possible to run Patrole via ``tox -eall`` in order to run Patrole
- isolated from other plugins. This can be accomplished by including the
- installation of services that currently use policy in code -- for example,
- Nova and Keystone. For example::
-
- $ tox -evenv-tempest -- pip install /opt/stack/patrole /opt/stack/keystone /opt/stack/nova
- $ tox -eall -- patrole_tempest_plugin.tests.api
-..
-
-To change the role that the patrole tests are being run as, edit
-``rbac_test_role`` in the ``patrole`` section of tempest.conf: ::
-
- [patrole]
- rbac_test_role = Member
- ...
-
-.. note::
-
- The ``rbac_test_role`` is service-specific. Member, for example,
- is an arbitrary role, but by convention is used to designate the default
- non-admin role in the system. Most patrole tests should be run with
- **admin** and **Member** roles. However, some services, like Heat, take
- advantage of a role called **heat_stack_user**, as it appears frequently
- in Heat's policy.json.
-
-For more information about the Member role,
-please see: `<https://ask.openstack.org/en/question/4759/member-vs-_member_/>`__.
-
-Unit Tests
-==========
-
-Patrole includes unit tests for its RBAC framework. They can be run by
-executing::
-
- $ tox -e py27
-
-or::
-
- $ tox -e py35
-
-against the Python 3.5 interpreter.