docs: Add symlink to readme from overview documentation
This patchset brings the README.rst and the overview.rst
documentation together. Previously they were maintained
separately which causes maintenance issues. This adds
a symlink from the overview.rst to the README.rst so
that the overview documentation only needs to be maintained
for the README.
This also adds a Terminology subsection underneath the
"How it works" section which expands on essential terminology
needed to understand Patrole testing architecture:
* Expected Result - The expected result of a given test.
* Actual Result - The actual result of a given test.
* Final Result - A match between both expected and actual results. A mismatch
in the expected result and the actual result will result in a test failure.
* Expected: Pass | Actual: Pass - Test Case Success
* Expected: Pass | Actual: Fail - Test Case Failure
* Expected: Fail | Actual: Pass - Test Case Failure
* Expected: Fail | Actual: Fail (Expected exception) - Test Case Success
* Expected: Fail | Actual: Fail (Unexpected exception) - Test Case Failure
Change-Id: I1d640200c55ce26cfd38197ec6face1161217b17
diff --git a/README.rst b/README.rst
index fb8976f..e9c03c8 100644
--- a/README.rst
+++ b/README.rst
@@ -8,10 +8,15 @@
Patrole - RBAC Integration Tempest Plugin
=========================================
-Patrole is a security validation tool for verifying that Role-Based Access
-Control is correctly configured and enforced in a system. It runs
-`Tempest`_-based API tests using specified RBAC roles, thus allowing
-deployments to verify that only intended roles have access to those APIs.
+Patrole is a set of integration tests to be run against a live OpenStack
+cluster. It has a battery of tests dedicated to validating the correctness and
+integrity of the cloud's RBAC implementation.
+
+More importantly, Patrole is a security validation tool for verifying that
+Role-Based Access Control is correctly configured and enforced in an OpenStack
+cloud. It runs `Tempest`_-based API tests using specified RBAC roles, thus
+allowing deployments to verify that only intended roles have access to those
+APIs.
Patrole currently offers testing for the following OpenStack services: Nova,
Neutron, Glance, Cinder and Keystone.
@@ -20,19 +25,25 @@
toward policy in code, Patrole will align its testing with the appropriate
documentation.
-.. _Tempest: https://docs.openstack.org/tempest/latest/
+* Free software: Apache license
+* Documentation: https://docs.openstack.org/patrole/latest
+* Source: https://git.openstack.org/cgit/openstack/patrole
+* Bugs: https://bugs.launchpad.net/patrole
+* Release notes: https://docs.openstack.org/releasenotes/patrole/
Design Principles
-----------------
-Patrole borrows some design principles from Tempest, but not all, as its
-testing scope is confined to policies.
+As a `Tempest plugin`_, Patrole borrows some `design principles`_ from Tempest,
+but not all, as its testing scope is confined to policies.
* *Stability*. Patrole uses OpenStack public interfaces. Tests in Patrole
should only touch public OpenStack APIs.
* *Atomicity*. Patrole tests should be atomic: they should test policies in
isolation. Unlike Tempest, a Patrole test strives to only call a single
- endpoint at a time.
+ endpoint at a time. This is because it is important to validate each policy
+ is authorized correctly and the best way to do that is to validate each
+ policy alone, to avoid test contamination.
* *Complete coverage*. Patrole should validate all policy in code defaults. For
testing, Patrole uses the API-to-policy mapping contained in each project's
`policy in code`_ documentation where applicable.
@@ -59,6 +70,8 @@
* *Self-testing*. Patrole should be self-testing.
+.. _Tempest plugin: https://docs.openstack.org/tempest/latest/plugin.html
+.. _design principles: https://docs.openstack.org/tempest/latest/overview.html#design-principles
.. _policy in code: https://specs.openstack.org/openstack/oslo-specs/specs/newton/policy-in-code.html
.. _Nova repository: https://github.com/openstack/nova/tree/master/nova/policies
.. _Keystone repository: https://github.com/openstack/keystone/tree/master/keystone/common/policies
@@ -86,45 +99,135 @@
expected result) and the actual result from test execution are compared to
each other: if both results match, then the test passes; else it fails.
-* Documentation: https://docs.openstack.org/patrole/latest/
-* Bugs: https://bugs.launchpad.net/patrole
+Terminology
+^^^^^^^^^^^
+* Expected Result - The expected result of a given test.
+* Actual Result - The actual result of a given test.
+* Final Result - A match between both expected and actual results. A mismatch
+ in the expected result and the actual result will result in a test failure.
+
+ * Expected: Pass | Actual: Pass - Test Case Success
+ * Expected: Pass | Actual: Fail - Test Case Under-Permission Failure
+ * Expected: Fail | Actual: Pass - Test Case Over-Permission Failure
+ * Expected: Fail | Actual: Fail (Expected exception) - Test Case Success
+ * Expected: Fail | Actual: Fail (Unexpected exception) - Test Case Failure
Quickstart
----------
-Tempest is a prerequisite for running Patrole. If you do not have Tempest
-installed, please reference the official Tempest documentation for guidance.
+To run Patrole, you must first have `Tempest`_ installed and configured
+properly. Please reference Tempest's `Quickstart`_ guide to do so. Follow all
+the steps outlined therein. Afterward, proceed with the steps below.
-Assuming Tempest is installed, the simplest way to configure Patrole is:
+#. You first need to install Patrole. This is done with pip after you check out
+ the Patrole repo::
-1. Open up the ``tempest.conf`` configuration file and include the following
-settings:
+ $ git clone https://git.openstack.org/openstack/patrole
+ $ pip install patrole/
-.. code-block:: ini
+ This can be done within a venv.
- [rbac]
- enable_rbac = True
- rbac_test_role = admin
+ .. note::
-These settings tell Patrole to run RBAC tests using the "admin" role (which
-is the default admin role in OpenStack) to verify the default policy
-definitions used by OpenStack services. Specifying a different role
-for ``rbac_test_role`` will run Patrole tests against that role. For additional
-information about Patrole's configuration settings, please refer to
-:ref:`patrole-configuration` and :ref:`patrole-sampleconf` for a sample
-configuration file.
+ You may also install Patrole from source code by running::
-2. You are now ready to run Patrole. To do so, you can use any testr-based test
-runner::
+ pip install -e patrole/
- $ testr run patrole_tempest_plugin.tests.api
+#. Next you must properly configure Patrole, which is relatively
+ straightforward. For details on configuring Patrole refer to the
+ :ref:`patrole-configuration`.
-or::
+#. Once the configuration is done you're now ready to run Patrole. This can
+ be done using the `tempest_run`_ command. This can be done by running::
- $ ostestr --regex '(?!.*\[.*\bslow\b.*\])(^patrole_tempest_plugin\.tests\.api)'
+ $ tempest run --regex '^patrole_tempest_plugin\.tests\.api'
-It is also possible to run Patrole using tox::
+ There is also the option to use testr directly, or any `testr`_ based test
+ runner, like `ostestr`_. For example, from the workspace dir run::
- tox -eall-plugin -- patrole_tempest_plugin.tests.api
+ $ stestr --regex '(?!.*\[.*\bslow\b.*\])(^patrole_tempest_plugin\.tests\.api))'
+
+ will run the same set of tests as the default gate jobs.
+
+ You can also run Patrole tests using `tox`_. To do so, ``cd`` into the
+ **Tempest** directory and run::
+
+ $ tox -eall-plugin -- patrole_tempest_plugin.tests.api
+
+ .. note::
+
+ It is possible to run Patrole via ``tox -eall`` in order to run Patrole
+ isolated from other plugins. This can be accomplished by including the
+ installation of services that currently use policy in code -- for example,
+ Nova and Keystone. For example::
+
+ $ tox -evenv-tempest -- pip install /opt/stack/patrole /opt/stack/keystone /opt/stack/nova
+ $ tox -eall -- patrole_tempest_plugin.tests.api
+
+#. Log information from tests is captured in ``tempest.log`` under the Tempest
+ repository. Some Patrole debugging information is captured in that log
+ related to expected test results and :ref:`role-overriding`.
+
+ More detailed RBAC testing log output is emitted to ``patrole.log`` under
+ the Patrole repository. To configure Patrole's logging, see the
+ :ref:`patrole-configuration` guide.
+
+.. _Tempest: https://github.com/openstack/tempest
+.. _Quickstart: https://docs.openstack.org/tempest/latest/overview.html#quickstart
+.. _tempest_run: https://docs.openstack.org/tempest/latest/run.html
+.. _testr: https://testrepository.readthedocs.org/en/latest/MANUAL.html
+.. _ostestr: https://docs.openstack.org/os-testr/latest/
+.. _tox: https://tox.readthedocs.io/en/latest/
+
+RBAC Tests
+----------
+
+To change the role that the patrole tests are being run as, edit
+``rbac_test_role`` in the ``patrole`` section of tempest.conf: ::
+
+ [patrole]
+ rbac_test_role = member
+ ...
+
+.. note::
+
+ The ``rbac_test_role`` is service-specific. member, for example,
+ is an arbitrary role, but by convention is used to designate the default
+ non-admin role in the system. Most Patrole tests should be run with
+ **admin** and **member** roles. However, other services may use entirely
+ different roles.
+
+For more information about the member role and its nomenclature,
+please see: `<https://ask.openstack.org/en/question/4759/member-vs-_member_/>`__.
+
+Unit Tests
+----------
+
+Patrole also has a set of unit tests which test the Patrole code itself. These
+tests can be run by specifying the test discovery path::
+
+ $ stestr --test-path ./patrole_tempest_plugin/tests/unit run
+
+By setting ``--test-path`` option to ``./patrole_tempest_plugin/tests/unit``
+it specifies that test discovery should only be run on the unit test directory.
+
+Alternatively, there are the py27 and py35 tox jobs which will run the unit
+tests with the corresponding version of Python.
+
+One common activity is to just run a single test; you can do this with tox
+simply by specifying to just run py27 or py35 tests against a single test::
+
+ $ tox -e py27 -- -n patrole_tempest_plugin.tests.unit.test_rbac_utils.RBACUtilsTest.test_override_role_with_missing_admin_role
+
+Or all tests in the test_rbac_utils.py file::
+
+ $ tox -e py27 -- -n patrole_tempest_plugin.tests.unit.test_rbac_utils
+
+You may also use regular expressions to run any matching tests::
+
+ $ tox -e py27 -- test_rbac_utils
+
+For more information on these options and details about stestr, please see the
+`stestr documentation <http://stestr.readthedocs.io/en/latest/MANUAL.html>`_.
Release Versioning
------------------