feature flag: Policy feature enabled config group

A new configuration group ``[policy_feature_enabled]`` has been added to
Patrole which will be responsible for collecting the feature flags to be
used for newly introduced policies or policies that were changed in a
backwards-incompatible way.

    * create_port_fixed_ips_ip_address_policy (Neutron)
    * update_port_fixed_ips_ip_address_policy (Neutron)
    * limits_extension_used_limits_policy (Cinder)
    * volume_extension_volume_actions_attach_policy (Cinder)
    * volume_extension_volume_actions_reserve_policy (Cinder)
    * volume_extension_volume_actions_unreserve_policy (Cinder)

These feature flags will be supported until Pike release cycle
is EOL.

The motivation behind these feature flags is [0] which adds
Pike/Queens gating to Patrole. However, in Queens, Neutron
and Cinder renamed or removed a few policies in a backwards-
incompatible way. These policies can be reviewed here: [1].

This PS requires another PS [2] in devstack's lib/tempest
because Patrole, being a branchless project and hosting the
Patrole devstack plugin itself, must fall back to Tempest's
devstack script to list out the backwards-incompatible
policies in Pike.

A documentation update will also come in a follow up with
information on these feature flags.

[0] I76c4a9b8737bf94f230ab141def652b054120f3b
[1] e.g. http://logs.openstack.org/51/547851/4/check/patrole-member-pike/139c534/job-output.txt.gz#_2018-03-22_21_46_08_392229
[2] I00bdeff9474c54d38b6d6844a041b305bec01ad8

Change-Id: Ia0d9847908a8e723446c16465d68cd7f622c04cc
diff --git a/devstack/plugin.sh b/devstack/plugin.sh
index 10d13f6..d56c963 100644
--- a/devstack/plugin.sh
+++ b/devstack/plugin.sh
@@ -10,16 +10,25 @@
 XTRACE=$(set +o | grep xtrace)
 set -o xtrace
 
-function install_patrole_tempest_plugin() {
-    if is_service_enabled tempest; then
-        setup_package $PATROLE_DIR -e
+function install_patrole_tempest_plugin {
+    setup_package $PATROLE_DIR -e
 
-        if [[ "$RBAC_TEST_ROLE" == "member" ]]; then
-            RBAC_TEST_ROLE="Member"
-        fi
+    if [[ "$RBAC_TEST_ROLE" == "member" ]]; then
+        RBAC_TEST_ROLE="Member"
+    fi
 
-        iniset $TEMPEST_CONFIG patrole enable_rbac True
-        iniset $TEMPEST_CONFIG patrole rbac_test_role $RBAC_TEST_ROLE
+    iniset $TEMPEST_CONFIG patrole enable_rbac True
+    iniset $TEMPEST_CONFIG patrole rbac_test_role $RBAC_TEST_ROLE
+
+    if [[ ${DEVSTACK_SERIES} == 'pike' ]]; then
+        # Policies used by Patrole testing that were changed in a backwards-incompatible way.
+        # TODO(fmontei): Remove these once stable/pike becomes EOL.
+        iniset $TEMPEST_CONFIG policy-feature-enabled create_port_fixed_ips_ip_address_policy False
+        iniset $TEMPEST_CONFIG policy-feature-enabled update_port_fixed_ips_ip_address_policy False
+        iniset $TEMPEST_CONFIG policy-feature-enabled limits_extension_used_limits_policy False
+        iniset $TEMPEST_CONFIG policy-feature-enabled volume_extension_volume_actions_attach_policy False
+        iniset $TEMPEST_CONFIG policy-feature-enabled volume_extension_volume_actions_reserve_policy False
+        iniset $TEMPEST_CONFIG policy-feature-enabled volume_extension_volume_actions_unreserve_policy False
     fi
 }
 
diff --git a/etc/patrole.conf.sample b/etc/patrole.conf.sample
index ed2b07c..5816ea9 100644
--- a/etc/patrole.conf.sample
+++ b/etc/patrole.conf.sample
@@ -100,3 +100,46 @@
 # is logged. This is combined withreport_log_name to generate the full
 # path. (string value)
 #report_log_path = .
+
+
+[policy-feature-enabled]
+
+#
+# From patrole.config
+#
+
+# Is the Neutron policy
+# "create_port:fixed_ips:ip_address" available in the cloud? This
+# policy was
+# changed in a backwards-incompatible way. (boolean value)
+#create_port_fixed_ips_ip_address_policy = true
+
+# Is the Neutron policy
+# "update_port:fixed_ips:ip_address" available in the cloud? This
+# policy was
+# changed in a backwards-incompatible way. (boolean value)
+#update_port_fixed_ips_ip_address_policy = true
+
+# Is the Cinder policy
+# "limits_extension:used_limits" available in the cloud? This policy
+# was
+# changed in a backwards-incompatible way. (boolean value)
+#limits_extension_used_limits_policy = true
+
+# Is the Cinder policy
+# "volume_extension:volume_actions:attach" available in the cloud?
+# This policy
+# was changed in a backwards-incompatible way. (boolean value)
+#volume_extension_volume_actions_attach_policy = true
+
+# Is the Cinder policy
+# "volume_extension:volume_actions:reserve" available in the cloud?
+# This policy
+# was changed in a backwards-incompatible way. (boolean value)
+#volume_extension_volume_actions_reserve_policy = true
+
+# Is the Cinder policy
+# "volume_extension:volume_actions:unreserve" available in the cloud?
+# This policy
+# was changed in a backwards-incompatible way. (boolean value)
+#volume_extension_volume_actions_unreserve_policy = true
diff --git a/patrole_tempest_plugin/config.py b/patrole_tempest_plugin/config.py
index 0077d19..5103888 100644
--- a/patrole_tempest_plugin/config.py
+++ b/patrole_tempest_plugin/config.py
@@ -94,6 +94,7 @@
 patrole_log_group = cfg.OptGroup(
     name='patrole_log', title='Patrole Logging Options')
 
+
 PatroleLogGroup = [
     cfg.BoolOpt('enable_reporting',
                 default=False,
@@ -112,6 +113,47 @@
 ]
 
 
+policy_feature_enabled = cfg.OptGroup(
+    name='policy-feature-enabled',
+    title='Feature Flags for New or Changed Policies')
+
+
+PolicyFeatureEnabledGroup = [
+    # TODO(felipemonteiro): The 6 feature flags below should be removed after
+    # Pike is EOL.
+    cfg.BoolOpt('create_port_fixed_ips_ip_address_policy',
+                default=True,
+                help="""Is the Neutron policy
+"create_port:fixed_ips:ip_address" available in the cloud? This policy was
+changed in a backwards-incompatible way."""),
+    cfg.BoolOpt('update_port_fixed_ips_ip_address_policy',
+                default=True,
+                help="""Is the Neutron policy
+"update_port:fixed_ips:ip_address" available in the cloud? This policy was
+changed in a backwards-incompatible way."""),
+    cfg.BoolOpt('limits_extension_used_limits_policy',
+                default=True,
+                help="""Is the Cinder policy
+"limits_extension:used_limits" available in the cloud? This policy was
+changed in a backwards-incompatible way."""),
+    cfg.BoolOpt('volume_extension_volume_actions_attach_policy',
+                default=True,
+                help="""Is the Cinder policy
+"volume_extension:volume_actions:attach" available in the cloud? This policy
+was changed in a backwards-incompatible way."""),
+    cfg.BoolOpt('volume_extension_volume_actions_reserve_policy',
+                default=True,
+                help="""Is the Cinder policy
+"volume_extension:volume_actions:reserve" available in the cloud? This policy
+was changed in a backwards-incompatible way."""),
+    cfg.BoolOpt('volume_extension_volume_actions_unreserve_policy',
+                default=True,
+                help="""Is the Cinder policy
+"volume_extension:volume_actions:unreserve" available in the cloud? This policy
+was changed in a backwards-incompatible way.""")
+]
+
+
 def list_opts():
     """Return a list of oslo.config options available.
 
@@ -120,7 +162,9 @@
     """
     opt_list = [
         (patrole_group, PatroleGroup),
-        (patrole_log_group, PatroleLogGroup)
+        (patrole_log_group, PatroleLogGroup),
+        (policy_feature_enabled, PolicyFeatureEnabledGroup)
+
     ]
 
     return opt_list
diff --git a/patrole_tempest_plugin/plugin.py b/patrole_tempest_plugin/plugin.py
index a214892..8af6a69 100644
--- a/patrole_tempest_plugin/plugin.py
+++ b/patrole_tempest_plugin/plugin.py
@@ -21,7 +21,7 @@
 from tempest import config
 from tempest.test_discover import plugins
 
-from patrole_tempest_plugin import config as project_config
+from patrole_tempest_plugin import config as pconfig
 
 RBACLOG = logging.getLogger('rbac_reporting')
 
@@ -64,16 +64,23 @@
     def register_opts(self, conf):
         config.register_opt_group(
             conf,
-            project_config.patrole_group,
-            project_config.PatroleGroup)
+            pconfig.patrole_group,
+            pconfig.PatroleGroup)
         config.register_opt_group(
             conf,
-            project_config.patrole_log_group,
-            project_config.PatroleLogGroup)
+            pconfig.patrole_log_group,
+            pconfig.PatroleLogGroup)
+        config.register_opt_group(
+            conf,
+            pconfig.policy_feature_enabled,
+            pconfig.PolicyFeatureEnabledGroup)
 
         if conf.patrole_log.enable_reporting:
             self._configure_per_test_logging(conf)
 
     def get_opt_lists(self):
-        return [(project_config.patrole_group.name,
-                 project_config.PatroleGroup)]
+        return [
+            (pconfig.patrole_group.name, pconfig.PatroleGroup),
+            (pconfig.policy_feature_enabled.name,
+                pconfig.PolicyFeatureEnabledGroup)
+        ]
diff --git a/patrole_tempest_plugin/tests/api/network/test_ports_rbac.py b/patrole_tempest_plugin/tests/api/network/test_ports_rbac.py
index 0b91e14..a8c7d68 100644
--- a/patrole_tempest_plugin/tests/api/network/test_ports_rbac.py
+++ b/patrole_tempest_plugin/tests/api/network/test_ports_rbac.py
@@ -15,6 +15,7 @@
 #
 
 import netaddr
+import testtools
 
 from tempest.common import utils
 from tempest.common.utils import net_utils
@@ -106,6 +107,9 @@
         with self.rbac_utils.override_role(self):
             self.create_port(**post_body)
 
+    @testtools.skipUnless(
+        CONF.policy_feature_enabled.create_port_fixed_ips_ip_address_policy,
+        '"create_port:fixed_ips:ip_address" must be available in the cloud.')
     @rbac_rule_validation.action(service="neutron",
                                  rule="create_port:fixed_ips:ip_address")
     @decorators.idempotent_id('2551e10d-006a-413c-925a-8c6f834c09ac')
@@ -268,6 +272,9 @@
         self.addCleanup(self.ports_client.update_port, self.port['id'],
                         mac_address=original_mac_address)
 
+    @testtools.skipUnless(
+        CONF.policy_feature_enabled.update_port_fixed_ips_ip_address_policy,
+        '"update_port:fixed_ips:ip_address" must be available in the cloud.')
     @rbac_rule_validation.action(service="neutron",
                                  rule="update_port:fixed_ips:ip_address")
     @decorators.idempotent_id('c091c825-532b-4c6f-a14f-affd3259c1c3')
diff --git a/patrole_tempest_plugin/tests/api/volume/test_limits_rbac.py b/patrole_tempest_plugin/tests/api/volume/test_limits_rbac.py
index 78cfd9a..aec5cb1 100644
--- a/patrole_tempest_plugin/tests/api/volume/test_limits_rbac.py
+++ b/patrole_tempest_plugin/tests/api/volume/test_limits_rbac.py
@@ -13,16 +13,24 @@
 #    License for the specific language governing permissions and limitations
 #    under the License.
 
+import testtools
+
+from tempest import config
 from tempest.lib import decorators
 
 from patrole_tempest_plugin import rbac_exceptions
 from patrole_tempest_plugin import rbac_rule_validation
 from patrole_tempest_plugin.tests.api.volume import rbac_base
 
+CONF = config.CONF
+
 
 class LimitsV3RbacTest(rbac_base.BaseVolumeRbacTest):
     _api_version = 3
 
+    @testtools.skipUnless(
+        CONF.policy_feature_enabled.limits_extension_used_limits_policy,
+        '"limits_extension:used_limits" must be available in the cloud.')
     @decorators.idempotent_id('dab04510-5b86-4479-a633-6e496ff405af')
     @rbac_rule_validation.action(service="cinder",
                                  rule="limits_extension:used_limits")
diff --git a/patrole_tempest_plugin/tests/api/volume/test_volume_actions_rbac.py b/patrole_tempest_plugin/tests/api/volume/test_volume_actions_rbac.py
index a755d48..46f7a3e 100644
--- a/patrole_tempest_plugin/tests/api/volume/test_volume_actions_rbac.py
+++ b/patrole_tempest_plugin/tests/api/volume/test_volume_actions_rbac.py
@@ -13,6 +13,8 @@
 #    License for the specific language governing permissions and limitations
 #    under the License.
 
+import testtools
+
 from tempest.common import compute
 from tempest.common import utils
 from tempest.common import waiters
@@ -65,6 +67,11 @@
         waiters.wait_for_volume_resource_status(
             self.volumes_client, volume_id, 'available')
 
+    @testtools.skipUnless(
+        CONF.policy_feature_enabled
+        .volume_extension_volume_actions_attach_policy,
+        '"volume_extension:volume_actions:attach" must be available in the '
+        'cloud.')
     @utils.services('compute')
     @rbac_rule_validation.action(
         service="cinder",
@@ -151,6 +158,11 @@
             self.volumes_client.set_bootable_volume(self.volume['id'],
                                                     bootable=True)
 
+    @testtools.skipUnless(
+        CONF.policy_feature_enabled
+        .volume_extension_volume_actions_reserve_policy,
+        '"volume_extension:volume_actions:reserve" must be available in the '
+        'cloud.')
     @decorators.idempotent_id('41566922-75a1-4484-99c7-9c8782ee99ac')
     @rbac_rule_validation.action(
         service="cinder",
@@ -159,6 +171,11 @@
         with self.rbac_utils.override_role(self):
             self.volumes_client.reserve_volume(self.volume['id'])
 
+    @testtools.skipUnless(
+        CONF.policy_feature_enabled
+        .volume_extension_volume_actions_unreserve_policy,
+        '"volume_extension:volume_actions:unreserve" must be available in the '
+        'cloud.')
     @decorators.idempotent_id('e5fa9564-77d9-4e57-b0c0-3e0ae4d08535')
     @rbac_rule_validation.action(
         service="cinder",
diff --git a/releasenotes/notes/backwards-incompatible-rule-feature-flag-ebe8b44c0aa663a8.yaml b/releasenotes/notes/backwards-incompatible-rule-feature-flag-ebe8b44c0aa663a8.yaml
new file mode 100644
index 0000000..1ee182e
--- /dev/null
+++ b/releasenotes/notes/backwards-incompatible-rule-feature-flag-ebe8b44c0aa663a8.yaml
@@ -0,0 +1,16 @@
+---
+features:
+  - |
+    A new configuration group ``[policy_feature_enabled]`` has been added to
+    Patrole which will be responsible for collecting the feature flags to be
+    used for newly introduced policies or policies that were changed in a
+    backwards-incompatible way.
+
+    * create_port_fixed_ips_ip_address_policy (Neutron)
+    * update_port_fixed_ips_ip_address_policy (Neutron)
+    * limits_extension_used_limits_policy (Cinder)
+    * volume_extension_volume_actions_attach_policy (Cinder)
+    * volume_extension_volume_actions_reserve_policy (Cinder)
+    * volume_extension_volume_actions_unreserve_policy (Cinder)
+
+    These feature flags will be supported until Pike release cycle is EOL.