Convert tempest-scenario-all job to tempest-slow
As per discussion on ML[1], we do not have any job which runs only
slow tests.
This commit provide a separate job which run only slow tests which can
be API or scenario tests. Further this job can be used cross projects
like nova, cinder, neutron.
Why we need separate job for slow tests:
- slow tests create gate issues on job timeout which we observed\
during Rocky cycle[2]. Not to effect gate let's start them executing
on separate job.
Why to convert the tempest-scenario-all job
- tempest-scenario-all job was created to run the slow tests along with
other scenario tests. But slow tests can be API tests[3] and mixing them
with other scenario or API tests is not good idea.
- All scenario tests excluding slow tests are run as part of tempest-full
job so let's not run scenario tests twice on gate.
[1] http://lists.openstack.org/pipermail/openstack-dev/2018-July/132527.html
[2] https://bugs.launchpad.net/tempest/+bug/1783405
[3] https://ethercalc.openstack.org/dorupfz6s9qt
Change-Id: I5b1d3041c0140bea11705ad287f1472e74aacb5c
diff --git a/.zuul.yaml b/.zuul.yaml
index 1cf67b0..c794d26 100644
--- a/.zuul.yaml
+++ b/.zuul.yaml
@@ -143,25 +143,24 @@
voting: false
- job:
- name: tempest-scenario-all
+ name: tempest-slow
parent: tempest-multinode-full
branches:
- master
description: |
- This multinode integration job will run all scenario tests including slow
- tests with lvm multibackend setup. This job will not run any API tests.
+ This multinode integration job will run all the tests tagged as slow.
+ It enables the lvm multibackend setup to cover few scenario tests.
+ This job will run only slow tests(API or Scenario) serially.
Former names for this job were:
* legacy-tempest-dsvm-neutron-scenario-multinode-lvm-multibackend
* tempest-scenario-multinode-lvm-multibackend
timeout: 10800
vars:
- # 'all' is used for applying the custom regex below.
- tox_envlist: all
+ tox_envlist: slow-serial
devstack_localrc:
CINDER_ENABLED_BACKENDS: lvm:lvmdriver-1,lvm:lvmdriver-2
tempest_concurrency: 2
- tempest_test_regex: (^tempest\.(scenario))
- job:
name: tempest-full-queens
@@ -353,7 +352,7 @@
- ^tempest/hacking/.*$
- ^tempest/tests/.*$
- tempest-tox-plugin-sanity-check
- - tempest-scenario-all:
+ - tempest-slow:
irrelevant-files:
- ^(test-|)requirements.txt$
- ^.*\.rst$
@@ -363,7 +362,6 @@
- ^setup.cfg$
- ^tempest/hacking/.*$
- ^tempest/tests/.*$
- - ^tempest/api/.*$
- nova-cells-v1:
irrelevant-files:
- ^(test-|)requirements.txt$
@@ -397,7 +395,7 @@
- ^setup.cfg$
- ^tempest/hacking/.*$
- ^tempest/tests/.*$
- - tempest-scenario-all:
+ - tempest-slow:
irrelevant-files:
- ^(test-|)requirements.txt$
- ^.*\.rst$
@@ -407,7 +405,6 @@
- ^setup.cfg$
- ^tempest/hacking/.*$
- ^tempest/tests/.*$
- - ^tempest/api/.*$
experimental:
jobs:
- tempest-cinder-v2-api:
diff --git a/tox.ini b/tox.ini
index de4f1b7..befa991 100644
--- a/tox.ini
+++ b/tox.ini
@@ -135,6 +135,16 @@
find . -type f -name "*.pyc" -delete
tempest run --serial --regex '\[.*\bsmoke\b.*\]' {posargs}
+[testenv:slow-serial]
+envdir = .tox/tempest
+sitepackages = {[tempestenv]sitepackages}
+setenv = {[tempestenv]setenv}
+deps = {[tempestenv]deps}
+# The regex below is used to select the slow tagged tests to run serially:
+commands =
+ find . -type f -name "*.pyc" -delete
+ tempest run --serial --regex '\[.*\bslow\b.*\]' {posargs}
+
[testenv:venv]
deps =
-c{env:UPPER_CONSTRAINTS_FILE:https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt}