| Attila Fazekas | 58d2330 | 2013-07-24 10:25:02 +0200 | [diff] [blame] | 1 | ============================ |
| Sean Dague | 7679b48 | 2013-05-24 08:49:44 -0400 | [diff] [blame] | 2 | Tempest Field Guide Overview |
| Attila Fazekas | 58d2330 | 2013-07-24 10:25:02 +0200 | [diff] [blame] | 3 | ============================ |
| Sean Dague | 31a57ad | 2013-05-07 15:24:05 -0400 | [diff] [blame] | 4 | |
| 5 | Tempest is designed to be useful for a large number of different |
| 6 | environments. This includes being useful for gating commits to |
| 7 | OpenStack core projects, being used to validate OpenStack cloud |
| 8 | implementations for both correctness, as well as a burn in tool for |
| 9 | OpenStack clouds. |
| 10 | |
| 11 | As such Tempest tests come in many flavors, each with their own rules |
| 12 | and guidelines. Below is the proposed Havana restructuring for Tempest |
| 13 | to make this clear. |
| 14 | |
| Xiao Hanyu | a4036d1 | 2013-06-19 14:12:23 +0800 | [diff] [blame] | 15 | | tempest/ |
| 16 | | api/ - API tests |
| 17 | | cli/ - CLI tests |
| 18 | | scenario/ - complex scenario tests |
| 19 | | stress/ - stress tests |
| 20 | | thirdparty/ - 3rd party api tests |
| Sean Dague | 31a57ad | 2013-05-07 15:24:05 -0400 | [diff] [blame] | 21 | |
| 22 | Each of these directories contains different types of tests. What |
| 23 | belongs in each directory, the rules and examples for good tests, are |
| 24 | documented in a README.rst file in the directory. |
| 25 | |
| 26 | |
| Sean Dague | 31a57ad | 2013-05-07 15:24:05 -0400 | [diff] [blame] | 27 | api |
| Attila Fazekas | 58d2330 | 2013-07-24 10:25:02 +0200 | [diff] [blame] | 28 | --- |
| Sean Dague | 31a57ad | 2013-05-07 15:24:05 -0400 | [diff] [blame] | 29 | |
| 30 | API tests are validation tests for the OpenStack API. They should not |
| 31 | use the existing python clients for OpenStack, but should instead use |
| 32 | the tempest implementations of clients. This allows us to test both |
| 33 | XML and JSON. Having raw clients also lets us pass invalid JSON and |
| 34 | XML to the APIs and see the results, something we could not get with |
| 35 | the native clients. |
| 36 | |
| 37 | When it makes sense, API testing should be moved closer to the |
| 38 | projects themselves, possibly as functional tests in their unit test |
| 39 | frameworks. |
| 40 | |
| Sean Dague | 31a57ad | 2013-05-07 15:24:05 -0400 | [diff] [blame] | 41 | |
| 42 | cli |
| Attila Fazekas | 58d2330 | 2013-07-24 10:25:02 +0200 | [diff] [blame] | 43 | --- |
| Sean Dague | 31a57ad | 2013-05-07 15:24:05 -0400 | [diff] [blame] | 44 | |
| 45 | CLI tests use the openstack CLI to interact with the OpenStack |
| 46 | cloud. CLI testing in unit tests is somewhat difficult because unlike |
| 47 | server testing, there is no access to server code to |
| 48 | instantiate. Tempest seems like a logical place for this, as it |
| 49 | prereqs having a running OpenStack cloud. |
| 50 | |
| Sean Dague | 31a57ad | 2013-05-07 15:24:05 -0400 | [diff] [blame] | 51 | |
| 52 | scenario |
| Attila Fazekas | 58d2330 | 2013-07-24 10:25:02 +0200 | [diff] [blame] | 53 | -------- |
| Sean Dague | 31a57ad | 2013-05-07 15:24:05 -0400 | [diff] [blame] | 54 | |
| 55 | Scenario tests are complex "through path" tests for OpenStack |
| 56 | functionality. They are typically a series of steps where complicated |
| 57 | state requiring multiple services is set up exercised, and torn down. |
| 58 | |
| 59 | Scenario tests can and should use the OpenStack python clients. |
| 60 | |
| Sean Dague | 31a57ad | 2013-05-07 15:24:05 -0400 | [diff] [blame] | 61 | |
| 62 | stress |
| Attila Fazekas | 58d2330 | 2013-07-24 10:25:02 +0200 | [diff] [blame] | 63 | ------ |
| Sean Dague | 31a57ad | 2013-05-07 15:24:05 -0400 | [diff] [blame] | 64 | |
| Marc Koderer | a58e851 | 2014-07-17 10:40:23 +0200 | [diff] [blame] | 65 | Stress tests are designed to stress an OpenStack environment by running a high |
| 66 | workload against it and seeing what breaks. The stress test framework runs |
| 67 | several test jobs in parallel and can run any existing test in Tempest as a |
| 68 | stress job. |
| Sean Dague | 31a57ad | 2013-05-07 15:24:05 -0400 | [diff] [blame] | 69 | |
| Sean Dague | 09761f6 | 2013-05-13 15:20:40 -0400 | [diff] [blame] | 70 | thirdparty |
| Attila Fazekas | 58d2330 | 2013-07-24 10:25:02 +0200 | [diff] [blame] | 71 | ---------- |
| Sean Dague | 09761f6 | 2013-05-13 15:20:40 -0400 | [diff] [blame] | 72 | |
| 73 | Many openstack components include 3rdparty API support. It is |
| ravikumar-venkatesan | bae88e1 | 2013-06-07 23:02:19 +0000 | [diff] [blame] | 74 | completely legitimate for Tempest to include tests of 3rdparty APIs, |
| 75 | but those should be kept separate from the normal OpenStack |
| Sean Dague | 09761f6 | 2013-05-13 15:20:40 -0400 | [diff] [blame] | 76 | validation. |