| Matthew Treinish | a9aef87 | 2014-07-25 16:03:08 -0400 | [diff] [blame] | 1 | .. _api_field_guide: |
| 2 | |
| Masayuki Igawa | c268dd6 | 2013-09-18 00:51:05 +0900 | [diff] [blame] | 3 | Tempest Field Guide to API tests |
| 4 | ================================ |
| Sean Dague | 1937d09 | 2013-05-17 16:36:38 -0400 | [diff] [blame] | 5 | |
| 6 | |
| 7 | What are these tests? |
| Attila Fazekas | 23fdf1d | 2013-06-09 16:35:23 +0200 | [diff] [blame] | 8 | --------------------- |
| Sean Dague | 1937d09 | 2013-05-17 16:36:38 -0400 | [diff] [blame] | 9 | |
| 10 | One of Tempest's prime function is to ensure that your OpenStack cloud |
| 11 | works with the OpenStack API as documented. The current largest |
| 12 | portion of Tempest code is devoted to test cases that do exactly this. |
| 13 | |
| Yong Sheng Gong | e99072d | 2013-08-15 13:52:41 +0800 | [diff] [blame] | 14 | It's also important to test not only the expected positive path on |
| Sean Dague | 1937d09 | 2013-05-17 16:36:38 -0400 | [diff] [blame] | 15 | APIs, but also to provide them with invalid data to ensure they fail |
| Jordan Pittier | 74a56ab | 2017-04-26 16:46:20 +0200 | [diff] [blame] | 16 | in expected and documented ways. The latter type of tests is called |
| 17 | ``negative tests`` in Tempest source code. Over the course of the OpenStack |
| Sean Dague | 1937d09 | 2013-05-17 16:36:38 -0400 | [diff] [blame] | 18 | project Tempest has discovered many fundamental bugs by doing just |
| 19 | this. |
| 20 | |
| Yong Sheng Gong | e99072d | 2013-08-15 13:52:41 +0800 | [diff] [blame] | 21 | In order for some APIs to return meaningful results, there must be |
| Sean Dague | 1937d09 | 2013-05-17 16:36:38 -0400 | [diff] [blame] | 22 | enough data in the system. This means these tests might start by |
| Yong Sheng Gong | e99072d | 2013-08-15 13:52:41 +0800 | [diff] [blame] | 23 | spinning up a server, image, etc, then operating on it. |
| Sean Dague | 1937d09 | 2013-05-17 16:36:38 -0400 | [diff] [blame] | 24 | |
| 25 | |
| Jordan Pittier | 74a56ab | 2017-04-26 16:46:20 +0200 | [diff] [blame] | 26 | Why are these tests in Tempest? |
| Attila Fazekas | 23fdf1d | 2013-06-09 16:35:23 +0200 | [diff] [blame] | 27 | ------------------------------- |
| Sean Dague | 1937d09 | 2013-05-17 16:36:38 -0400 | [diff] [blame] | 28 | |
| 29 | This is one of the core missions for the Tempest project, and where it |
| 30 | started. Many people use this bit of function in Tempest to ensure |
| 31 | their clouds haven't broken the OpenStack API. |
| 32 | |
| 33 | It could be argued that some of the negative testing could be done |
| 34 | back in the projects themselves, and we might evolve there over time, |
| 35 | but currently in the OpenStack gate this is a fundamentally important |
| 36 | place to keep things. |
| 37 | |
| 38 | |
| 39 | Scope of these tests |
| Attila Fazekas | 23fdf1d | 2013-06-09 16:35:23 +0200 | [diff] [blame] | 40 | -------------------- |
| Sean Dague | 1937d09 | 2013-05-17 16:36:38 -0400 | [diff] [blame] | 41 | |
| 42 | API tests should always use the Tempest implementation of the |
| 43 | OpenStack API, as we want to ensure that bugs aren't hidden by the |
| 44 | official clients. |
| 45 | |
| 46 | They should test specific API calls, and can build up complex state if |
| 47 | it's needed for the API call to be meaningful. |
| 48 | |
| 49 | They should send not only good data, but bad data at the API and look |
| 50 | for error codes. |
| 51 | |
| 52 | They should all be able to be run on their own, not depending on the |
| 53 | state created by a previous test. |