| Masayuki Igawa | c268dd6 | 2013-09-18 00:51:05 +0900 | [diff] [blame] | 1 | Tempest Field Guide to API tests |
| 2 | ================================ |
| Sean Dague | 1937d09 | 2013-05-17 16:36:38 -0400 | [diff] [blame] | 3 | |
| 4 | |
| 5 | What are these tests? |
| Attila Fazekas | 23fdf1d | 2013-06-09 16:35:23 +0200 | [diff] [blame] | 6 | --------------------- |
| Sean Dague | 1937d09 | 2013-05-17 16:36:38 -0400 | [diff] [blame] | 7 | |
| 8 | One of Tempest's prime function is to ensure that your OpenStack cloud |
| 9 | works with the OpenStack API as documented. The current largest |
| 10 | portion of Tempest code is devoted to test cases that do exactly this. |
| 11 | |
| Yong Sheng Gong | e99072d | 2013-08-15 13:52:41 +0800 | [diff] [blame] | 12 | It's also important to test not only the expected positive path on |
| Sean Dague | 1937d09 | 2013-05-17 16:36:38 -0400 | [diff] [blame] | 13 | APIs, but also to provide them with invalid data to ensure they fail |
| 14 | in expected and documented ways. Over the course of the OpenStack |
| 15 | project Tempest has discovered many fundamental bugs by doing just |
| 16 | this. |
| 17 | |
| Yong Sheng Gong | e99072d | 2013-08-15 13:52:41 +0800 | [diff] [blame] | 18 | In order for some APIs to return meaningful results, there must be |
| Sean Dague | 1937d09 | 2013-05-17 16:36:38 -0400 | [diff] [blame] | 19 | enough data in the system. This means these tests might start by |
| Yong Sheng Gong | e99072d | 2013-08-15 13:52:41 +0800 | [diff] [blame] | 20 | spinning up a server, image, etc, then operating on it. |
| Sean Dague | 1937d09 | 2013-05-17 16:36:38 -0400 | [diff] [blame] | 21 | |
| 22 | |
| 23 | Why are these tests in tempest? |
| Attila Fazekas | 23fdf1d | 2013-06-09 16:35:23 +0200 | [diff] [blame] | 24 | ------------------------------- |
| Sean Dague | 1937d09 | 2013-05-17 16:36:38 -0400 | [diff] [blame] | 25 | |
| 26 | This is one of the core missions for the Tempest project, and where it |
| 27 | started. Many people use this bit of function in Tempest to ensure |
| 28 | their clouds haven't broken the OpenStack API. |
| 29 | |
| 30 | It could be argued that some of the negative testing could be done |
| 31 | back in the projects themselves, and we might evolve there over time, |
| 32 | but currently in the OpenStack gate this is a fundamentally important |
| 33 | place to keep things. |
| 34 | |
| 35 | |
| 36 | Scope of these tests |
| Attila Fazekas | 23fdf1d | 2013-06-09 16:35:23 +0200 | [diff] [blame] | 37 | -------------------- |
| Sean Dague | 1937d09 | 2013-05-17 16:36:38 -0400 | [diff] [blame] | 38 | |
| 39 | API tests should always use the Tempest implementation of the |
| 40 | OpenStack API, as we want to ensure that bugs aren't hidden by the |
| 41 | official clients. |
| 42 | |
| 43 | They should test specific API calls, and can build up complex state if |
| 44 | it's needed for the API call to be meaningful. |
| 45 | |
| 46 | They should send not only good data, but bad data at the API and look |
| 47 | for error codes. |
| 48 | |
| 49 | They should all be able to be run on their own, not depending on the |
| 50 | state created by a previous test. |