| David Kranz | 6308ec2 | 2012-02-22 09:36:48 -0500 | [diff] [blame] | 1 | Quanta Research Cambridge OpenStack Stress Test System
|
| 2 | ======================================================
|
| 3 |
|
| 4 | Nova is a distributed, asynchronous system that is prone to race condition
|
| 5 | bugs. These bugs will not be easily found during
|
| David Kranz | d10601c | 2012-03-15 15:58:28 -0400 | [diff] [blame] | 6 | functional testing but will be encountered by users in large deployments in a
|
| David Kranz | 6308ec2 | 2012-02-22 09:36:48 -0500 | [diff] [blame] | 7 | way that is hard to debug. The stress test tries to cause these bugs to happen
|
| 8 | in a more controlled environment.
|
| 9 |
|
| David Kranz | d10601c | 2012-03-15 15:58:28 -0400 | [diff] [blame] | 10 | The basic idea of the test is that there are a number of actions, roughly
|
| David Kranz | 6308ec2 | 2012-02-22 09:36:48 -0500 | [diff] [blame] | 11 | corresponding to the Compute API, that are fired pseudo-randomly at a nova
|
| 12 | cluster as fast as possible. These actions consist of what to do, how to
|
| 13 | verify success, and a state filter to make sure that the operation makes sense.
|
| 14 | For example, if the action is to reboot a server and none are active, nothing
|
| 15 | should be done. A test case is a set of actions to be performed and the
|
| David Kranz | d10601c | 2012-03-15 15:58:28 -0400 | [diff] [blame] | 16 | probability that each action should be selected. There are also parameters
|
| David Kranz | 6308ec2 | 2012-02-22 09:36:48 -0500 | [diff] [blame] | 17 | controlling rate of fire and stuff like that.
|
| 18 |
|
| 19 | This test framework is designed to stress test a Nova cluster. Hence,
|
| 20 | you must have a working Nova cluster.
|
| 21 |
|
| 22 | Environment
|
| 23 | ------------
|
| 24 | This particular framework assumes your working Nova cluster understands Nova
|
| 25 | API 2.0. The stress tests can read the logs from the cluster. To enable this
|
| David Kranz | 30fe84a | 2012-03-20 16:25:47 -0400 | [diff] [blame^] | 26 | you have to provide the hostname to call 'nova-manage' and
|
| 27 | the private key and user name for ssh to the cluster in the
|
| David Kranz | 6308ec2 | 2012-02-22 09:36:48 -0500 | [diff] [blame] | 28 | [stress] section of tempest.conf. You also need to provide the
|
| 29 | value of --logdir in nova.conf:
|
| 30 |
|
| 31 | host_private_key_path=<path to private ssh key>
|
| 32 | host_admin_user=<name of user for ssh command>
|
| 33 | nova_logdir=<value of --logdir in nova.conf>
|
| David Kranz | 30fe84a | 2012-03-20 16:25:47 -0400 | [diff] [blame^] | 34 | controller=<hostname for calling nova-manage>
|
| David Kranz | 6308ec2 | 2012-02-22 09:36:48 -0500 | [diff] [blame] | 35 |
|
| 36 | The stress test needs the top-level tempest directory to be on PYTHONPATH
|
| 37 | if you are not using nosetests to run.
|
| 38 |
|
| David Kranz | d10601c | 2012-03-15 15:58:28 -0400 | [diff] [blame] | 39 | For real stress, you need to remove "ratelimit" from the pipeline in
|
| David Kranz | 6308ec2 | 2012-02-22 09:36:48 -0500 | [diff] [blame] | 40 | api-paste.ini.
|
| 41 |
|
| 42 |
|
| 43 | Running the sample test
|
| 44 | -----------------------
|
| 45 |
|
| 46 | To test your installation, do the following (from the tempest directory):
|
| 47 |
|
| 48 | PYTHONPATH=. python stress/tests/user_script_sample.py
|
| 49 |
|
| 50 | This sample test tries to create a few VMs and kill a few VMs.
|
| 51 |
|
| 52 |
|
| 53 | Additional Tools
|
| 54 | ----------------
|
| 55 |
|
| 56 | Sometimes the tests don't finish, or there are failures. In these
|
| 57 | cases, you may want to clean out the nova cluster. We have provided
|
| 58 | some scripts to do this in the ``tools`` subdirectory. To use these
|
| 59 | tools, you will need to install python-novaclient.
|
| 60 | You can then use the following script to destroy any keypairs,
|
| 61 | floating ips, and servers::
|
| 62 |
|
| 63 | stress/tools/nova_destroy_all.py
|