Implemented HW2HW network performance testing
Implemented HW2HW network performance test:
- Added K8S client manager and API methods
- Added method for collecting HW compute nodes
- Improved and extended global config
- Extended requirements.txt, pinned some modules
- Extended and improved SSH methods
Added some small improvements:
- extended .gitignore
- Added some custom exceptions instead of basic ones
- Renamed some classes
- Set iperf v2 to be default for multi-threads tests
- Updated README file
Related-PROD: PROD-36943
Change-Id: I265058967ccc01d96bf3bca532a8a0ae2a26f1f2
diff --git a/.gitignore b/.gitignore
index 7fb7fcd..48e10d7 100644
--- a/.gitignore
+++ b/.gitignore
@@ -144,3 +144,4 @@
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
.idea/
+.DS_Store
diff --git a/README.md b/README.md
index bfc8fde..072e0ee 100644
--- a/README.md
+++ b/README.md
@@ -15,13 +15,11 @@
Configuration
--
- Open _global_config.yaml_ file to override the settings, or export the
- environment variables.
+ Open _global_config.yaml_ file to override the settings.
Settings
--
- The following options can be set in _global_config.yaml_ file, or by exporting
- the environment variables.
+ The following options can be set in _global_config.yaml_ file.
* **test_glance** allows next overrides:
@@ -29,30 +27,45 @@
| --- | --- | --- |
| IMAGE_SIZE_MB | 9000 | Specific image size (in MB) to upload/download at Glance |
-* **test_vm2vm** allows next overrides:
+* **test_vm2vm and test_vm2vm_different_routers** allow next overrides:
-| Environment Variable | Default | Description |
-| --- | --- | --- |
-| flavor_name | spt-test | Flavor name |
-| flavor_ram | 1536 | To define RAM allocation for specific flavor, MB |
-| flavor_vcpus | 1 | To define a count of vCPU for flavor |
-| flavor_disk | 5 | To define a count of disks on flavor, GB |
-| image_name | cvp.ubuntu.2004 | Cloud Ubuntu image to create VMs. Use 20.04 image in case internet_at_vms=false since the offline packages are set for Ubuntu 20.04. You can use any other Ubuntu image in case internet_at_vms=true. |
-| CMP_HOSTS | "" | Pair of compute hosts to create VMs at different hosts. By default, some random pair from nova compute list will be selected. To set some pair, set _CMP_HOSTS: ["cmp001", "cmp002"]_ in _global_config.yaml_ file, or export CMP_HOSTS="cmp001,cmp002". |
-| skipped_nodes | "" | Skip some compute hosts, so they are not selected at CMP_HOSTS pair. To set some nodes to skip, set _skipped_nodes: ["cmp003"]_ in _global_config.yaml_ file, or export skipped_nodes="cmp003".|
-| nova_timeout | 300 | Timeout to VM to be ACTIVE, seconds. |
-| external_network | public | External network name to allocate the Floating IPs |
-| custom_mtu | default | The MTU to set at the VMs. If "default" is set, the MTU will be set automatically from the newly created SPT internal networks. The default value in the networks comes from the Neutron configuration. In case you want to test the bandwidth with some specific custom MTU, set the value like 8950. |
-| ssh_timeout | 500 | Timeout to VM to be reachable via SSH, seconds. |
-| iperf_prep_string | "sudo /bin/bash -c 'echo \"91.189.88.161 archive.ubuntu.com\" >> /etc/hosts'" | Preparation string to set ubuntu repository host in /etc/hosts of VMs |
-| internet_at_vms | 'true' | In case True, the Internet is present at VMs, and the tests are able to install iperf3 by _apt update; apt install iperf3_. In case VMs have no Internet, set 'false' and the iperf3 will be installed from offline *.deb packages. |
-| iperf_deb_package_dir_path | /opt/packages/ | Path to the local directory where the iperf3 *.deb packages are present. In the toolset offline images they are located at /opt/packages. Or you can download iperf3 deb package and its dependencies and put them at some custom folder. |
-| iperf_time | 60 | iperf3 -t option value: time in seconds to transmit for (iperf -t option). |
-| multiple_threads_number | 10 | Number of iperf/iperf3 parallel client threads to run (iperf3/iperf -P option value) |
-| multiple_threads_iperf_utility | 'iperf3' | The tool for bandwidth measurements. Options to set: 'iperf' to use v2 and 'iperf3' to use v3. Eventually, this is the name of the utility that is installed at Ubuntu VMs from apt. |
+| Environment Variable | Default | Description |
+|--------------------------------|--------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| flavor_name | spt-test | Flavor name |
+| flavor_ram | 1536 | To define RAM allocation for specific flavor, MB |
+| flavor_vcpus | 1 | To define a count of vCPU for flavor |
+| flavor_disk | 5 | To define a count of disks on flavor, GB |
+| image_name | cvp.ubuntu.2004 | Cloud Ubuntu image to create VMs. Use 20.04 image in case internet_at_vms=false since the offline packages are set for Ubuntu 20.04. You can use any other Ubuntu image in case internet_at_vms=true. |
+| CMP_HOSTS | [] | Pair of compute hosts to create VMs at different hosts. By default, some random pair from nova compute list will be selected. To set some pair, set _CMP_HOSTS: ["cmp001", "cmp002"]_ in _global_config.yaml_ file. |
+| skipped_nodes | [] | Skip some compute hosts, so they are not selected at CMP_HOSTS pair. To set some nodes to skip, set _skipped_nodes: ["cmp003"]_ in _global_config.yaml_ file. Applies the hw2hw test as well. |
+| nova_timeout | 300 | Timeout to VM to be ACTIVE, seconds. |
+| external_network | public | External network name to allocate the Floating IPs |
+| custom_mtu | default | The MTU to set at the VMs. If "default" is set, the MTU will be set automatically from the newly created SPT internal networks. The default value in the networks comes from the Neutron configuration. In case you want to test the bandwidth with some specific custom MTU, set the value like 8950. |
+| ssh_timeout | 500 | Timeout to VM to be reachable via SSH, seconds. |
+| iperf_prep_string | "sudo /bin/bash -c 'echo \"91.189.88.161 archive.ubuntu.com\" >> /etc/hosts'" | Preparation string to set ubuntu repository host in /etc/hosts of VMs. |
+| internet_at_vms | 'true' | In case True, the Internet is present at VMs, and the tests are able to install iperf3 by _apt update; apt install iperf3_. In case VMs have no Internet, set 'false' and the iperf3 will be installed from offline *.deb packages. |
+| iperf_deb_package_dir_path | /opt/packages/ | Path to the local directory where the iperf3 *.deb packages are present. In the toolset offline images they are located at /opt/packages. Or you can download iperf3 deb package and its dependencies and put them at some custom folder. |
+| iperf_time | 60 | iperf3 -t option value: time in seconds to transmit for (iperf -t option). Applies the hw2hw test as well. |
+| multiple_threads_number | 10 | Number of iperf/iperf3 parallel client threads to run (iperf3/iperf -P option value). Applies the hw2hw test as well. |
+| multiple_threads_iperf_utility | 'iperf' | The tool for bandwidth measurements. Options to set: 'iperf' to use v2 and 'iperf3' to use v3. Eventually, this is the name of the utility that is installed at Ubuntu VMs from apt. Applies the hw2hw test as well. |
In case _internet_at_vms=false_, please make sure that _iperf_deb_package_dir_path_ is set correctly and has iperf3 deb package and its dependencies.
+* **test_hw2hw** allows next overrides:
+
+| Environment Variable | Default | Description |
+|--------------------------------|------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| skipped_nodes | [] | Skip some compute hosts, so they are not selected at HW2HW pair. To set some nodes to skip, set their names from "kubectl get nodes", for example, _skipped_nodes: ["cmp003", "some_node_name"]_ in _global_config.yaml_ file. Applies to the VM2VM tests as well. |
+| iperf_time | 60 | iperf3 -t option value: time in seconds to transmit for (iperf -t option). Applies to the VM2VM tests as well. |
+| multiple_threads_number | 10 | Number of iperf/iperf3 parallel client threads to run (iperf3/iperf -P option value). Applies to the VM2VM tests as well. |
+| multiple_threads_iperf_utility | 'iperf' | The tool for bandwidth measurements. Options to set: 'iperf' to use v2 and 'iperf3' to use v3. Eventually, this is the name of the utility that is installed at Ubuntu VMs from apt. Applies to the VM2VM tests as well. |
+| hw_nodes_list | [] | List of the compute hosts to measure the performance between them. By default, some random pairs from "kubectl get nodes -l openvswitch=enabled,openstack-compute-node=enabled" list will be selected. To set some pair, set its names from the k8s nodes list, e.g. _hw_nodes_list: ["cmp001", "cmp002"]_ in _global_config.yaml_ file. |
+| mos_kubeconfig_path | "" | Path to the MOSK K8S config file. In case executing inside toolset pod, copy the config inside the pod and set its path. |
+| node_ssh_key_path | "" | Path to the private SSH key to log in to the K8S nodes. In case executing inside toolset pod, copy the key file inside the pod and set its path. |
+| node_ssh_username | "mcc-user" | Username to be used to connect to the K8S nodes via SSH using the ssh key. |
+| network_cidr | "" | Network CIRD to be used for running iperf between the nodes, e.g. find the CIDR of the br-tenant or the storage network. Example: "10.23.195.0/25". |
+
+
Executing tests
--
Run tests:
diff --git a/fixtures/base.py b/fixtures/base.py
index fb4fb43..9774e5b 100644
--- a/fixtures/base.py
+++ b/fixtures/base.py
@@ -6,11 +6,27 @@
import logging
from utils import helpers
+from utils.k8s_client import K8SClientManager
from utils import os_client
logger = logging.getLogger(__name__)
+nodes = utils.get_pairs()
+hw_nodes = utils.get_hw_pairs()
+
+
+@pytest.fixture(scope='session', params=list(nodes.values()),
+ ids=list(nodes.keys()))
+def pair(request):
+ return request.param
+
+
+@pytest.fixture(scope='session', params=list(hw_nodes.values()),
+ ids=list(hw_nodes.keys()))
+def hw_pair(request):
+ return request.param
+
@pytest.fixture(scope='session')
def openstack_clients():
@@ -24,15 +40,6 @@
)
-nodes = utils.get_pairs()
-
-
-@pytest.fixture(scope='session', params=list(nodes.values()),
- ids=list(nodes.keys()))
-def pair(request):
- return request.param
-
-
@pytest.fixture(scope='session')
def os_resources(openstack_clients):
logger.info("Setting up resources in admin project...")
@@ -214,6 +221,15 @@
alt_project.name))
+@pytest.fixture(scope='module')
+def k8s_v1_client():
+ config = utils.get_configuration()
+ logger.info("Getting the K8S config path from the global_config.yaml file.")
+ k8s_config_path = config.get('mos_kubeconfig_path') or ""
+ k8s_manager = K8SClientManager(k8s_config_path=k8s_config_path)
+ return k8s_manager.k8s_v1
+
+
@pytest.fixture(scope="session")
def html_report():
yield
diff --git a/global_config.yaml b/global_config.yaml
index 7e2105a..463cf90 100644
--- a/global_config.yaml
+++ b/global_config.yaml
@@ -2,21 +2,30 @@
# parameters for glance image test
IMAGE_SIZE_MB: 9000
-# parameters for vm2vm test
+# parameters for vm2vm tests
CMP_HOSTS: []
image_name: "cvp.ubuntu.2004" # Use Ubuntu 20.04 LTS image
-flavor_name: 'spt-test'
+flavor_name: "spt-test"
flavor_ram: 1536
flavor_vcpus: 1
flavor_disk: 5
nova_timeout: 300
-external_network: 'public'
+external_network: "public"
custom_mtu: 'default' # 'default' or some value like 8950
-iperf_prep_string: "sudo /bin/bash -c 'echo \"91.189.88.161 archive.ubuntu.com\" >> /etc/hosts'"
-internet_at_vms: 'true' # whether Internet is present at OpenStack VMs and iperf can be installed with apt
-iperf_deb_package_dir_path: '/opt/packages/'
+iperf_prep_string: "sudo /bin/bash -c 'echo \"91.189.88.161 archive.ubuntu.com\" >> /etc/hosts'" # for VMs
+internet_at_vms: "true" # whether Internet is present at OpenStack VMs and iperf can be installed with apt
+ssh_timeout: 500 # timeout to connect to VMs' floating IPs
+
+# common parameters for vm2vm, hw2hw tests
+skipped_nodes: []
+iperf_deb_package_dir_path: "/opt/packages/"
iperf_time: 60 # time in seconds to transmit for (iperf -t option)
multiple_threads_number: 10
-multiple_threads_iperf_utility: "iperf3" # set "iperf" for v2, "iperf3" for v3
-ssh_timeout: 500
-skipped_nodes: []
+multiple_threads_iperf_utility: "iperf" # set "iperf" for v2, "iperf3" for v3
+
+# parameters for hw2hw tests
+hw_nodes_list: [] # list of HW node names
+mos_kubeconfig_path: ""
+node_ssh_key_path: ""
+node_ssh_username: "mcc-user"
+network_cidr: ""
\ No newline at end of file
diff --git a/requirements.txt b/requirements.txt
index 556da96..19159fc 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,12 +1,14 @@
jinja2 # BSD License (BSD-3-Clause)
+kubernetes==26.1.*
pandas==1.5.3 # BSD License (BSD-3-Clause)
paramiko==2.7.2 # LGPLv2.1+
-pytest==4.6.11 # MIT
+pytest==7.4.4 # MIT
python-cinderclient==6.0.0 # Apache-2.0
python-glanceclient==3.0.0 # Apache-2.0
python-keystoneclient==3.22.0 # Apache-2.0
python-neutronclient==7.1.0 # Apache-2.0
python-novaclient==7.1.0
PyYAML>=5.4 # MIT
-requests==2.24.0 # Apache-2.0
+requests>=2.25.0 # Apache-2.0
texttable==1.2.0
+urllib3>=1.26.0
diff --git a/tests/test_hw2hw.py b/tests/test_hw2hw.py
new file mode 100644
index 0000000..38bd58d
--- /dev/null
+++ b/tests/test_hw2hw.py
@@ -0,0 +1,126 @@
+import logging
+import sys
+
+import pytest
+from texttable import Texttable
+
+import utils
+from utils import helpers
+from utils import ssh
+from utils.k8s_client import K8SCliActions
+
+
+logger = logging.getLogger(__name__)
+
+
+def test_hw2hw(hw_pair, k8s_v1_client, request, html_report):
+ """
+ Simplified Performance Test: HW to HW test via some specific interface
+ 1. Get the nodes list from the config, or get computes from K8S nodes list
+ 2. Connect to the nodes, install iperf, iperf3 there
+ 3. Start iperf, iperf3 at some specific interface
+ 4. Measure HW to HW nodes via some interface IP, 1 thread
+ 5. Measure HW to HW nodes via some interface IP, 1 thread, multiple threads
+ (10 by default)
+ 6. Draw the table with all pairs and results
+ """
+ k8s_actions = K8SCliActions(k8s_v1_client)
+ config = utils.get_configuration()
+ node_ssh_key_path = config.get("node_ssh_key_path", "")
+ node_ssh_username = config.get("node_ssh_username", "mcc-user")
+ iperf_time = int(config.get('iperf_time', 60))
+ threads = int(config.get('multiple_threads_number', 10))
+ iperf_utility = config.get('multiple_threads_iperf_utility', 'iperf3')
+ result_table = Texttable(max_width=120)
+ network_cidr = config.get('network_cidr', '')
+ cleanup_list = []
+
+ if not node_ssh_key_path:
+ pytest.skip("No private key for the MOSK K8S nodes is provided. "
+ "Please set the ssh key path in the config option "
+ "'node_ssh_key_path'.")
+
+ if not network_cidr:
+ # TODO: implement setting several comma-separated net ranges
+ # TODO: take some default (storage or tenant) interface if not set
+ pytest.skip("No network interface is provided. Please set the "
+ "network range in the config option 'network_cidr'.")
+
+ # get K8S nodes' names and Internal IPs for the hw_pair list
+ nodes_info = k8s_actions.get_nodes_info_by_names(hw_pair)
+ # connect to the nodes and prepare iperf
+ node_connect = ssh.SSHTransport(address=nodes_info[0]["address"],
+ username=node_ssh_username,
+ private_key=node_ssh_key_path)
+ for node in nodes_info:
+ iperf_ip = node_connect.get_node_ip_addresses_from_cidr(
+ node['address'], network_cidr, user=node_ssh_username,
+ private_key=node_ssh_key_path)
+ node["iperf_ip"] = iperf_ip
+ logger.info(
+ f"Connecting to the node {node['name']}, IP {node['address']}")
+ iperf = ssh.IperfAtNode(
+ ip=node["address"], iperf_test_ip=iperf_ip,
+ user=node_ssh_username,
+ private_key=node_ssh_key_path)
+ if iperf.install_iperf:
+ cleanup_list.append(node["address"])
+
+ # Prepare the result table and run iperf3
+ table_rows = [[
+ 'Test Case', 'Node 1', 'Node 2', 'Network', 'Result'
+ ]]
+ # Do iperf3 measurement #1
+ measurement1 = f"HW to HW via {network_cidr} interface, 1 thread; iperf3"
+ logger.info("Doing '{}' measurement...".format(measurement1))
+ command1 = "iperf3 -c {} -t {} -B {} | grep sender | tail -n 1".format(
+ nodes_info[1]["iperf_ip"], iperf_time, nodes_info[0]["iperf_ip"])
+ logger.info(f"Running the command: {command1}")
+ result1 = node_connect.exec_command(command1)
+ res1 = (b" ".join(result1.split()[-4:-2:])).decode('utf-8')
+ logger.info("Result #1 is {}".format(res1))
+ table_rows.append([measurement1,
+ "{}".format(nodes_info[0]["name"]),
+ "{}".format(nodes_info[1]["name"]),
+ "{}".format(network_cidr),
+ "{}".format(res1)])
+
+ # Do iperf/iperf3 measurement #2
+ measurement2 = (f"HW to HW via {network_cidr} interface, {threads} "
+ f"threads; {iperf_utility}")
+ logger.info(f"Doing '{measurement2}' measurement...")
+ if iperf_utility == "iperf3":
+ command2 = '{} -c {} -P {} -t {} -B {} | grep sender | tail -n 1'\
+ .format(iperf_utility, nodes_info[1]["iperf_ip"],
+ threads, iperf_time, nodes_info[0]["iperf_ip"])
+ logger.info(f"Running the command: {command2}")
+ result2 = node_connect.exec_command(command2)
+ res2 = (b" ".join(result2.split()[-4:-2:])).decode('utf-8')
+ else:
+ iperf_utility = "iperf"
+ command2 = '{} -c {} -P {} -t {} -B {} | tail -n 1'.format(
+ iperf_utility, nodes_info[1]["iperf_ip"], threads, iperf_time,
+ nodes_info[0]["iperf_ip"])
+ logger.info(f"Running the command: {command2}")
+ result2 = node_connect.exec_command(command2)
+ res2 = (b" ".join(result2.split()[-2::])).decode('utf-8')
+ logger.info("Result #2 is {}".format(res2))
+ table_rows.append([measurement2,
+ "{}".format(nodes_info[0]["name"]),
+ "{}".format(nodes_info[1]["name"]),
+ "{}".format(network_cidr),
+ "{}".format(res2)])
+
+ # Draw the results table
+ logger.info("Drawing the table with iperf results...")
+ result_table.add_rows(table_rows)
+ sys.stdout.write('\n{}\n'.format(result_table.draw()))
+
+ # Send the results to CSV file at reports/ directory
+ helpers.create_test_result_table_csv_file(
+ table_rows, request.node.name)
+
+ # Delete the iperf, iperf3 packages if they were installed
+ for ip in cleanup_list:
+ ssh.IperfAtNode.remove_iperf_packages(
+ ip, node_ssh_username, node_ssh_key_path)
diff --git a/tests/test_vm2vm.py b/tests/test_vm2vm.py
index 8a94600..51dccdd 100644
--- a/tests/test_vm2vm.py
+++ b/tests/test_vm2vm.py
@@ -107,7 +107,8 @@
for i in range(4):
if transport1.check_vm_is_reachable_ssh(
floating_ip=vm_info[i]['fip'], timeout=ssh_timeout):
- ssh.prepare_iperf(vm_info[i]['fip'], private_key=private_key)
+ ssh.IperfAtVM(
+ vm_info[i]['fip'], private_key=private_key)
mtus.append(transport1.get_mtu_from_vm(
vm_info[i]['fip'], private_key=private_key))
logger.info("MTU at networks: {}, {}".format(
diff --git a/tests/test_vm2vm_different_routers.py b/tests/test_vm2vm_different_routers.py
index 31c4478..612e5be 100644
--- a/tests/test_vm2vm_different_routers.py
+++ b/tests/test_vm2vm_different_routers.py
@@ -125,7 +125,8 @@
for i in range(len(vms)):
if transport1.check_vm_is_reachable_ssh(
floating_ip=vm_info[i]['fip'], timeout=ssh_timeout):
- ssh.prepare_iperf(vm_info[i]['fip'], private_key=private_key)
+ ssh.IperfAtVM(
+ vm_info[i]['fip'], private_key=private_key)
mtus.append(transport1.get_mtu_from_vm(
vm_info[i]['fip'], private_key=private_key))
logger.info(
diff --git a/utils/__init__.py b/utils/__init__.py
index e4de308..155c03f 100644
--- a/utils/__init__.py
+++ b/utils/__init__.py
@@ -3,7 +3,11 @@
import sys
import yaml
+from kubernetes import client as kclient, config as kconfig
+
+from utils import exceptions
from utils import os_client
+from utils import k8s_client
logger = logging.getLogger(__name__)
@@ -23,8 +27,8 @@
cmp_hosts = config.get('CMP_HOSTS') or []
skipped_nodes = config.get('skipped_nodes') or []
if skipped_nodes:
- sys.stdout.write(("\nNotice: {} nodes will be skipped for vm2vm test"
- "".format(",".join(skipped_nodes))))
+ sys.stdout.write(("\nNotice: {} node(s) will be skipped for vm2vm "
+ "test.\n".format(", ".join(skipped_nodes))))
logger.info("Skipping nodes {}".format(",".join(skipped_nodes)))
if not cmp_hosts:
openstack_clients = os_client.OfficialClientManager(
@@ -38,13 +42,13 @@
os_actions = os_client.OSCliActions(openstack_clients)
nova_computes = os_actions.list_nova_computes()
if len(nova_computes) < 2:
- raise BaseException(
+ raise exceptions.NotEnoughNodes(
"At least 2 compute hosts are needed for VM2VM test, "
"now: {}.".format(len(nova_computes)))
cmp_hosts = [n.host_name for n in nova_computes
if n.host_name not in skipped_nodes]
if len(cmp_hosts) < 2:
- raise BaseException(
+ raise exceptions.NotEnoughNodes(
"At least 2 compute hosts are needed for VM2VM test. "
"Cannot create a pair from {}. Please check skip list, at "
"least 2 computes should be tested.".format(cmp_hosts))
@@ -54,6 +58,39 @@
return compile_pairs(cmp_hosts)
+def get_hw_pairs():
+ # get the K8S config, check whether the HW nodes list is set
+ config = get_configuration()
+ logger.info("Getting the K8S config path from the global_config.yaml file.")
+ k8s_config_path = config.get("mos_kubeconfig_path", "")
+ hw_nodes_list = config.get("hw_nodes_list", [])
+
+ # if the specific HW nodes list is not set in the config, get from K8S
+ hw_nodes = None
+ if not hw_nodes_list:
+ # fetch only compute nodes
+ label_selector = "openstack-compute-node=enabled," \
+ "openvswitch=enabled"
+ k8s_api = k8s_client.K8SClientManager(k8s_config_path=k8s_config_path)
+ k8s_actions = k8s_client.K8SCliActions(k8s_api.k8s_v1)
+ hw_nodes_list = k8s_actions.list_nodes_names(
+ label_selector=label_selector)
+
+ # remove some skipped nodes if any
+ skipped_nodes = config.get('skipped_nodes', [])
+ if skipped_nodes:
+ print(f"Notice: {', '.join(skipped_nodes)} node(s) will be skipped for"
+ f" hw2hw test.\n")
+ hw_nodes = [node for node in hw_nodes_list
+ if node not in skipped_nodes]
+ if len(hw_nodes) < 2:
+ raise exceptions.NotEnoughNodes(
+ f"At least 2 HW nodes are required to run hw2hw test. Cannot "
+ f"create a pair from {hw_nodes}. Check whether the cluster has at"
+ f" least 2 compute nodes, or the nodes are not in the skip list.")
+ return compile_pairs(hw_nodes)
+
+
def get_configuration():
"""function returns configuration for environment
and for test if it's specified"""
@@ -77,8 +114,9 @@
def check_iperf_utility(actual_iperf_utility):
valid_values = ["iperf", "iperf3"]
if actual_iperf_utility not in valid_values:
- raise BaseException("The iperf utility for multiple threads test case "
- "is not correct. Valid value is one of {}. Actual "
- "value is {}. Please set the correct value in "
- "global_config.yaml:multiple_threads_iperf_utility"
- "".format(valid_values, actual_iperf_utility))
+ raise exceptions.InvalidConfigException(
+ "The iperf utility for multiple threads test case is not correct. "
+ "Valid value is one of {}. Actual value is {}. Please set the "
+ "correct value in global_config.yaml:"
+ "multiple_threads_iperf_utility".format(
+ valid_values, actual_iperf_utility))
diff --git a/utils/exceptions.py b/utils/exceptions.py
new file mode 100644
index 0000000..1d014fc
--- /dev/null
+++ b/utils/exceptions.py
@@ -0,0 +1,10 @@
+class NotEnoughNodes(Exception):
+ pass
+
+
+class InvalidConfigException(Exception):
+ pass
+
+
+class NoPackageInstalled(Exception):
+ pass
diff --git a/utils/k8s_client.py b/utils/k8s_client.py
new file mode 100644
index 0000000..72c8b06
--- /dev/null
+++ b/utils/k8s_client.py
@@ -0,0 +1,57 @@
+import logging
+
+from kubernetes import client as kclient
+from kubernetes import config as kconfig
+
+from utils import exceptions
+
+logger = logging.getLogger(__name__)
+
+
+class K8SClientManager(object):
+ def __init__(self, k8s_config_path=None):
+ self.k8s_config_path = k8s_config_path
+ self._k8s_v1 = None
+
+ def get_k8s_v1_client(self):
+ if not self.k8s_config_path:
+ raise exceptions.InvalidConfigException(
+ "Please provide the Kubernetes config file path at "
+ "'mos_kubeconfig_path' config option."
+ )
+ kconfig.load_kube_config(config_file=self.k8s_config_path)
+ return kclient.CoreV1Api()
+
+ @property
+ def k8s_v1(self):
+ if self._k8s_v1 is None:
+ self._k8s_v1 = self.get_k8s_v1_client()
+ return self._k8s_v1
+
+
+class K8SCliActions(object):
+ INTERNAL_IP_TYPE = "InternalIP"
+
+ def __init__(self, k8s_v1_client):
+ self.k8s_v1_client = k8s_v1_client
+
+ def list_nodes_names(self, watch=False, label_selector=None):
+ nodes = self.k8s_v1_client.list_node(
+ watch=watch, label_selector=label_selector)
+ return [node.metadata.name for node in nodes.items]
+
+ def get_nodes_info_by_names(self, list_of_node_names):
+ # Get the nodes' info (just names and Internal IPs) for
+ # the specific list of the K8S nodes names
+ result_nodes_info = []
+ for node in list_of_node_names:
+ node_info = self.k8s_v1_client.read_node(name=node)
+ result_nodes_info.append({
+ "name": node_info.metadata.name,
+ "address": [
+ address.address
+ for address in node_info.status.addresses
+ if address.type == self.INTERNAL_IP_TYPE
+ ][0],
+ })
+ return result_nodes_info
diff --git a/utils/ssh.py b/utils/ssh.py
index e9e5f3a..caca30e 100644
--- a/utils/ssh.py
+++ b/utils/ssh.py
@@ -6,6 +6,8 @@
import time
import os
+from netaddr import IPNetwork, IPAddress
+
logger = logging.getLogger(__name__)
# Suppress paramiko logging
@@ -20,6 +22,10 @@
self.username = username
self.password = password
if private_key is not None:
+ if os.path.isfile(private_key):
+ with open(private_key, 'r') as key_file:
+ private_key_content = key_file.read()
+ private_key = private_key_content
self.private_key = paramiko.RSAKey.from_private_key(
StringIO(private_key))
else:
@@ -131,7 +137,7 @@
any(req in pack for req in required_packages) if
pack.endswith('.deb')]
if not iperf_deb_files:
- raise BaseException(
+ raise utils.exceptions.NoPackageInstalled(
"iperf3 or iperf *.deb packages are not found locally at path"
" {}. Please recheck 'iperf_deb_package_dir_path' variable in "
"global_config.yaml and check *.deb packages are manually "
@@ -178,7 +184,8 @@
"seconds.".format(floating_ip, attempts, bsleep))
time.sleep(bsleep)
- def get_mtu_from_vm(self, floating_ip, user='ubuntu', password='password',
+ @staticmethod
+ def get_mtu_from_vm(floating_ip, user='ubuntu', password='password',
private_key=None):
transport = SSHTransport(floating_ip, user, password, private_key)
iface = (transport.exec_command(
@@ -186,8 +193,29 @@
mtu = transport.exec_command('cat /sys/class/net/{}/mtu'.format(iface))
return mtu.decode("utf-8")
+ @staticmethod
+ def get_node_ip_addresses_from_cidr(ip, cidr, user='mcc-user',
+ private_key=None):
+ transport = SSHTransport(
+ ip, username=user, private_key=private_key)
+ command = "ip -4 addr show scope global"
+ output = transport.exec_command(command)
+ try:
+ all_ip_addresses = [line.split()[1] for line in
+ output.decode().splitlines()
+ if "/" in line.split()[1]]
+ for address in all_ip_addresses:
+ ip_addr = address.split('/')[0]
+ if IPAddress(ip_addr) in IPNetwork(cidr):
+ return ip_addr
+ except Exception as e:
+ raise utils.exceptions.InvalidConfigException(
+ f"Could not find the IP at the interface {cidr} at the node "
+ f"with K8S Private IP {ip}. Please check the configuration. "
+ f"\nException: {e}")
-class prepare_iperf(object):
+
+class IperfAtVM(object):
def __init__(self, fip, user='ubuntu', password='password',
private_key=None):
@@ -218,13 +246,49 @@
logger.info(check.decode('utf-8'))
if not check:
if internet_at_vms.lower() == 'true':
- info = "Please check the Internet access at VM."
+ info = "Please check the Internet access at VM"
else:
info = "Could not put offline iperf packages from {} to the " \
- "VM.".format(path_to_iperf_deb)
- raise BaseException("iperf3 is not installed at VM with FIP {}. "
- "{}.\nStdout, stderr at VM:\n{}\n{}"
- "".format(fip, info, stdout, stderr))
+ "VM".format(path_to_iperf_deb)
+ raise utils.exceptions.NoPackageInstalled(
+ "iperf3 is not installed at VM with FIP {}. {}.\nStdout, "
+ "stderr at VM:\n{}\n{}".format(fip, info, stdout, stderr))
# Staring iperf server
transport.exec_command('nohup iperf3 -s > file 2>&1 &')
transport.exec_command('nohup iperf -s > file 2>&1 &')
+
+
+class IperfAtNode(object):
+
+ def __init__(self, ip, iperf_test_ip, user='mcc-user', private_key=None):
+ transport = SSHTransport(ip, user, private_key=private_key)
+ # TODO: to avoid looping, both packages can be installed by one
+ # 'install -y iperf iperf3' command. 'which' command can also be
+ # executed for multiple packages at once.
+ packages = ["iperf", "iperf3"]
+ for p in packages:
+ check_path = transport.exec_command(f'which {p}')
+ if not check_path:
+ self.install_iperf = True
+ # Install iperf/iperf3 at MOSK nodes
+ logger.info(f"Installing {p} at MOSK nodes...")
+ _, stdout, stderr = transport.exec_sync(
+ f"sudo apt update && sudo apt install -y {p}")
+ else:
+ self.install_iperf = False
+ check_path = transport.exec_command(f'which {p}')
+ # Log whether iperf is installed
+ logger.info(f"{p} package path: {check_path.decode('utf-8')}")
+ if not check_path:
+ raise utils.exceptions.NoPackageInstalled(
+ f"{p} is not installed at the MOSK node with IP {ip}.\n"
+ f"Stdout, stderr at VM:\n{stdout}\n{stderr}")
+ # Staring iperf/iperf3 server
+ transport.exec_command(
+ f"nohup {p} -s -B {iperf_test_ip} > file 2>&1 &")
+
+ @staticmethod
+ def remove_iperf_packages(ip, user='mcc-user', private_key=None):
+ transport = SSHTransport(ip, user, private_key=private_key)
+ logger.info(f"Removing iperf,iperf3 packages from the node {ip}...")
+ transport.exec_command("sudo apt remove iperf iperf3 -y")