Added test to check mounted file systems on control plane VMs
Problem: sometimes it happens that after KVM host is rebooted,
its VMs come up with missing mounts (e.g. ctl node can have
keystone volumes missed). We had such an issue. It can happen
because of the hardware / host system performance issue or
network misconfiguration, but such issue can appear after
e.g. HA testing, or when KVM is rebooted. So, such test
will detect the inconsistent mounts.
Created the test to check that mounted file systems are
consistent on the virtual control plane nodes (e.g. on ctl,
prx, etc nodes). The nodes like kvm, cmp, ceph OSD, nodes
with docker (like k8s nodes, cid, mon) are skipped.
To skip other nodes if needed, add the node or group in the
config (skipped_nodes, skipped_groups).
Change-Id: Iab5311060790bd2fdfc8587e4cb8fc63cc3a0a13
Related-PROD: PROD-28247
diff --git a/test_set/cvp-sanity/tests/test_mounts.py b/test_set/cvp-sanity/tests/test_mounts.py
new file mode 100644
index 0000000..3b5b697
--- /dev/null
+++ b/test_set/cvp-sanity/tests/test_mounts.py
@@ -0,0 +1,43 @@
+import json
+import pytest
+
+
+def test_mounted_file_systems(local_salt_client, nodes_in_group):
+ """
+ # Get all mount points from each node in the group with the next command: `df -h | awk '{print $1}'`
+ # Check that all mount points are similar for each node in the group
+ """
+ mounts_by_nodes = local_salt_client.cmd("L@"+','.join(nodes_in_group),
+ 'cmd.run',
+ ["df -h | awk '{print $1}'"],
+ expr_form='compound')
+
+ # Let's exclude cmp, kvm, ceph OSD nodes, mon, cid, k8s-ctl, k8s-cmp nodes
+ # These nodes will have different mounts and this is expected
+ exclude_nodes = local_salt_client.cmd(
+ ("I@nova:compute or "
+ "I@ceph:osd or "
+ "I@salt:control or "
+ "I@prometheus:server and not I@influxdb:server or "
+ "I@kubernetes:* and not I@etcd:* or "
+ "I@docker:host and not I@prometheus:server and not I@kubernetes:*"),
+ 'test.ping', expr_form='compound').keys()
+
+ if len(mounts_by_nodes.keys()) < 2:
+ pytest.skip("Nothing to compare - only 1 node")
+
+ result = {}
+ pretty_result = {}
+
+ for node in mounts_by_nodes:
+ if node in exclude_nodes:
+ continue
+ result[node] = "\n".join(sorted(mounts_by_nodes[node].split()))
+ pretty_result[node] = sorted(mounts_by_nodes[node].split())
+
+ if not result:
+ pytest.skip("These nodes are skipped")
+
+ assert len(set(result.values())) == 1,\
+ "The nodes in the same group have different mounts:\n{}".format(
+ json.dumps(pretty_result, indent=4))