Add tools to run jenkins jobs and remote commands
- ./tcp_tests/utils/create_devops_env.py
Creates a fuel-devops enviromnet with VMs in disabled
state, to generate networks and addresses for inventory.
Required parameters:
export ENV_NAME=test
export LAB_CONFIG_NAME=<template directory with underlay.yml>
export MANAGER=devops
Other parameters may be required for the underlay.yml
CLI example:
export PYTHONPATH=$(pwd)
python ./tcp_tests/utils/create_devops_env.py
- ./tcp_tests/utils/run_jenkins_job.py
Run a jenkins job with parameters, wait for completion,
print the console output to stdout while waiting.
Required parameters:
export JENKINS_URL=http://host:port/
export JENKINS_USER=admin
export JENKINS_PASS=admin
CLI example:
JOB_PARAMETERS="{
\"SALT_MASTER_URL\": \"${SALTAPI_URL}\",
\"STACK_INSTALL\": \"core,cicd\"
}"
JOB_PREFIX="[ {job_name} #{build_number}:cicd {time} ] "
python ./tcp_tests/utils/run_jenkins_job.py \
--verbose \
--job-name=deploy_openstack \
--job-parameters="$JOB_PARAMETERS" \
--job-output-prefix="$JOB_PREFIX"
- ./tcp_tests/utils/get_param.py
Get a single parameter from the salt pillar.
Useful to get addresses and other scalar values.
Required parameters are the same as for 'pepper' CLI:
export SALTAPI_URL=http://${SALT_MASTER_IP}:6969/
export SALTAPI_USER='salt'
export SALTAPI_PASS='icecream12345!'
export SALTAPI_EAUTH='pam'
CLI example:
export JENKINS_HOST=$(./tcp_tests/utils/get_param.py \
-C 'I@docker:client:stack:jenkins' \
pillar.get jenkins:client:master:host)
- ./tcp_tests/utils/run_template_commands.py
Run remote commands from the ./tcp_tests/templates/
No environment varialbes are required, but may be
useful to provide the INI config from some completed
deployment.
CLI example:
export TESTS_CONFIGS=$(pwd)/test_salt_deployed.ini
./tcp_tests/utils/run_template_commands.py \
./tcp_tests/templates/<lab_name>/common_services.yaml
- some env files for sourcing to get access to different APIs.
This will simplify using the scripts above.
. ./tcp_tests/utils/env_salt # salt-api access
. ./tcp_tests/utils/env_jenkins_day01 # jenkins on salt-master
. ./tcp_tests/utils/env_jenkins_cicd # jenkins on cicd
. ./tcp_tests/utils/env_k8s # k8s api access
- fixed UnderlayManager.sudo_check_call() to remove
deprecation warning.
Improvements to JenkisClient:
- Add JenkinsWrapper class to workaround the bug
https://bugs.launchpad.net/python-jenkins/+bug/1775047
which is happened to CICD Jenkins behind the haproxy
- improved waiting for start of the job in run_build()
- new argument 'interval' in wait_end_of_build(), to set
the polling interval while waiting the job
- new argument 'job_output_prefix' in wait_end_of_build(),
which allows to set the prefix to each line of the console
output of the job; with some pre-defined template keys.
- improved printing the job output in case of non-unicode characters
Change-Id: Ie7d1324d8247e55ba9c0f0492ca39fc176ff4935
diff --git a/tcp_tests/utils/__init__.py b/tcp_tests/utils/__init__.py
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/tcp_tests/utils/__init__.py
diff --git a/tcp_tests/utils/create_devops_env.py b/tcp_tests/utils/create_devops_env.py
new file mode 100755
index 0000000..d7c062c
--- /dev/null
+++ b/tcp_tests/utils/create_devops_env.py
@@ -0,0 +1,27 @@
+#!/usr/bin/env python
+
+import os
+import sys
+
+sys.path.append(os.getcwd())
+try:
+ from tcp_tests.helpers import ext
+ from tcp_tests.fixtures import config_fixtures
+ from tcp_tests.managers import envmanager_devops
+except ImportError:
+ print("ImportError: Run the application from the tcp-qa directory or "
+ "set the PYTHONPATH environment variable to directory which contains"
+ " ./tcp_tests")
+ sys.exit(1)
+
+
+def main():
+ """Create fuel-devops environment from template"""
+ config = config_fixtures.config()
+ env = envmanager_devops.EnvironmentManager(config=config)
+ if not env.has_snapshot(ext.SNAPSHOT.hardware):
+ env.create_snapshot(ext.SNAPSHOT.hardware)
+
+
+if __name__ == '__main__':
+ sys.exit(main())
diff --git a/tcp_tests/utils/env_jenkins_cicd b/tcp_tests/utils/env_jenkins_cicd
new file mode 100755
index 0000000..04cdd56
--- /dev/null
+++ b/tcp_tests/utils/env_jenkins_cicd
@@ -0,0 +1,39 @@
+#!/bin/bash
+
+# Source file to set access credentials to Jenkins API on CICD cluster
+# Requires parameters to work with libpepper:
+# - SALTAPI_URL
+# - SALTAPI_USER
+# - SALTAPI_PASS
+# Example usage:
+# $> export ENV_NAME=some-test-environment
+# $> . ./tcp_tests/utils/env_salt
+# $> . ./tcp_tests/utils/env_jenkins_cicd
+# $> ./tcp_tests/utils/run_jenkins_job.py --verbose --job-name=deploy_openstack --job-parameters="{... json ...}"
+
+CURRENT_DIR=$(dirname $(readlink -f "${BASH_SOURCE[0]}"))
+export PYTHONPATH="${CURRENT_DIR}/../.."
+
+if [ -z "$SALTAPI_URL" ]; then
+ echo "$SALTAPI_URL not found in the environment variables, getting values from salt-master is impossible."
+ unset JENKINS_URL
+ unset JENKINS_USER
+ unset JENKINS_PASS
+ unset JENKINS_START_TIMEOUT
+ unset JENKINS_BUILD_TIMEOUT
+else
+ MASTER="get_param.py -C I@docker:client:stack:jenkins pillar.get jenkins:client:master"
+ export JENKINS_HOST=$(${CURRENT_DIR}/${MASTER}:host)
+ export JENKINS_PORT=$(${CURRENT_DIR}/${MASTER}:port)
+ export JENKINS_URL=http://${JENKINS_HOST}:${JENKINS_PORT}
+ export JENKINS_USER=$(${CURRENT_DIR}/${MASTER}:username)
+ export JENKINS_PASS=$(${CURRENT_DIR}/${MASTER}:password)
+ export JENKINS_START_TIMEOUT=60
+ export JENKINS_BUILD_TIMEOUT=1800
+fi
+
+echo "export JENKINS_URL='$JENKINS_URL' # Jenkins API URL"
+echo "export JENKINS_USER='${JENKINS_USER}' # Jenkins API username"
+echo "export JENKINS_PASS='${JENKINS_PASS}' # Jenkins API password or token"
+echo "export JENKINS_START_TIMEOUT='${JENKINS_START_TIMEOUT}' # Timeout waiting for job in queue to start building"
+echo "export JENKINS_BUILD_TIMEOUT='${JENKINS_BUILD_TIMEOUT}' # Timeout waiting for building job to complete"
diff --git a/tcp_tests/utils/env_jenkins_day01 b/tcp_tests/utils/env_jenkins_day01
new file mode 100755
index 0000000..c084a52
--- /dev/null
+++ b/tcp_tests/utils/env_jenkins_day01
@@ -0,0 +1,36 @@
+#!/bin/bash
+
+# Source file to set access credentials to Jenkins API on salt-master node (day01 deployment steps)
+# Requires:
+# - ENV_NAME
+# Example usage:
+# $> . ./tcp_tests/utils/env_salt
+# $> . ./tcp_tests/utils/env_jenkins_day01
+# $> ./tcp_tests/utils/run_jenkins_job.py --verbose --job-name=deploy_openstack --job-parameters="{... json ...}"
+
+CURRENT_DIR=$(dirname $(readlink -f "${BASH_SOURCE[0]}"))
+export PYTHONPATH=${CURRENT_DIR}/../..
+
+SALT_MASTER_IP=${SALT_MASTER_IP:-$(for node in $(dos.py slave-ip-list --address-pool-name admin-pool01 ${ENV_NAME}); do echo $node|grep cfg01|cut -d',' -f2; done)}
+
+if [ -z "$SALT_MASTER_IP" ]; then
+ echo "SALT_MASTER_IP not found in the environment '${ENV_NAME}'"
+ unset JENKINS_URL
+ unset JENKINS_USER
+ unset JENKINS_PASS
+ unset JENKINS_START_TIMEOUT
+ unset JENKINS_BUILD_TIMEOUT
+else
+ # For run_jenkins_job.py
+ export JENKINS_URL=http://${SALT_MASTER_IP}:8081
+ export JENKINS_USER=admin
+ export JENKINS_PASS=r00tme
+ export JENKINS_START_TIMEOUT=60
+ export JENKINS_BUILD_TIMEOUT=1800
+fi
+
+echo "export JENKINS_URL='$JENKINS_URL' # Jenkins API URL"
+echo "export JENKINS_USER='${JENKINS_USER}' # Jenkins API username"
+echo "export JENKINS_PASS='${JENKINS_PASS}' # Jenkins API password or token"
+echo "export JENKINS_START_TIMEOUT='${JENKINS_START_TIMEOUT}' # Timeout waiting for job in queue to start building"
+echo "export JENKINS_BUILD_TIMEOUT='${JENKINS_BUILD_TIMEOUT}' # Timeout waiting for building job to complete"
diff --git a/tcp_tests/utils/env_k8s b/tcp_tests/utils/env_k8s
new file mode 100755
index 0000000..01e20fd
--- /dev/null
+++ b/tcp_tests/utils/env_k8s
@@ -0,0 +1,34 @@
+#!/bin/bash
+
+# Source file to set access credentials to Kubernetes API
+# Requires parameters to work with libpepper:
+# - SALTAPI_URL
+# - SALTAPI_USER
+# - SALTAPI_PASS
+# Example usage:
+# $> export ENV_NAME=some-test-environment
+# $> . ./tcp_tests/utils/env_salt
+# $> . ./tcp_tests/utils/env_k8s
+# # now you can run tcp-qa test cases for k8s
+
+CURRENT_DIR=$(dirname $(readlink -f "${BASH_SOURCE[0]}"))
+export PYTHONPATH=${CURRENT_DIR}/../..
+
+if [ -z "$SALTAPI_URL" ]; then
+ echo "$SALTAPI_URL not found in the environment variables, getting values from salt-master is impossible."
+ unset kube_host
+ unset kube_apiserver_port
+ unset kubernetes_admin_user
+ unset kubernetes_admin_password
+else
+ KUBE_TARGET='I@haproxy:proxy:enabled:true and I@kubernetes:master and *01*'
+ export kube_host=$(${PYTHONPATH}/tcp_tests/utils/get_param.py -C "${KUBE_TARGET}" pillar.get haproxy:proxy:listen:k8s_secure:binds:address)
+ export kube_apiserver_port=$(${PYTHONPATH}/tcp_tests/utils/get_param.py -C "${KUBE_TARGET}" pillar.get haproxy:proxy:listen:k8s_secure:binds:port)
+ export kubernetes_admin_user=$(${PYTHONPATH}/tcp_tests/utils/get_param.py -C "${KUBE_TARGET}" pillar.get kubernetes:master:admin:username)
+ export kubernetes_admin_password=$(${PYTHONPATH}/tcp_tests/utils/get_param.py -C "${KUBE_TARGET}" pillar.get kubernetes:master:admin:password)
+fi
+
+echo "export kube_host='$kube_host' # Kubernetes API host"
+echo "export kube_apiserver_port='${kube_apiserver_port}' # Kubernetes API port"
+echo "export kubernetes_admin_user='${kubernetes_admin_user}' # Kubernetes admin user"
+echo "export kubernetes_admin_password='${kubernetes_admin_password}' # Kubernetes admin password"
diff --git a/tcp_tests/utils/env_salt b/tcp_tests/utils/env_salt
new file mode 100755
index 0000000..12c4575
--- /dev/null
+++ b/tcp_tests/utils/env_salt
@@ -0,0 +1,34 @@
+#!/bin/bash
+
+# Source file to set access credentials to salt-api for using with libpepper
+# Requires:
+# - ENV_NAME
+# Example usage:
+# $> . ./tcp_tests/utils/env_salt
+# $> pepper -C 'I@linux:system' test.ping
+
+CURRENT_DIR=$(dirname $(readlink -f "${BASH_SOURCE[0]}"))
+export PYTHONPATH=${CURRENT_DIR}/../..
+
+export SALT_MASTER_IP=${SALT_MASTER_IP:-$(for node in $(dos.py slave-ip-list --address-pool-name admin-pool01 ${ENV_NAME}); do echo $node|grep cfg01|cut -d',' -f2; done)}
+
+if [ -z "$SALT_MASTER_IP" ]; then
+ echo "SALT_MASTER_IP not found in the environment '${ENV_NAME}'"
+ unset SALT_MASTER_IP
+ unset SALTAPI_URL
+ unset SALTAPI_USER
+ unset SALTAPI_PASS
+ unset SALTAPI_EAUTH
+else
+ # For pepper client
+ export SALTAPI_URL=http://${SALT_MASTER_IP}:6969/
+ export SALTAPI_USER='salt'
+ export SALTAPI_PASS='hovno12345!'
+ export SALTAPI_EAUTH='pam'
+fi
+
+echo "export SALT_MASTER_IP='${SALT_MASTER_IP}'"
+echo "export SALTAPI_URL='${SALTAPI_URL}'"
+echo "export SALTAPI_USER='${SALTAPI_USER}'"
+echo "export SALTAPI_PASS='${SALTAPI_PASS}'"
+echo "export SALTAPI_EAUTH='${SALTAPI_EAUTH}'"
diff --git a/tcp_tests/utils/get_param.py b/tcp_tests/utils/get_param.py
new file mode 100755
index 0000000..6179e70
--- /dev/null
+++ b/tcp_tests/utils/get_param.py
@@ -0,0 +1,62 @@
+#!/usr/bin/env python
+
+"""
+A wrapper to ``pepper``, a CLI interface to a remote salt-api instance.
+
+Return a single parameter from the salt model for specified
+target and pillar.
+
+Fails if the result contains more than one parameter.
+
+Use the pepper CLI parameters to set salt-api access parameters
+or set the environment variables:
+
+ export SALTAPI_URL=http://${SALT_MASTER_IP}:6969/;
+ export SALTAPI_USER='salt';
+ export SALTAPI_PASS='pass';
+ export SALTAPI_EAUTH='pam';
+"""
+from __future__ import print_function
+
+import sys
+import json
+
+from pepper import cli
+
+
+runner = cli.PepperCli()
+runner.parser.description = __doc__
+
+
+if len(sys.argv) <= 1:
+ sys.argv.append('--help')
+
+results = []
+for res in runner.run():
+ results.append(res)
+
+if not results:
+ print("Empty response", file=sys.stderr)
+ sys.exit(1)
+
+if len(results) > 1:
+ print("Too many results", file=sys.stderr)
+ sys.exit(1)
+
+if results[0][0] != 0:
+ print("Error code returned", file=sys.stderr)
+ sys.exit(results[0][0])
+
+data = json.loads(results[0][1])
+nodes = data['return'][0].keys()
+
+if not nodes:
+ print("Wrong target: no minions selected", file=sys.stderr)
+ sys.exit(1)
+
+if len(nodes) > 1:
+ print("Wrong target: too many minions selected: {0}"
+ .format(nodes), file=sys.stderr)
+ sys.exit(1)
+
+print(data['return'][0][nodes[0]])
diff --git a/tcp_tests/utils/run_jenkins_job.py b/tcp_tests/utils/run_jenkins_job.py
new file mode 100755
index 0000000..00ebec6
--- /dev/null
+++ b/tcp_tests/utils/run_jenkins_job.py
@@ -0,0 +1,163 @@
+#!/usr/bin/env python
+
+import argparse
+import os
+import sys
+
+import json
+
+sys.path.append(os.getcwd())
+try:
+ from tcp_tests.managers.jenkins.client import JenkinsClient
+except ImportError:
+ print("ImportError: Run the application from the tcp-qa directory or "
+ "set the PYTHONPATH environment variable to directory which contains"
+ " ./tcp_tests")
+ sys.exit(1)
+
+
+EXIT_CODES = {
+ "SUCCESS": 0,
+ # 1 - python runtime execution error
+ # 2 - job unknown status
+ "FAILURE": 3,
+ "UNSTABLE": 4,
+ "ABORTED": 5,
+ "DISABLED": 6
+ # 10 - invalid cli options
+}
+
+
+def load_params():
+ """
+ Parse CLI arguments and environment variables
+
+ Returns: ArgumentParser instance
+ """
+ env_host = os.environ.get('JENKINS_URL', None)
+ env_username = os.environ.get('JENKINS_USER', None)
+ env_password = os.environ.get('JENKINS_PASS', None)
+ env_start_timeout = os.environ.get('JENKINS_START_TIMEOUT', 1800)
+ env_build_timeout = os.environ.get('JENKINS_BUILD_TIMEOUT', 3600 * 4)
+
+ parser = argparse.ArgumentParser(description=(
+ 'Host, username and password may be specified either by the command '
+ 'line arguments or using environment variables: JENKINS_URL, '
+ 'JENKINS_USER, JENKINS_PASS, JENKINS_START_TIMEOUT, '
+ 'JENKINS_BUILD_TIMEOUT. \nCommand line arguments have the highest '
+ 'priority, after that the environment variables are used as defaults.'
+ ))
+ parser.add_argument('--host',
+ metavar='JENKINS_URL',
+ help='Jenkins Host',
+ default=env_host)
+ parser.add_argument('--username',
+ metavar='JENKINS_USER',
+ help='Jenkins Username', default=env_username)
+ parser.add_argument('--password',
+ metavar='JENKINS_PASS',
+ help='Jenkins Password or API token',
+ default=env_password)
+ parser.add_argument('--start-timeout',
+ metavar='JENKINS_START_TIMEOUT',
+ help='Timeout waiting until build is started',
+ default=env_start_timeout,
+ type=int)
+ parser.add_argument('--build-timeout',
+ metavar='JENKINS_BUILD_TIMEOUT',
+ help='Timeout waiting until build is finished',
+ default=env_build_timeout,
+ type=int)
+ parser.add_argument('--job-name',
+ help='Jenkins job name to run',
+ default=None)
+ parser.add_argument('--job-parameters',
+ metavar='json-dict',
+ help=('Job parameters to use instead of default '
+ 'values, as a json string, for example: '
+ '--job-parameters=\'{"SALT_MASTER_URL": '
+ '"http://localhost:6969"}\''),
+ default={}, type=json.loads)
+ parser.add_argument('--job-output-prefix',
+ help=('Jenkins job output prefix for each line in the '
+ 'output, if --verbose is enabled. Useful for the'
+ ' pipelines that use multiple different runs of '
+ 'jobs. The string is a template for python '
+ 'format() function where the following arguments'
+ ' are allowed: job_name, build_number. '
+ 'Example: --job-output-prefix=\"[ {job_name} '
+ '#{build_number}, core ]\"'),
+ default='',
+ type=str)
+ parser.add_argument('--verbose',
+ action='store_const',
+ const=True,
+ help='Show build console output',
+ default=False)
+ return parser
+
+
+def print_build_header(build, job_params, opts):
+ print('\n#############################################################')
+ print('##### Building job [{0}] #{1} (timeout={2}) with the following '
+ 'parameters:'.format(build[0], build[1], opts.build_timeout))
+ print('##### ' + '\n##### '.join(
+ [str(key) + ": " + str(val) for key, val in job_params.iteritems()]
+ ))
+ print('#############################################################')
+
+
+def print_build_footer(build, result, url):
+ print('\n\n#############################################################')
+ print('##### Completed job [{0}] #{1} at {2}: {3}'
+ .format(build[0], build[1], url, result))
+ print('#############################################################\n')
+
+
+def run_job(opts):
+
+ jenkins = JenkinsClient(
+ host=opts.host,
+ username=opts.username,
+ password=opts.password)
+
+ job_params = jenkins.make_defults_params(opts.job_name)
+ job_params.update(opts.job_parameters)
+
+ build = jenkins.run_build(opts.job_name,
+ job_params,
+ verbose=opts.verbose,
+ timeout=opts.start_timeout)
+ if opts.verbose:
+ print_build_header(build, job_params, opts)
+
+ jenkins.wait_end_of_build(
+ name=build[0],
+ build_id=build[1],
+ timeout=opts.build_timeout,
+ interval=1,
+ verbose=opts.verbose,
+ job_output_prefix=opts.job_output_prefix)
+ result = jenkins.build_info(name=build[0],
+ build_id=build[1])['result']
+ if opts.verbose:
+ print_build_footer(build, result, opts.host)
+
+ return EXIT_CODES.get(result, 2)
+
+
+def main(args=None):
+ parser = load_params()
+ opts = parser.parse_args()
+
+ if opts.host is None or opts.job_name is None:
+ print("JENKINS_URL and a job name are required!")
+ parser.print_help()
+ return 10
+ else:
+ exit_code = run_job(opts)
+ return exit_code
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/tcp_tests/utils/run_template_commands.py b/tcp_tests/utils/run_template_commands.py
new file mode 100755
index 0000000..5d2c178
--- /dev/null
+++ b/tcp_tests/utils/run_template_commands.py
@@ -0,0 +1,62 @@
+#!/usr/bin/env python
+
+import argparse
+import os
+import sys
+
+sys.path.append(os.getcwd())
+try:
+ from tcp_tests.fixtures import config_fixtures
+ from tcp_tests.managers import underlay_ssh_manager
+ from tcp_tests.managers import execute_commands
+except ImportError:
+ print("ImportError: Run the application from the tcp-qa directory or "
+ "set the PYTHONPATH environment variable to directory which contains"
+ " ./tcp_tests")
+ sys.exit(1)
+
+
+def load_params():
+ """
+ Parse CLI arguments and environment variables
+
+ Returns: ArgumentParser instance
+ """
+ parser = argparse.ArgumentParser(description=(
+ 'Run commands from yaml templates'
+ ))
+ parser.add_argument('path_to_template',
+ help='Path to YAML template')
+ parser.add_argument('--template-steps-label',
+ help=('Text that will be shown as steps label'),
+ default='',
+ type=str)
+
+ return parser
+
+
+def main():
+ """Create fuel-devops environment from template"""
+ parser = load_params()
+ opts = parser.parse_args()
+
+ if opts.path_to_template is None:
+ parser.print_help()
+ return 10
+
+ path = os.path.abspath(opts.path_to_template)
+ label = opts.template_steps_label
+ if not label:
+ label = os.path.basename(path).split('.')[0]
+
+ config = config_fixtures.config()
+ underlay = underlay_ssh_manager.UnderlaySSHManager(config)
+
+ commands = underlay.read_template(path)
+
+ commander = execute_commands.ExecuteCommandsMixin(config, underlay)
+ commander.execute_commands(commands, label=label)
+
+
+if __name__ == '__main__':
+ sys.exit(main())