Make runPepper more resistant against SaltReqTimeoutError
* In rare condition, salt zmq threads may flap with raise
SaltReqTimeoutError. During those period, salt-api may return
50X answer and drop whole deployment process.
Unfortunatly, in salt no option to increase SaltReqTimeoutError
for all threads, (default 3 for many different threads).
Salt itself allow to configure only few *_tries option for exact
threads(like auth or job render).
* Those patch add simply crutch with retry, in case 50X error
has been detected in stderr from pepper call
* Misc:
- Extend mk.Python.runVirtualenvCommand
- Extend mk.Common.runPepperCommand
- add failover for 50X and retry
- Refactor mirantis.mk.Common.shCmdStatus
Prod-relaeted: PROD-30839(PROD:30839)
Change-Id: I18b152c5f22c8fb602a21a34ea06a4c543d8ae26
diff --git a/src/com/mirantis/mk/Common.groovy b/src/com/mirantis/mk/Common.groovy
index aecbc9b..296dd38 100644
--- a/src/com/mirantis/mk/Common.groovy
+++ b/src/com/mirantis/mk/Common.groovy
@@ -505,18 +505,19 @@
*/
def shCmdStatus(cmd) {
+ // Set +x , to hide odd messages about temp file manipulations
def res = [:]
- def stderr = sh(script: 'mktemp', returnStdout: true).trim()
- def stdout = sh(script: 'mktemp', returnStdout: true).trim()
+ def stderr = sh(script: 'set +x ; mktemp', returnStdout: true).trim()
+ def stdout = sh(script: 'set +x ; mktemp', returnStdout: true).trim()
try {
def status = sh(script: "${cmd} 1>${stdout} 2>${stderr}", returnStatus: true)
- res['stderr'] = sh(script: "cat ${stderr}", returnStdout: true)
- res['stdout'] = sh(script: "cat ${stdout}", returnStdout: true)
+ res['stderr'] = sh(script: "set +x; cat ${stderr}", returnStdout: true).trim()
+ res['stdout'] = sh(script: "set +x; cat ${stdout}", returnStdout: true).trim()
res['status'] = status
} finally {
- sh(script: "rm ${stderr}", returnStdout: true)
- sh(script: "rm ${stdout}", returnStdout: true)
+ sh(script: "set +x; rm ${stderr}")
+ sh(script: "set +x; rm ${stdout}")
}
return res