Fix race condition in reload_on_sighup functional
Race condition appears in next situation:
1. First thread calls _set_config_value for one section.
2. Second thread calls _set_config_value for another section.
3. First thread. Config option value set, calls
open(self.config_file, 'wb'), which erases all file content.
4. Second thread. In previous point moment second thread tries to
set config option value to self.config_file, which is empty (see 3).
So, NoSectionError exception raised.
This patch adds ten retries for setting option value, if NoSectionError
raised, i.e. try to wait until self.config_file is busy.
Change-Id: Ic54ea287ebe4724511f75d42677cae5dfdec4e57
Closes-bug: #1535766
diff --git a/common/config.py b/common/config.py
index f8ae075..3aee48f 100644
--- a/common/config.py
+++ b/common/config.py
@@ -130,6 +130,11 @@
default=30,
help="Timeout in seconds to wait for adding or removing child"
"process after receiving of sighup signal"),
+ cfg.IntOpt('sighup_config_edit_retries',
+ default=10,
+ help='Count of retries to edit config file during sighup. If '
+ 'another worker already edit config file, file can be '
+ 'busy, so need to wait and try edit file again.'),
cfg.StrOpt('heat-config-notify-script',
default=('heat-config-notify'),
help="Path to the script heat-config-notify"),
diff --git a/functional/test_reload_on_sighup.py b/functional/test_reload_on_sighup.py
index cba5386..b014f49 100644
--- a/functional/test_reload_on_sighup.py
+++ b/functional/test_reload_on_sighup.py
@@ -28,8 +28,27 @@
def _set_config_value(self, service, key, value):
config = configparser.ConfigParser()
- config.read(self.config_file)
- config.set(service, key, value)
+
+ # NOTE(prazumovsky): If there are several workers, there can be
+ # situation, when one thread opens self.config_file for writing
+ # (so config_file erases with opening), in that moment other thread
+ # intercepts to this file and try to set config option value, i.e.
+ # write to file, which is already erased by first thread, so,
+ # NoSectionError raised. So, should wait until first thread writes to
+ # config_file.
+ retries_count = self.conf.sighup_config_edit_retries
+ while True:
+ config.read(self.config_file)
+ try:
+ config.set(service, key, value)
+ except configparser.NoSectionError:
+ if retries_count <= 0:
+ raise
+ retries_count -= 1
+ eventlet.sleep(1)
+ else:
+ break
+
with open(self.config_file, 'wb') as f:
config.write(f)