Clear pool for each new HTTP Connection
The memory consumption of Tempest can be high when HTTPS is used and
Tempest uses an older version of urllib3 as a dependency (1.26.x). This
is caused by Tempest, in conjunction with urllib3, loading certificates
into memory for each instance of RestClient and releasing them only at
the end of tests execution. The issue is particularly noticeable when
the certificate file is large.
This patch fixes this issue by clearing the pool in PoolManager after
each HTTP request.
Closes-Bug: #2088074
Change-Id: Iaede004522cd2e2386bac14ff725fa8e31d4991f
diff --git a/tempest/lib/common/http.py b/tempest/lib/common/http.py
index d163968..5bdcecd 100644
--- a/tempest/lib/common/http.py
+++ b/tempest/lib/common/http.py
@@ -60,6 +60,14 @@
retry = urllib3.util.Retry(redirect=False)
r = super(ClosingProxyHttp, self).request(method, url, retries=retry,
*args, **new_kwargs)
+
+ # Clearing the pool is necessary to free memory that holds certificates
+ # loaded by the HTTPConnection class in urllib3. This line can be
+ # removed once we require a newer version of urllib3 (e.g., 2.2.3) that
+ # does not retain certificates in memory for each HTTPConnection
+ # managed by the PoolManager.
+ self.clear()
+
if not kwargs.get('preload_content', True):
# This means we asked urllib3 for streaming content, so we
# need to return the raw response and not read any data yet
@@ -114,6 +122,14 @@
retry = urllib3.util.Retry(redirect=False)
r = super(ClosingHttp, self).request(method, url, retries=retry,
*args, **new_kwargs)
+
+ # Clearing the pool is necessary to free memory that holds certificates
+ # loaded by the HTTPConnection class in urllib3. This line can be
+ # removed once we require a newer version of urllib3 (e.g., 2.2.3) that
+ # does not retain certificates in memory for each HTTPConnection
+ # managed by the PoolManager.
+ self.clear()
+
if not kwargs.get('preload_content', True):
# This means we asked urllib3 for streaming content, so we
# need to return the raw response and not read any data yet