Add sandesh_send_rate_limit parameter
Related-Bug: PROD-29404 (PROD:29404)
Generators of big environments with high traffic and modification requests
can generate huge amount of sandesh messages. It can cause oveloading
and high resource consumption of analytics services.
Amount of sandesh messages can be decreased using
`sandesh_send_rate_limit`.
Change-Id: Idf6055c57e5ca5d2b608ee59ad2c104daadc5c0a
(cherry picked from commit 4c4d45a70c029ac3ed0ee7c254eb6ebc983c6d6a)
diff --git a/README.rst b/README.rst
index 36034ca..6220886 100644
--- a/README.rst
+++ b/README.rst
@@ -1438,6 +1438,96 @@
``contrail-schema``, ``contrail-device-manager`` and ``contrail-svc-monitor`` config services already
have additional ``backup`` state by default.
+Setup Sandesh rate limit
+------------------------
+Sandesh send rate limit can be used to throttle system logs transmitted per second. System logs are
+dropped if the sending rate is exceeded.
+
+It is possible to use only global limit for all services of contrail component.
+
+.. code-block:: yaml
+
+ opencontrail:
+ control:
+ ....
+ sandesh_send_rate_limits:
+ global: 10
+ ....
+
+
+Global limit and limit for specific service can be defined together where specific service limit
+has higher priority.
+
+.. code-block:: yaml
+
+ opencontrail:
+ config:
+ ....
+ sandesh_send_rate_limits:
+ global: 10
+ config_api: 3
+ schema: 5
+ ....
+
+Only specific service limit can be defined. In this case other services of Contrail component
+keep default value (0) of `sandesh_send_rate_limit`
+
+.. code-block:: yaml
+
+ opencontrail:
+ collector:
+ ....
+ sandesh_send_rate_limits:
+ collector: 100
+ analytics-api: 50
+ ....
+
+Full list:
+
+.. code-block:: yaml
+
+ opencontrail:
+ control:
+ ....
+ sandesh_send_rate_limits:
+ global: 100
+ control: 10
+ dns: 10
+ nodemgr: 10
+ ....
+
+ config:
+ ....
+ sandesh_send_rate_limits:
+ global: 100
+ config_api: 10
+ schema: 10
+ svc_monitor: 10
+ device_manager: 10
+ nodemgr: 10
+ ....
+
+ collector:
+ ....
+ sandesh_send_rate_limits:
+ global: 100
+ collector: 10
+ analytics_api: 10
+ query_engine: 10
+ alarm_gen: 10
+ snmp_collector: 10
+ topology: 10
+ nodemgr: 10
+ ....
+
+ compute:
+ ....
+ sandesh_send_rate_limits:
+ global: 100
+ agent: 10
+ nodemgr: 10
+ ....
+
Usage
=====
diff --git a/opencontrail/files/4.0/contrail-alarm-gen.conf b/opencontrail/files/4.0/contrail-alarm-gen.conf
index b5d7cff..00b957c 100644
--- a/opencontrail/files/4.0/contrail-alarm-gen.conf
+++ b/opencontrail/files/4.0/contrail-alarm-gen.conf
@@ -10,8 +10,13 @@
kafka_broker_list = {% for member in collector.database.members %}{{ member.host }}:9092 {% endfor %}
partitions=30
zk_list = {% for member in collector.database.members %}{{ member.host }}:2182 {% endfor %}
-{%- if collector.version >= 4.1 %}
+{%- if collector.get('sandesh_send_rate_limits', {}).get('alarm_gen') %}
+sandesh_send_rate_limit={{ collector.sandesh_send_rate_limits.alarm_gen }}
+{%- elif collector.get('sandesh_send_rate_limits', {}).get('global') %}
+sandesh_send_rate_limit={{ collector.sandesh_send_rate_limits.global }}
+{%- endif %}
+{%- if collector.version >= 4.1 %}
[CONFIGDB]
{%- endif %}
#rabbitmq_server_list = xx.xx.xx.xx
diff --git a/opencontrail/files/4.0/contrail-analytics-api.conf b/opencontrail/files/4.0/contrail-analytics-api.conf
index 6743e22..98ba5ed 100644
--- a/opencontrail/files/4.0/contrail-analytics-api.conf
+++ b/opencontrail/files/4.0/contrail-analytics-api.conf
@@ -29,6 +29,11 @@
analytics_config_audit_ttl={{ collector.data_ttl }}
analytics_statistics_ttl={{ collector.data_ttl }}
analytics_flow_ttl={{ collector.data_ttl }}
+{%- if collector.get('sandesh_send_rate_limits', {}).get('analytics_api') %}
+sandesh_send_rate_limit={{ collector.sandesh_send_rate_limits.analytics_api }}
+{%- elif collector.get('sandesh_send_rate_limits', {}).get('global') %}
+sandesh_send_rate_limit={{ collector.sandesh_send_rate_limits.global }}
+{%- endif %}
[REDIS]
redis_query_port = 6379
diff --git a/opencontrail/files/4.0/contrail-analytics-nodemgr.conf b/opencontrail/files/4.0/contrail-analytics-nodemgr.conf
index 48f812d..3db776d 100644
--- a/opencontrail/files/4.0/contrail-analytics-nodemgr.conf
+++ b/opencontrail/files/4.0/contrail-analytics-nodemgr.conf
@@ -1,3 +1,13 @@
{%- from "opencontrail/map.jinja" import collector with context %}
+[DEFAULTS]
+# Sandesh send rate limit can be used to throttle system logs transmitted per
+# second. System logs are dropped if the sending rate is exceeded
+# sandesh_send_rate_limit=
+{%- if collector.get('sandesh_send_rate_limits', {}).get('nodemgr') %}
+sandesh_send_rate_limit={{ collector.sandesh_send_rate_limits.nodemgr }}
+{%- elif collector.get('sandesh_send_rate_limits', {}).get('global') %}
+sandesh_send_rate_limit={{ collector.sandesh_send_rate_limits.global }}
+{%- endif %}
+
[COLLECTOR]
server_list={% for member in collector.analytics.members %}{{ member.host }}:8086 {% endfor %}
\ No newline at end of file
diff --git a/opencontrail/files/4.0/contrail-api.conf b/opencontrail/files/4.0/contrail-api.conf
index 8ccb5c1..975b093 100644
--- a/opencontrail/files/4.0/contrail-api.conf
+++ b/opencontrail/files/4.0/contrail-api.conf
@@ -47,6 +47,11 @@
{%- if config.get('list_optimization_enabled', False) %}
list_optimization_enabled=True
{%- endif %}
+{%- if config.get('sandesh_send_rate_limits', {}).get('config_api') %}
+sandesh_send_rate_limit={{ config.sandesh_send_rate_limits.config_api }}
+{%- elif config.get('sandesh_send_rate_limits', {}).get('global') %}
+sandesh_send_rate_limit={{ config.sandesh_send_rate_limits.global }}
+{%- endif %}
# (StrOpt) default encoding to use for string resources.
# this is optional variable. Define it if you want to override
diff --git a/opencontrail/files/4.0/contrail-collector.conf b/opencontrail/files/4.0/contrail-collector.conf
index f7beae2..505ad1d 100644
--- a/opencontrail/files/4.0/contrail-collector.conf
+++ b/opencontrail/files/4.0/contrail-collector.conf
@@ -58,6 +58,11 @@
# Sandesh send rate limit can be used to throttle system logs transmitted per
# second. System logs are dropped if the sending rate is exceeded
# sandesh_send_rate_limit=
+{%- if collector.get('sandesh_send_rate_limits', {}).get('collector') %}
+sandesh_send_rate_limit={{ collector.sandesh_send_rate_limits.collector }}
+{%- elif collector.get('sandesh_send_rate_limits', {}).get('global') %}
+sandesh_send_rate_limit={{ collector.sandesh_send_rate_limits.global }}
+{%- endif %}
[COLLECTOR]
port=8086
diff --git a/opencontrail/files/4.0/contrail-config-nodemgr.conf b/opencontrail/files/4.0/contrail-config-nodemgr.conf
index 6653427..55f27fb 100644
--- a/opencontrail/files/4.0/contrail-config-nodemgr.conf
+++ b/opencontrail/files/4.0/contrail-config-nodemgr.conf
@@ -5,6 +5,11 @@
hostip={{ database.bind.get('host', '127.0.0.1') }}
db_port={{ database.bind.get('port_configdb', database.bind.get('port', 9042)) }}
{%- endif %}
+{%- if config.get('sandesh_send_rate_limits', {}).get('nodemgr') %}
+sandesh_send_rate_limit={{ config.sandesh_send_rate_limits.nodemgr }}
+{%- elif config.get('sandesh_send_rate_limits', {}).get('global') %}
+sandesh_send_rate_limit={{ config.sandesh_send_rate_limits.global }}
+{%- endif %}
[COLLECTOR]
server_list={% for member in config.analytics.members %}{{ member.host }}:8086 {% endfor %}
diff --git a/opencontrail/files/4.0/contrail-control-nodemgr.conf b/opencontrail/files/4.0/contrail-control-nodemgr.conf
index 2f77929..7ed15b9 100644
--- a/opencontrail/files/4.0/contrail-control-nodemgr.conf
+++ b/opencontrail/files/4.0/contrail-control-nodemgr.conf
@@ -1,3 +1,13 @@
{%- from "opencontrail/map.jinja" import control with context %}
+[DEFAULTS]
+# Sandesh send rate limit can be used to throttle system logs transmitted per
+# second. System logs are dropped if the sending rate is exceeded
+# sandesh_send_rate_limit=
+{%- if control.get('sandesh_send_rate_limits', {}).get('nodemgr') %}
+sandesh_send_rate_limit={{ control.sandesh_send_rate_limits.nodemgr }}
+{%- elif control.get('sandesh_send_rate_limits', {}).get('global') %}
+sandesh_send_rate_limit={{ control.sandesh_send_rate_limits.global }}
+{%- endif %}
+
[COLLECTOR]
server_list={% for member in control.analytics.members %}{{ member.host }}:8086 {% endfor %}
diff --git a/opencontrail/files/4.0/contrail-control.conf b/opencontrail/files/4.0/contrail-control.conf
index ed67242..4962e30 100644
--- a/opencontrail/files/4.0/contrail-control.conf
+++ b/opencontrail/files/4.0/contrail-control.conf
@@ -36,6 +36,11 @@
# Sandesh send rate limit can be used to throttle system logs transmitted per
# second. System logs are dropped if the sending rate is exceeded
# sandesh_send_rate_limit=
+{%- if control.get('sandesh_send_rate_limits', {}).get('control') %}
+sandesh_send_rate_limit={{ control.sandesh_send_rate_limits.control }}
+{%- elif control.get('sandesh_send_rate_limits', {}).get('global') %}
+sandesh_send_rate_limit={{ control.sandesh_send_rate_limits.global }}
+{%- endif %}
[CONFIGDB]
#AMQP related configs
diff --git a/opencontrail/files/4.0/contrail-device-manager.conf b/opencontrail/files/4.0/contrail-device-manager.conf
index caaae67..a654f9d 100644
--- a/opencontrail/files/4.0/contrail-device-manager.conf
+++ b/opencontrail/files/4.0/contrail-device-manager.conf
@@ -24,6 +24,11 @@
collectors = {% for member in config.analytics.members %}{{ member.host }}:8086 {% endfor %}
log_local=1
log_level=SYS_NOTICE
+{%- if config.get('sandesh_send_rate_limits', {}).get('device_manager') %}
+sandesh_send_rate_limit={{ config.sandesh_send_rate_limits.device_manager }}
+{%- elif config.get('sandesh_send_rate_limits', {}).get('global') %}
+sandesh_send_rate_limit={{ config.sandesh_send_rate_limits.global }}
+{%- endif %}
[SANDESH]
#sandesh_ssl_enable=False
diff --git a/opencontrail/files/4.0/contrail-dns.conf b/opencontrail/files/4.0/contrail-dns.conf
index 2517778..7a4ee8a 100644
--- a/opencontrail/files/4.0/contrail-dns.conf
+++ b/opencontrail/files/4.0/contrail-dns.conf
@@ -41,6 +41,11 @@
# Sandesh send rate limit can be used to throttle system logs transmitted per
# second. System logs are dropped if the sending rate is exceeded
# sandesh_send_rate_limit=
+{%- if control.get('sandesh_send_rate_limits', {}).get('dns') %}
+sandesh_send_rate_limit={{ control.sandesh_send_rate_limits.dns }}
+{%- elif control.get('sandesh_send_rate_limits', {}).get('global') %}
+sandesh_send_rate_limit={{ control.sandesh_send_rate_limits.global }}
+{%- endif %}
[CONFIGDB]
#AMQP related configs
diff --git a/opencontrail/files/4.0/contrail-query-engine.conf b/opencontrail/files/4.0/contrail-query-engine.conf
index 1e797bb..f147212 100644
--- a/opencontrail/files/4.0/contrail-query-engine.conf
+++ b/opencontrail/files/4.0/contrail-query-engine.conf
@@ -27,6 +27,11 @@
# Sandesh send rate limit can be used to throttle system logs transmitted per
# second. System logs are dropped if the sending rate is exceeded
# sandesh_send_rate_limit=
+{%- if collector.get('sandesh_send_rate_limits', {}).get('query_engine') %}
+sandesh_send_rate_limit={{ collector.sandesh_send_rate_limits.query_engine }}
+{%- elif collector.get('sandesh_send_rate_limits', {}).get('global') %}
+sandesh_send_rate_limit={{ collector.sandesh_send_rate_limits.global }}
+{%- endif %}
[REDIS]
port=6379
diff --git a/opencontrail/files/4.0/contrail-schema.conf b/opencontrail/files/4.0/contrail-schema.conf
index af77e9e..df61af7 100644
--- a/opencontrail/files/4.0/contrail-schema.conf
+++ b/opencontrail/files/4.0/contrail-schema.conf
@@ -25,6 +25,11 @@
rabbit_password={{config.message_queue.password}}
{%- endif %}
collectors = {% for member in config.analytics.members %}{{ member.host }}:8086 {% endfor %}
+{%- if config.get('sandesh_send_rate_limits', {}).get('schema') %}
+sandesh_send_rate_limit={{ config.sandesh_send_rate_limits.schema }}
+{%- elif config.get('sandesh_send_rate_limits', {}).get('global') %}
+sandesh_send_rate_limit={{ config.sandesh_send_rate_limits.global }}
+{%- endif %}
[SECURITY]
use_certs=False
diff --git a/opencontrail/files/4.0/contrail-snmp-collector.conf b/opencontrail/files/4.0/contrail-snmp-collector.conf
index 0d229a6..d99fdb9 100644
--- a/opencontrail/files/4.0/contrail-snmp-collector.conf
+++ b/opencontrail/files/4.0/contrail-snmp-collector.conf
@@ -8,6 +8,11 @@
log_file=/var/log/contrail/contrail-snmp-collector.log
zookeeper = {% for member in collector.database.members %}{{ member.host }}:2182{% if not loop.last %},{% endif %}{% endfor %}
#scan_frequency=600
+{%- if collector.get('sandesh_send_rate_limits', {}).get('snmp_collector') %}
+sandesh_send_rate_limit={{ collector.sandesh_send_rate_limits.snmp_collector }}
+{%- elif collector.get('sandesh_send_rate_limits', {}).get('global') %}
+sandesh_send_rate_limit={{ collector.sandesh_send_rate_limits.global }}
+{%- endif %}
[API_SERVER]
# List of api-servers in ip:port format separated by space
diff --git a/opencontrail/files/4.0/contrail-svc-monitor.conf b/opencontrail/files/4.0/contrail-svc-monitor.conf
index d1e2dcc..d23c6ad 100644
--- a/opencontrail/files/4.0/contrail-svc-monitor.conf
+++ b/opencontrail/files/4.0/contrail-svc-monitor.conf
@@ -27,6 +27,11 @@
rabbit_password={{config.message_queue.password}}
{%- endif %}
collectors={% for member in config.analytics.members %}{{ member.host }}:8086 {% endfor %}
+{%- if config.get('sandesh_send_rate_limits', {}).get('svc_monitor') %}
+sandesh_send_rate_limit={{ config.sandesh_send_rate_limits.svc_monitor }}
+{%- elif config.get('sandesh_send_rate_limits', {}).get('global') %}
+sandesh_send_rate_limit={{ config.sandesh_send_rate_limits.global }}
+{%- endif %}
[SCHEDULER]
# Analytics server list used to get vrouter status and schedule service instance
diff --git a/opencontrail/files/4.0/contrail-topology.conf b/opencontrail/files/4.0/contrail-topology.conf
index f146173..9492b3c 100644
--- a/opencontrail/files/4.0/contrail-topology.conf
+++ b/opencontrail/files/4.0/contrail-topology.conf
@@ -14,6 +14,12 @@
#zookeeper=127.0.0.1:2182
+{%- if collector.get('sandesh_send_rate_limits', {}).get('topology') %}
+sandesh_send_rate_limit={{ collector.sandesh_send_rate_limits.topology }}
+{%- elif collector.get('sandesh_send_rate_limits', {}).get('global') %}
+sandesh_send_rate_limit={{ collector.sandesh_send_rate_limits.global }}
+{%- endif %}
+
[API_SERVER]
# List of api-servers in ip:port format separated by space
api_server_list={% for member in collector.config.members %}{{ member.host }}:9100 {% endfor %}
diff --git a/opencontrail/files/4.0/contrail-vrouter-agent.conf b/opencontrail/files/4.0/contrail-vrouter-agent.conf
index a0a742b..397c35d 100644
--- a/opencontrail/files/4.0/contrail-vrouter-agent.conf
+++ b/opencontrail/files/4.0/contrail-vrouter-agent.conf
@@ -86,6 +86,11 @@
# Sandesh send rate limit can be used to throttle system logs transmitted per
# second. System logs are dropped if the sending rate is exceeded
# sandesh_send_rate_limit=
+{%- if compute.get('sandesh_send_rate_limits', {}).get('agent') %}
+sandesh_send_rate_limit={{ compute.sandesh_send_rate_limits.agent }}
+{%- elif compute.get('sandesh_send_rate_limits', {}).get('global') %}
+sandesh_send_rate_limit={{ compute.sandesh_send_rate_limits.global }}
+{%- endif %}
# Enable/Disable SSL based XMPP Authentication
{%- if compute.xmpp.tls.enabled %}
diff --git a/opencontrail/files/4.0/contrail-vrouter-nodemgr.conf b/opencontrail/files/4.0/contrail-vrouter-nodemgr.conf
index 952a2a5..9ae7b81 100644
--- a/opencontrail/files/4.0/contrail-vrouter-nodemgr.conf
+++ b/opencontrail/files/4.0/contrail-vrouter-nodemgr.conf
@@ -1,3 +1,13 @@
{%- from "opencontrail/map.jinja" import compute with context %}
+[DEFAULTS]
+# Sandesh send rate limit can be used to throttle system logs transmitted per
+# second. System logs are dropped if the sending rate is exceeded
+# sandesh_send_rate_limit=
+{%- if compute.get('sandesh_send_rate_limits', {}).get('nodemgr') %}
+sandesh_send_rate_limit={{ compute.sandesh_send_rate_limits.nodemgr }}
+{%- elif compute.get('sandesh_send_rate_limits', {}).get('global') %}
+sandesh_send_rate_limit={{ compute.sandesh_send_rate_limits.global }}
+{%- endif %}
+
[COLLECTOR]
server_list = {% for member in compute.collector.members %}{{ member.host }}:8086 {% endfor %}