Stacklight (#71)
* Stacklight integration
* Round 2
* Variable service_name is missing for systemd file
* preserve_data and ticker_interval are not strings
preserve_data is a boolean, and ticker_interval is a number, so their values
shouldn't have quotes.
* Use "ignore missing" with the j2 include statement
* Added cache dir
* Use module_dir instead of module_directory
This fixes a bug where module_directory is used as the variable name instead of
module_dir.
* Use the proper module directory
The stacklight module dir is /usr/share/lma_collector/common, not
/usr/share/lma_collector_modules. This fixes it.
* Add the extra_fields.lua module
This commit adds the extra_fields Lua module. The extra fields table defined in
this module is empty right now. Eventually, this file will be a Jinja2 template
and the content of the extra fields table will be generated based on the user
configuration.
* Regex encoder fix
* Fix the decoder configuration
This commit uses proper decoder names in heka/meta/heka.yml. It also removes
the aggregator input for now, because it does not have an associated decoder.
* Make Heka send metrics to InfluxDB
* Add HTTP metrics filter to log_collector
* Add logs counter filter to log_collector
* Templatize extra_fields.lua file
* Make InfluxDB time precision configurable
* Configure Elasticsearch output through Pillar
* Use influxdb_time_precision for InfluxDB output
This uses influxdb_time_precision set on metric_collector for configuring the
time precision in the InfluxDB output. This is to use just one parameter for
both the InfluxDB accumulator filter and InfluxDB output.
* Increase maximum open files limit to 102400
* Add alarming support
* Revert "[WIP] Add alarming support"
* Remove the aggregator output for now
This removes the aggregator output for now, as the aggregator doesn't work for
now. This is to avoid output errors in Heka.
* Do not place Heka logs in /var/log/upstart
With this commit all the Heka logs are sent to /var/log/<heka_service>.log.
Previously, stdout was sent to /var/log/<heka_service>.log and stderr was sent
to /var/log/upstart/<heka_service>.log, which was confusing to the operator.
* Remove http check input plugin
Because it is not used anymore.
* Add alarming support
* Make the aggregator load heka/meta/heka.yml
Currently _service.sls does not load aggregator metadata from
heka/meta/heka.yml. This commit fixes that.
* Use filter_by to merge node grains data
* Make the output/tcp.toml template extendable
* Add an aggregator.toml output template
This template extends the tcp.toml output template.
* Add generic timezone support to decoders
This change add a new parameter 'adjust_timezone' for the sandbox
decoder. This parameter should be set to true when the data to be
decoded doesn't contain the proper timezone information.
* Add a run_lua_tests.sh script
This script will be used to run the Lua tests (yet to be added).
To run the script:
cd tests
./run_lua_tests.sh
* Copy Lua tests from fuel-plugin-lma-collector
* Fix the afd tests
* Fix the gse tests
* Add aggregator config to support metadata
* Fix the definition of the remote_collector service
This change removes unneeded plugins and adds the ones that are
otherwise required.
* Fix state dependency
* Add monitoring of the Heka processes
* Set influxdb_time_precision in aggregator class
* Disable the heka service completely
Without this patch `service heka status` reports that the heka service is
running. For example:
root@ctl01:/etc/init.d# /etc/init.d/heka status
* hekad is running
* Define the highest_severity policy
* Generate the gse_policies Lua module
* Generate gse topology module for each alarm cluster
* Generate gse filter toml for each cluster alarm
* Adapt GSE Lua code
* Remove gse cluster_field parameter
This parameter is not needed anymore. Heka's message_matchers are now used to
match input messages.
* Support dimensions in gse metrics
* Do not rely on pacemaker_local_resource_active
* Define the majority_of_members policy
* Define the availability_of_members policy
* Configure outputs in support metadata
* Fix bug in map.jinja
Fix a bug in map.jinja where the filter_by for the metric_collector modified
the influxdb_defaults dict re-used for the remote_collector. The filter_by
function does deep merges, so some caution is required.
* Cleaning useless default map keys
* Make remote collector send only afd metrics to influx
* Add aggregator output to remote collector
* Extend collectd decoder to support vrrp metrics
* Update map.jinja
* Update collectd decoder to parse ntpd metrics
* Redefine alerting property
The alerting property can be one of 'disabled', 'enabled' or
'enabled_with_notification'
* Fix the gse_policies structure
The structure of the generated gse_policies.lua file is not correct. This
commit fixes that.
* Add Nagios output for metric_collector
The patch embeds the Lua sandbox encoder for Nagios.
* Add Nagios output for the aggregator
* Send only alarm-related data to mine
* Fix the grains_for_mine function
* Fix flake8 in heka_alarming.py
* Configure Hekad poolsize by pillar data
The poolsize must be increased depending on the number of filters.
Typically, the metric_collector on controller nodes and the aggregator on
monitoring node(s) should probably use poolsize=200.
* Make Heka service watch Lua dir
In this way the service will restart when the content of
/usr/share/lma_collector changes.
* Enable collection of notifications
* Add missing hostname variable in GSE code
* Add a log decoder for Galera
* Simplify message matchers
This removes the "Field[aggregator] == NIL" part in the Heka message matchers.
We used to use a scribbler decoder to tag input messages coming in through the
aggregator input. We now have a dedicated Heka "aggregator" instance, so this
mechanism is not necessary anymore.
* Update collectd decoder for nginx metrics
* Return an err message when set_member_status fails
With this commit an explicit error message is displayed in the Heka logs when
set_member_status fails because the cluster has "group_by" set to "hostname"
and an input message with no "hostname" field is received.
This addresses a comment from @SwannCroiset in #51.
* Add contrail log parsers
* Fix the heka grains for the aggregator/remote_collector
Previously, the heka salt grains of the node running aggregator/remote_collector
get all the metric_collector alarms from all nodes (/etc/salt/grains.d/heka).
The resulting mines data is then wrong for the monitoring node, while that
situtation fortunately has no impact regarding metric_collector alarm
configurations, the Nagios service leverging mine data get a wrong list of
alarms for the monitoring node.
This patch fixes the issue with minimal changes but it appears that the logic
behind _service.sls state is not optimal and become hard to understand.
This state is executed several times with different contexts for every heka
'server' types and is not indempotent, indeed the /etc/salt/grains.d/heka file
content is different between 'local' servers (metric|log)_collector and
'remote' servers remote_collector|aggregator.
* Fix issue in lma_alarm.lua template
* Add a log decoder for GlusterFS
* Fix collectd Lua decoder for system metrics
The regression has been introduced by 74ad71d41.
* Update collectd decoder for disk metrics
The disk plugin shipping with the 5.5. version of collectd (installed on
Xenial) provides new metrics: disk_io_time and disk_weighted_io_time.
* Use a dimension key for the Nagios host displaying alarm clusters
* Add redis log parser
* Add zookeeper log parser
* Add cassandra log parser
* Set actual swap_size in collectd decoder
Salt does not create Swap-related grains, but the "ps" module has
a "swap_memory" function that can be used to get Swap data. This commit
uses that function to set swap_size in the collectd decoder.
* Send annotations to InfluxDB
* Add ifmap log parser
* Support remote_collector and aggregator in cluster
When deployed in a cluster, the remote_collector and aggregator
services are only started when the node holds the virtual IP address.
* Add an os_telemetry_collector service
os_telemetry_collector implements reading of Сeilometer samples
from RabbitMQ and pulling them to InfluxDB (samples) and
ElasticSearch (resources)
* heka server role, backward compat
diff --git a/tests/lua/test_value_matching.lua b/tests/lua/test_value_matching.lua
new file mode 100644
index 0000000..3142fb5
--- /dev/null
+++ b/tests/lua/test_value_matching.lua
@@ -0,0 +1,229 @@
+-- Copyright 2015 Mirantis, Inc.
+--
+-- Licensed under the Apache License, Version 2.0 (the "License");
+-- you may not use this file except in compliance with the License.
+-- You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+
+EXPORT_ASSERT_TO_GLOBALS=true
+require('luaunit')
+package.path = package.path .. ";../heka/files/lua/common/?.lua;lua/mocks/?.lua"
+local M = require('value_matching')
+
+TestValueMatching = {}
+
+function TestValueMatching:test_simple_matching()
+ local tests = {
+ {'/var/log', '/var/log'},
+ {'== /var/log', '/var/log'},
+ {'==/var/log', '/var/log'},
+ {'==/var/log ', '/var/log'},
+ {'==\t/var/log', '/var/log'},
+ {'=="/var/log"', '/var/log'},
+ {'== "/var/log"', '/var/log'},
+ {'== " /var/log"', ' /var/log'},
+ {'== "/var/log "', '/var/log '},
+ {'== " /var/log "', ' /var/log '},
+ {8, 8},
+ {8, '8'},
+ {"9", "9"},
+ {"==10", " 10"},
+ {"10", 10},
+ {"== 10", " 10"},
+ {"== 10.0", " 10.0"},
+ {"== -10.01", " -10.01"},
+ {"== 10 ", " 10 "},
+ {' <=11', '-11'},
+ {"!= -12", 42},
+ {"!= 12", 42},
+ {" > 13", 42},
+ {">= 13", 13},
+ {">= -13", 42},
+ {"< 14", -0},
+ {"<= 14 ", 0},
+ {"<= 14", "14"},
+ }
+ local r
+ for _, v in ipairs(tests) do
+ local exp, value = v[1], v[2]
+ local m = M.new(exp)
+ r = m:matches(value)
+ assertTrue(r)
+ end
+end
+
+function TestValueMatching:test_simple_not_matching()
+ local tests = {
+ {'/var/log', '/var/log/mysql'},
+ {'== "/var/log"' , '/var/log '},
+ {'"/var/log"', '/var/log '},
+ {'"/var/log "', '/var/log'},
+ {'nova-api', 'nova-compute'},
+ {'== /var/log', '/var/log/mysql'},
+ {'==/var/log', '/var/log/mysql'},
+ {'!=/var/log', '/var/log'},
+ {'!= /var/log', '/var/log'},
+ {'>10', '5'},
+ {'> 10', '5 '},
+ {' <11', '11'},
+ {' >=11', '-11'},
+ {' >=11 && <= 42', '-11'},
+ {' >=11 || == 42', '-11'},
+ }
+
+ for _, v in ipairs(tests) do
+ local exp, value = v[1], v[2]
+ local m = M.new(exp)
+ r = m:matches(value)
+ assertFalse(r)
+ end
+end
+
+function TestValueMatching:test_string_matching()
+ local tests = {
+ {'== "foo.bar"', "foo.bar", true},
+ {'== foo.bar', "foo.bar", true},
+ {'== foo.bar ', "foo.bar", true},
+ {'== foo || bar', "bar", true},
+ {'== foo || bar', "foo", true},
+ {'== foo || bar', "??", false},
+ {'!= foo || != bar', "42", true},
+ }
+
+ for _, v in ipairs(tests) do
+ local exp, value, expected = v[1], v[2], v[3]
+ local m = M.new(exp)
+ r = m:matches(value)
+ assertEquals(r, expected)
+ end
+
+end
+
+function TestValueMatching:test_invalid_expression()
+ local tests = {
+ '&& 1 && 1',
+ ' && 1',
+ '|| == 1',
+ '&& != 12',
+ ' ',
+ ' ',
+ '\t',
+ '',
+ nil,
+ }
+ for _, exp in ipairs(tests) do
+ assertError(M.new, exp)
+ end
+end
+
+function TestValueMatching:test_range_matching()
+ local tests = {
+ {'>= 200 && < 300', 200, true},
+ {'>=200&&<300' , 200, true},
+ {' >=200&&<300' , 200, true},
+ {'>= 200 && < 300', 204, true},
+ {'>= 200 && < 300', 300, false},
+ {'>= 200 && < 300', 42, false},
+ {'>= 200 && < 300', 0, false},
+ }
+
+ for _, v in ipairs(tests) do
+ local exp, value, expected = v[1], v[2], v[3]
+ local m = M.new(exp)
+ r = m:matches(value)
+ assertEquals(r, expected)
+ end
+end
+
+function TestValueMatching:test_wrong_data()
+ local tests = {
+ {'>= 200 && < 300', "foo", false},
+ {'>= 200 && < 300', "" , false},
+ {'== 200' , "bar", false},
+ {'== foo' , "10" , false},
+ {'!= foo' , " 10", true},
+ }
+ for _, v in ipairs(tests) do
+ local exp, value, expected = v[1], v[2], v[3]
+ local m = M.new(exp)
+ r = m:matches(value)
+ assertEquals(r, expected)
+ end
+end
+
+function TestValueMatching:test_precedence()
+ local tests = {
+ {'>= 200 && < 300 || >500', "200", true},
+ {'>= 200 && < 300 || >500', "501", true},
+ {'>= 200 && < 300 || >=500', "500", true},
+ {'>400 || >= 200 && < 300', "500", true},
+ {'>=300 && <500 || >= 200 && < 300', "300", true},
+ {'>=300 && <500 || >= 200 && < 300', "500", false},
+ }
+
+ for _, v in ipairs(tests) do
+ local exp, value, expected = v[1], v[2], v[3]
+ local m = M.new(exp)
+ r = m:matches(value)
+ assertEquals(r, expected)
+ end
+end
+
+function TestValueMatching:test_pattern_matching()
+ local tests = {
+ {'=~ /var/lib/ceph/osd/ceph%-%d+', "/var/lib/ceph/osd/ceph-1", true},
+ {'=~ /var/lib/ceph/osd/ceph%-%d+', "/var/lib/ceph/osd/ceph-42", true},
+ {'=~ ^/var/lib/ceph/osd/ceph%-%d+$', "/var/lib/ceph/osd/ceph-42", true},
+ {'=~ "/var/lib/ceph/osd/ceph%-%d+"', "/var/lib/ceph/osd/ceph-42", true},
+ {'=~ "ceph%-%d+"', "/var/lib/ceph/osd/ceph-42", true},
+ {'=~ "/var/lib/ceph/osd/ceph%-%d+$"', "/var/lib/ceph/osd/ceph-42 ", false}, -- trailing space
+ {'=~ /var/lib/ceph/osd/ceph%-%d+', "/var/log", false},
+ {'=~ /var/lib/ceph/osd/ceph%-%d+ || foo', "/var/lib/ceph/osd/ceph-1", true},
+ {'=~ "foo||bar" || foo', "foo||bar", true},
+ {'=~ "foo||bar" || foo', "foo", true},
+ {'=~ "foo&&bar" || foo', "foo&&bar", true},
+ {'=~ "foo&&bar" || foo', "foo", true},
+ {'=~ bar && /var/lib/ceph/osd/ceph%-%d+', "/var/lib/ceph/osd/ceph-1", false},
+ {'=~ -', "-", true},
+ {'=~ %-', "-", true},
+ {'!~ /var/lib/ceph/osd/ceph', "/var/log", true},
+ {'!~ /var/lib/ceph/osd/ceph%-%d+', "/var/log", true},
+ {'!~ .+osd%-%d+', "/var/log", true},
+ {'!~ osd%-%d+', "/var/log", true},
+ --{'=~ [', "[", true},
+ }
+
+ for _, v in ipairs(tests) do
+ local exp, value, expected = v[1], v[2], v[3]
+ local m = M.new(exp)
+ r = m:matches(value)
+ assertEquals(r, expected)
+ end
+end
+
+function TestValueMatching:test_wrong_patterns_never_match()
+ -- These patterns raise errors like:
+ -- malformed pattern (missing ']')
+ local tests = {
+ {'=~ [', "[", false},
+ {'!~ [', "[", false},
+ }
+
+ for _, v in ipairs(tests) do
+ local exp, value, expected = v[1], v[2], v[3]
+ local m = M.new(exp)
+ r = m:matches(value)
+ assertEquals(r, expected)
+ end
+end
+
+lu = LuaUnit
+lu:setVerbosity( 1 )
+os.exit( lu:run() )