Stacklight (#71)

* Stacklight integration

* Round 2

* Variable service_name is missing for systemd file

* preserve_data and ticker_interval are not strings

preserve_data is a boolean, and ticker_interval is a number, so their values
shouldn't have quotes.

* Use "ignore missing" with the j2 include statement

* Added cache dir

* Use module_dir instead of module_directory

This fixes a bug where module_directory is used as the variable name instead of
module_dir.

* Use the proper module directory

The stacklight module dir is /usr/share/lma_collector/common, not
/usr/share/lma_collector_modules. This fixes it.

* Add the extra_fields.lua module

This commit adds the extra_fields Lua module. The extra fields table defined in
this module is empty right now. Eventually, this file will be a Jinja2 template
and the content of the extra fields table will be generated based on the user
configuration.

* Regex encoder fix

* Fix the decoder configuration

This commit uses proper decoder names in heka/meta/heka.yml. It also removes
the aggregator input for now, because it does not have an associated decoder.

* Make Heka send metrics to InfluxDB

* Add HTTP metrics filter to log_collector

* Add logs counter filter to log_collector

* Templatize extra_fields.lua file

* Make InfluxDB time precision configurable

* Configure Elasticsearch output through Pillar

* Use influxdb_time_precision for InfluxDB output

This uses influxdb_time_precision set on metric_collector for configuring the
time precision in the InfluxDB output. This is to use just one parameter for
both the InfluxDB accumulator filter and InfluxDB output.

* Increase maximum open files limit to 102400

* Add alarming support

* Revert "[WIP] Add alarming support"

* Remove the aggregator output for now

This removes the aggregator output for now, as the aggregator doesn't work for
now. This is to avoid output errors in Heka.

* Do not place Heka logs in /var/log/upstart

With this commit all the Heka logs are sent to /var/log/<heka_service>.log.
Previously, stdout was sent to /var/log/<heka_service>.log and stderr was sent
to /var/log/upstart/<heka_service>.log, which was confusing to the operator.

* Remove http check input plugin

Because it is not used anymore.

* Add alarming support

* Make the aggregator load heka/meta/heka.yml

Currently _service.sls does not load aggregator metadata from
heka/meta/heka.yml. This commit fixes that.

* Use filter_by to merge node grains data

* Make the output/tcp.toml template extendable

* Add an aggregator.toml output template

This template extends the tcp.toml output template.

* Add generic timezone support to decoders

This change add a new parameter 'adjust_timezone' for the sandbox
decoder. This parameter should be set to true when the data to be
decoded doesn't contain the proper timezone information.

* Add a run_lua_tests.sh script

This script will be used to run the Lua tests (yet to be added).

To run the script:

    cd tests
    ./run_lua_tests.sh

* Copy Lua tests from fuel-plugin-lma-collector

* Fix the afd tests

* Fix the gse tests

* Add aggregator config to support metadata

* Fix the definition of the remote_collector service

This change removes unneeded plugins and adds the ones that are
otherwise required.

* Fix state dependency

* Add monitoring of the Heka processes

* Set influxdb_time_precision in aggregator class

* Disable the heka service completely

Without this patch `service heka status` reports that the heka service is
running. For example:

root@ctl01:/etc/init.d# /etc/init.d/heka status
 * hekad is running

* Define the highest_severity policy

* Generate the gse_policies Lua module

* Generate gse topology module for each alarm cluster

* Generate gse filter toml for each cluster alarm

* Adapt GSE Lua code

* Remove gse cluster_field parameter

This parameter is not needed anymore. Heka's message_matchers are now used to
match input messages.

* Support dimensions in gse metrics

* Do not rely on pacemaker_local_resource_active

* Define the majority_of_members policy

* Define the availability_of_members policy

* Configure outputs in support metadata

* Fix bug in map.jinja

Fix a bug in map.jinja where the filter_by for the metric_collector modified
the influxdb_defaults dict re-used for the remote_collector. The filter_by
function does deep merges, so some caution is required.

* Cleaning useless default map keys

* Make remote collector send only afd metrics to influx

* Add aggregator output to remote collector

* Extend collectd decoder to support vrrp metrics

* Update map.jinja

* Update collectd decoder to parse ntpd metrics

* Redefine alerting property

The alerting property can be one of 'disabled', 'enabled' or
'enabled_with_notification'

* Fix the gse_policies structure

The structure of the generated gse_policies.lua file is not correct. This
commit fixes that.

* Add Nagios output for metric_collector

The patch embeds the Lua sandbox encoder for Nagios.

* Add Nagios output for the aggregator

* Send only alarm-related data to mine

* Fix the grains_for_mine function

* Fix flake8 in heka_alarming.py

* Configure Hekad poolsize by pillar data

The poolsize must be increased depending on the number of filters.
Typically, the metric_collector on controller nodes and the aggregator on
monitoring node(s) should probably use poolsize=200.

* Make Heka service watch Lua dir

In this way the service will restart when the content of
/usr/share/lma_collector changes.

* Enable collection of notifications

* Add missing hostname variable in GSE code

* Add a log decoder for Galera

* Simplify message matchers

This removes the "Field[aggregator] == NIL" part in the Heka message matchers.

We used to use a scribbler decoder to tag input messages coming in through the
aggregator input. We now have a dedicated Heka "aggregator" instance, so this
mechanism is not necessary anymore.

* Update collectd decoder for nginx metrics

* Return an err message when set_member_status fails

With this commit an explicit error message is displayed in the Heka logs when
set_member_status fails because the cluster has "group_by" set to "hostname"
and an input message with no "hostname" field is received.

This addresses a comment from @SwannCroiset in #51.

* Add contrail log parsers

* Fix the heka grains for the aggregator/remote_collector

Previously, the heka salt grains of the node running aggregator/remote_collector
get all the metric_collector alarms from all nodes (/etc/salt/grains.d/heka).
The resulting mines data is then wrong for the monitoring node, while that
situtation fortunately has no impact regarding metric_collector alarm
configurations, the Nagios service leverging mine data get a wrong list of
alarms for the monitoring node.

This patch fixes the issue with minimal changes but it appears that the logic
behind _service.sls state is not optimal and become hard to understand.
This state is executed several times with different contexts for every heka
'server' types and is not indempotent, indeed the /etc/salt/grains.d/heka file
content is different between 'local' servers (metric|log)_collector and
'remote' servers remote_collector|aggregator.

* Fix issue in lma_alarm.lua template

* Add a log decoder for GlusterFS

* Fix collectd Lua decoder for system metrics

The regression has been introduced by 74ad71d41.

* Update collectd decoder for disk metrics

The disk plugin shipping with the 5.5. version of collectd (installed on
Xenial) provides new metrics: disk_io_time and disk_weighted_io_time.

* Use a dimension key for the Nagios host displaying alarm clusters

* Add redis log parser

* Add zookeeper log parser

* Add cassandra log parser

* Set actual swap_size in collectd decoder

Salt does not create Swap-related grains, but the "ps" module has
a "swap_memory" function that can be used to get Swap data. This commit
uses that function to set swap_size in the collectd decoder.

* Send annotations to InfluxDB

* Add ifmap log parser

* Support remote_collector and aggregator in cluster

When deployed in a cluster, the remote_collector and aggregator
services are only started when the node holds the virtual IP address.

* Add an os_telemetry_collector service

os_telemetry_collector implements reading of Сeilometer samples
from RabbitMQ and pulling them to InfluxDB (samples) and
ElasticSearch (resources)

* heka server role, backward compat
diff --git a/tests/lua/test_value_matching.lua b/tests/lua/test_value_matching.lua
new file mode 100644
index 0000000..3142fb5
--- /dev/null
+++ b/tests/lua/test_value_matching.lua
@@ -0,0 +1,229 @@
+-- Copyright 2015 Mirantis, Inc.
+--
+-- Licensed under the Apache License, Version 2.0 (the "License");
+-- you may not use this file except in compliance with the License.
+-- You may obtain a copy of the License at
+--
+--     http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+
+EXPORT_ASSERT_TO_GLOBALS=true
+require('luaunit')
+package.path = package.path .. ";../heka/files/lua/common/?.lua;lua/mocks/?.lua"
+local M = require('value_matching')
+
+TestValueMatching = {}
+
+function TestValueMatching:test_simple_matching()
+    local tests = {
+        {'/var/log',        '/var/log'},
+        {'== /var/log',     '/var/log'},
+        {'==/var/log',      '/var/log'},
+        {'==/var/log ',      '/var/log'},
+        {'==\t/var/log',    '/var/log'},
+        {'=="/var/log"',    '/var/log'},
+        {'== "/var/log"',   '/var/log'},
+        {'== " /var/log"',  ' /var/log'},
+        {'== "/var/log "',  '/var/log '},
+        {'== " /var/log "', ' /var/log '},
+        {8,                 8},
+        {8,                '8'},
+        {"9",               "9"},
+        {"==10",            " 10"},
+        {"10",              10},
+        {"== 10",           " 10"},
+        {"== 10.0",         " 10.0"},
+        {"== -10.01",       " -10.01"},
+        {"== 10 ",          " 10 "},
+        {' <=11',           '-11'},
+        {"!= -12",          42},
+        {"!= 12",           42},
+        {" > 13",            42},
+        {">= 13",           13},
+        {">= -13",           42},
+        {"< 14",            -0},
+        {"<= 14 ",           0},
+        {"<= 14",           "14"},
+    }
+    local r
+    for _, v in ipairs(tests) do
+        local exp, value = v[1], v[2]
+        local m = M.new(exp)
+        r = m:matches(value)
+        assertTrue(r)
+    end
+end
+
+function TestValueMatching:test_simple_not_matching()
+    local tests = {
+        {'/var/log',       '/var/log/mysql'},
+        {'== "/var/log"'   , '/var/log '},
+        {'"/var/log"',     '/var/log '},
+        {'"/var/log "',    '/var/log'},
+        {'nova-api',       'nova-compute'},
+        {'== /var/log',    '/var/log/mysql'},
+        {'==/var/log',     '/var/log/mysql'},
+        {'!=/var/log',     '/var/log'},
+        {'!= /var/log',    '/var/log'},
+        {'>10',            '5'},
+        {'> 10',           '5 '},
+        {' <11',           '11'},
+        {' >=11',          '-11'},
+        {' >=11 && <= 42', '-11'},
+        {' >=11 || == 42', '-11'},
+    }
+
+    for _, v in ipairs(tests) do
+        local exp, value = v[1], v[2]
+        local m = M.new(exp)
+        r = m:matches(value)
+        assertFalse(r)
+    end
+end
+
+function TestValueMatching:test_string_matching()
+    local tests = {
+        {'== "foo.bar"', "foo.bar", true},
+        {'== foo.bar', "foo.bar", true},
+        {'== foo.bar ', "foo.bar", true},
+        {'== foo || bar', "bar", true},
+        {'== foo || bar', "foo", true},
+        {'== foo || bar', "??", false},
+        {'!= foo || != bar', "42", true},
+    }
+
+    for _, v in ipairs(tests) do
+        local exp, value, expected = v[1], v[2], v[3]
+        local m = M.new(exp)
+        r = m:matches(value)
+        assertEquals(r, expected)
+    end
+
+end
+
+function TestValueMatching:test_invalid_expression()
+    local tests = {
+        '&& 1 && 1',
+        ' && 1',
+        '|| == 1',
+        '&& != 12',
+        ' ',
+        '   ',
+        '\t',
+        '',
+        nil,
+    }
+    for _, exp in ipairs(tests) do
+        assertError(M.new, exp)
+    end
+end
+
+function TestValueMatching:test_range_matching()
+    local tests = {
+        {'>= 200 && < 300', 200, true},
+        {'>=200&&<300'    , 200, true},
+        {' >=200&&<300'   , 200, true},
+        {'>= 200 && < 300', 204, true},
+        {'>= 200 && < 300', 300, false},
+        {'>= 200 && < 300', 42,  false},
+        {'>= 200 && < 300', 0,  false},
+    }
+
+    for _, v in ipairs(tests) do
+        local exp, value, expected = v[1], v[2], v[3]
+        local m = M.new(exp)
+        r = m:matches(value)
+        assertEquals(r, expected)
+    end
+end
+
+function TestValueMatching:test_wrong_data()
+    local tests = {
+        {'>= 200 && < 300', "foo", false},
+        {'>= 200 && < 300', ""   , false},
+        {'== 200'         , "bar", false},
+        {'== foo'         , "10" , false},
+        {'!= foo'         , " 10", true},
+    }
+    for _, v in ipairs(tests) do
+        local exp, value, expected = v[1], v[2], v[3]
+        local m = M.new(exp)
+        r = m:matches(value)
+        assertEquals(r, expected)
+    end
+end
+
+function TestValueMatching:test_precedence()
+    local tests = {
+        {'>= 200 && < 300 || >500', "200", true},
+        {'>= 200 && < 300 || >500', "501", true},
+        {'>= 200 && < 300 || >=500', "500", true},
+        {'>400 || >= 200 && < 300', "500", true},
+        {'>=300 && <500 || >= 200 && < 300', "300", true},
+        {'>=300 && <500 || >= 200 && < 300', "500", false},
+    }
+
+    for _, v in ipairs(tests) do
+        local exp, value, expected = v[1], v[2], v[3]
+        local m = M.new(exp)
+        r = m:matches(value)
+        assertEquals(r, expected)
+    end
+end
+
+function TestValueMatching:test_pattern_matching()
+    local tests = {
+        {'=~ /var/lib/ceph/osd/ceph%-%d+', "/var/lib/ceph/osd/ceph-1", true},
+        {'=~ /var/lib/ceph/osd/ceph%-%d+', "/var/lib/ceph/osd/ceph-42", true},
+        {'=~ ^/var/lib/ceph/osd/ceph%-%d+$', "/var/lib/ceph/osd/ceph-42", true},
+        {'=~ "/var/lib/ceph/osd/ceph%-%d+"', "/var/lib/ceph/osd/ceph-42", true},
+        {'=~ "ceph%-%d+"', "/var/lib/ceph/osd/ceph-42", true},
+        {'=~ "/var/lib/ceph/osd/ceph%-%d+$"', "/var/lib/ceph/osd/ceph-42 ", false}, -- trailing space
+        {'=~ /var/lib/ceph/osd/ceph%-%d+', "/var/log", false},
+        {'=~ /var/lib/ceph/osd/ceph%-%d+ || foo', "/var/lib/ceph/osd/ceph-1", true},
+        {'=~ "foo||bar" || foo', "foo||bar", true},
+        {'=~ "foo||bar" || foo', "foo", true},
+        {'=~ "foo&&bar" || foo', "foo&&bar", true},
+        {'=~ "foo&&bar" || foo', "foo", true},
+        {'=~ bar && /var/lib/ceph/osd/ceph%-%d+', "/var/lib/ceph/osd/ceph-1", false},
+        {'=~ -', "-", true},
+        {'=~ %-', "-", true},
+        {'!~ /var/lib/ceph/osd/ceph', "/var/log", true},
+        {'!~ /var/lib/ceph/osd/ceph%-%d+', "/var/log", true},
+        {'!~ .+osd%-%d+', "/var/log", true},
+        {'!~ osd%-%d+', "/var/log", true},
+        --{'=~ [', "[", true},
+    }
+
+    for _, v in ipairs(tests) do
+        local exp, value, expected = v[1], v[2], v[3]
+        local m = M.new(exp)
+        r = m:matches(value)
+        assertEquals(r, expected)
+    end
+end
+
+function TestValueMatching:test_wrong_patterns_never_match()
+    -- These patterns raise errors like:
+    -- malformed pattern (missing ']')
+    local tests = {
+        {'=~ [', "[", false},
+        {'!~ [', "[", false},
+    }
+
+    for _, v in ipairs(tests) do
+        local exp, value, expected = v[1], v[2], v[3]
+        local m = M.new(exp)
+        r = m:matches(value)
+        assertEquals(r, expected)
+    end
+end
+
+lu = LuaUnit
+lu:setVerbosity( 1 )
+os.exit( lu:run() )