Gitiles
Code Review Sign In
gerrit.mcp.mirantis.com / salt-formulas / heka / 1002cd99f784e6846b987f63a507a146bc4186b9 / . / README.rst
blob: d3b67118ee4d666081871a0c4d5621679a3f5177 [file] [log] [blame]
Jakub Pavlike7d12cd2015-09-03 19:02:45 +0200[diff] [blame]1
2============
3Heka Formula
4============
5
6Heka is an open source stream processing software system developed by Mozilla. Heka is a Swiss Army Knife type tool for data processing
7
8Sample pillars
9==============
10
jan kaufman1002cd92015-09-16 16:30:48 +0200[diff] [blame^]11Basic log shipper streaming decoded rsyslog's logfiles using amqp broker as transport.
12From every message there is one amqp message and it's also logged to heka's logfile in RST format.
13
Jakub Pavlike7d12cd2015-09-03 19:02:45 +0200[diff] [blame]14.. code-block:: yaml
15
jan kaufman1002cd92015-09-16 16:30:48 +0200[diff] [blame^]16
Jakub Pavlike7d12cd2015-09-03 19:02:45 +0200[diff] [blame]17 heka:
jan kaufman1002cd92015-09-16 16:30:48 +0200[diff] [blame^]18 server:
Jakub Pavlike7d12cd2015-09-03 19:02:45 +0200[diff] [blame]19 enabled: true
jan kaufman1002cd92015-09-16 16:30:48 +0200[diff] [blame^]20 input:
21 rsyslog-syslog:
22 engine: logstreamer
23 log_directory: /var/log
24 file_match: syslog\.?(?P<Index>\d+)?(.gz)?
25 decoder: RsyslogDecoder
26 priority: ["^Index"]
27 rsyslog-auth:
28 engine: logstreamer
29 log_directory: /var/log
30 file_match: auth\.log\.?(?P<Index>\d+)?(.gz)?
31 decoder: RsyslogDecoder
32 priority: ["^Index"]
33 decoder:
34 rsyslog:
35 engine: rsyslog
36 template: %TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n
37 hostname_keep: TRUE
38 tz = Europe/Prague
Jakub Pavlik6014f912015-09-03 19:05:05 +0200[diff] [blame]39 output:
jan kaufman1002cd92015-09-16 16:30:48 +0200[diff] [blame^]40 rabbitmq:
41 engine: amqp
Jakub Pavlik6014f912015-09-03 19:05:05 +0200[diff] [blame]42 host: localhost
jan kaufman1002cd92015-09-16 16:30:48 +0200[diff] [blame^]43 user: guest
44 password: guest
45 vhost: /logs
46 exchange: logs
47 exchange_type: fanout
48 encoder: ProtobufEncoder
49 use_framing: true
50 heka-logfile:
51 engine: logoutput
52 encoder: RstEncoder
53 message_matcher: TRUE
54 encoder:
55 heka-logfile:
56 engine: RstEncoder
57
58
59Heka acting as message router and dashboard.
60Messages are consumed from amqp and sent to elasticsearch server.
61
62
63.. code-block:: yaml
64
65
66 heka:
67 server:
68 enabled: true
Jakub Pavlik6014f912015-09-03 19:05:05 +0200[diff] [blame]69 input:
70 rabbitmq:
71 engine: amqp
72 host: localhost
73 user: guest
74 password: guest
jan kaufman1002cd92015-09-16 16:30:48 +0200[diff] [blame^]75 vhost: /logs
76 exchange: logs
77 exchange_type: fanout
78 decoder: ProtoBufDecoder
79 splitter: HekaFramingSplitter
80 rsyslog-syslog:
81 engine: logstreamer
82 log_directory: /var/log
83 file_match: syslog\.?(?P<Index>\d+)?(.gz)?
84 decoder: RsyslogDecoder
85 priority: ["^Index"]
86 rsyslog-auth:
87 engine: logstreamer
88 log_directory: /var/log
89 file_match: auth\.log\.?(?P<Index>\d+)?(.gz)?
90 decoder: RsyslogDecoder
91 priority: ["^Index"]
92 decoder:
93 rsyslog:
94 engine: rsyslog
95 template: %TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n
96 hostname_keep: TRUE
97 tz = Europe/Prague
98 output:
99 elasticsearch01:
100 engine: elasticsearch
101 host: localhost
102 port: 9200
103 encoder: es_json
104 message_matcher: TRUE
105 dashboard01:
106 engine: dashboard
107 ticker_interval: 30
108 encoder:
109 es-json:
110 engine: es-json
111 message_matcher: TRUE
112 index = logfile-%{%Y.%m.%d}
Jakub Pavlike7d12cd2015-09-03 19:02:45 +0200[diff] [blame]113
114Read more
115=========
116
jan kaufman1002cd92015-09-16 16:30:48 +0200[diff] [blame^]117* https://hekad.readthedocs.org/en/latest/index.html