| Filip Pytloun | 28938a9 | 2015-10-07 14:03:38 +0200 | [diff] [blame^] | 1 | |
| 2 | ============ |
| 3 | Heka Formula |
| 4 | ============ |
| 5 | |
| 6 | Heka is an open source stream processing software system developed by Mozilla. Heka is a Swiss Army Knife type tool for data processing |
| 7 | |
| 8 | Sample pillars |
| 9 | ============== |
| 10 | |
| 11 | Basic log shipper streaming decoded rsyslog's logfiles using amqp broker as transport. |
| 12 | From every message there is one amqp message and it's also logged to heka's logfile in RST format. |
| 13 | |
| 14 | .. code-block:: yaml |
| 15 | |
| 16 | |
| 17 | heka: |
| 18 | server: |
| 19 | enabled: true |
| 20 | input: |
| 21 | rsyslog-syslog: |
| 22 | engine: logstreamer |
| 23 | log_directory: /var/log |
| 24 | file_match: syslog\.?(?P<Index>\d+)?(.gz)? |
| 25 | decoder: RsyslogDecoder |
| 26 | priority: ["^Index"] |
| 27 | rsyslog-auth: |
| 28 | engine: logstreamer |
| 29 | log_directory: /var/log |
| 30 | file_match: auth\.log\.?(?P<Index>\d+)?(.gz)? |
| 31 | decoder: RsyslogDecoder |
| 32 | priority: ["^Index"] |
| 33 | decoder: |
| 34 | rsyslog: |
| 35 | engine: rsyslog |
| 36 | template: %TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n |
| 37 | hostname_keep: TRUE |
| 38 | tz: Europe/Prague |
| 39 | output: |
| 40 | rabbitmq: |
| 41 | engine: amqp |
| 42 | host: localhost |
| 43 | user: guest |
| 44 | password: guest |
| 45 | vhost: /logs |
| 46 | exchange: logs |
| 47 | exchange_type: fanout |
| 48 | encoder: ProtobufEncoder |
| 49 | use_framing: true |
| 50 | heka-logfile: |
| 51 | engine: logoutput |
| 52 | encoder: RstEncoder |
| 53 | message_matcher: TRUE |
| 54 | encoder: |
| 55 | heka-logfile: |
| 56 | engine: RstEncoder |
| 57 | |
| 58 | |
| 59 | Heka acting as message router and dashboard. |
| 60 | Messages are consumed from amqp and sent to elasticsearch server. |
| 61 | |
| 62 | |
| 63 | .. code-block:: yaml |
| 64 | |
| 65 | |
| 66 | heka: |
| 67 | server: |
| 68 | enabled: true |
| 69 | input: |
| 70 | rabbitmq: |
| 71 | engine: amqp |
| 72 | host: localhost |
| 73 | user: guest |
| 74 | password: guest |
| 75 | vhost: /logs |
| 76 | exchange: logs |
| 77 | exchange_type: fanout |
| 78 | decoder: ProtoBufDecoder |
| 79 | splitter: HekaFramingSplitter |
| 80 | rsyslog-syslog: |
| 81 | engine: logstreamer |
| 82 | log_directory: /var/log |
| 83 | file_match: syslog\.?(?P<Index>\d+)?(.gz)? |
| 84 | decoder: RsyslogDecoder |
| 85 | priority: ["^Index"] |
| 86 | rsyslog-auth: |
| 87 | engine: logstreamer |
| 88 | log_directory: /var/log |
| 89 | file_match: auth\.log\.?(?P<Index>\d+)?(.gz)? |
| 90 | decoder: RsyslogDecoder |
| 91 | priority: ["^Index"] |
| 92 | decoder: |
| 93 | rsyslog: |
| 94 | engine: rsyslog |
| 95 | template: %TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n |
| 96 | hostname_keep: TRUE |
| 97 | tz: Europe/Prague |
| 98 | output: |
| 99 | elasticsearch01: |
| 100 | engine: elasticsearch |
| 101 | host: localhost |
| 102 | port: 9200 |
| 103 | encoder: es_json |
| 104 | message_matcher: TRUE |
| 105 | dashboard01: |
| 106 | engine: dashboard |
| 107 | ticker_interval: 30 |
| 108 | encoder: |
| 109 | es-json: |
| 110 | engine: es-json |
| 111 | message_matcher: TRUE |
| 112 | index = logfile-%{%Y.%m.%d} |
| 113 | |
| 114 | Read more |
| 115 | ========= |
| 116 | |
| 117 | * https://hekad.readthedocs.org/en/latest/index.html |