SRE-78: Implement Multiworker Configuration
- In the booking.com clouds, we find that fluentd cannot keep up with
volume of logs it is expected to process.
- Research into https://docs.fluentd.org/deployment/multi-process-workers
has provided us with the option to configure a multiworker setup, which has
largely resolved the problem.
- MCP largely uses the systemd and tail inputs, which do not support
operating on multiple workers. However, we can still assign them to separate
workers on a multiworker setup. Since all workers will process outputs, we can
easily separate concerns this way and remove CPU bottlenecks.
- By default, we configure these inputs to work on worker 0.
Related-Prod: PROD-36710
Change-Id: I68af85f1a62da72fa2bf1f996e52465af3caa5a5
4 files changed