fluent-bit redis output plugin

This plugin is used to have redis output from fluent-bit. You can use fluent-bit redis instead of logstash in a configuration where you have a redis and optional stunnel in front of your elasticsearch infrastructure.

The configuration typically looks like:

fluent-bit --> stunnel --> redis <-- logstash --> elasticsearch

If you have multiple elastic search servers, each covered with a redis cache in front it might look like this:

/-> stunnel --> redis <-- logstash --> elasticsearch 
           |
fluent-bit --> stunnel --> redis <-- logstash --> elasticsearch
           |
           \-> stunnel --> redis <-- logstash --> elasticsearch

Usage

docker run -it --rm -v /path/to/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf majst01/fluent-bit-go-redis-output

Building

docker build --no-cache --tag fluent-bit-go-redis-output .
docker run -it --rm -v /path/to/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf fluent-bit-go-redis-output

Configuration Options

| Key | Description | Default | | --------------|------------------------------------------------|----------------| | Hosts | Host(s) of redis servers, whitespace separated ip/host:port | 127.0.0.1:6379 | | Password | Optional redis password for all redis instances | "" | | DB | redis database (integer) | 0 | | UseTLS | connect to redis with tls | False | | TlsSkipVerify | if tls is configured skip tls certificate validation for self signed certificates | True | | Key | the key where to store the entries in redis | "logstash" |

Example:

add this section to fluent-bit.conf

[Output]
    Name redis
    Match *
    UseTLS true
    TLSSkipVerify true
    # if port is ommited, 6379 is used
    Hosts 172.17.0.1 172.17.0.1:6380 172.17.0.1:6381 172.17.0.1:6382 172.17.0.1:6383
    Password homer
    DB 0
    Key elastic-logstash

Useful links

Redis format

• logrus-redis-hook

Logstash Redis Output

• logstash-redis-docu

TODO

Strategies for redis connection error handling

• crash on connection errors

Given a list of 4 Redis databases, we pick on start a random one, if during operation this fails we panic and on restart the next hopefully working is selected.

• rely on FLB_RETRY

With a list of redis databases we can create a list of pools, one pool per database and instead of doing a pool.Get(), call list.Get() with selects the next random redis database. If a failure occurs return FLB_RETRY and the library will retry.