Elastic Stack on Docker

Preconfigured Security, Tools, and Self-Monitoring

Configured to be ready to be used for Log, Metrics, APM, Alerting, Machine Learning, and Security (SIEM) usecases.

Introduction

Elastic Stack (ELK) Docker Composition, preconfigured with Security, Monitoring, and Tools; Up with a Single Command.

Suitable for Demoing, MVPs and small production deployments.

Stack Version: 9.4.2 🎉 - Based on Official Elastic Docker Images

You can change Elastic Stack version by setting ELK_VERSION in .env file and rebuild your images. Any version >= 9.0.0 is compatible with this template.

⚠️ Upgrading from 8.x? See the Upgrade Notes section below for breaking changes and migration steps.

Main Features 📜

• Configured as a Production Single Node Cluster. (With a multi-node cluster option for experimenting).
• Security Enabled By Default.
• Configured to Enable:

• Use Docker Compose and .env to configure your entire stack parameters.
• Persist Elasticsearch's Keystore and SSL Certifications.
• Self-Monitoring Metrics Enabled (using Metricbeat for ES 9+).
• Prometheus Exporters for Stack Metrics.
• Embedded Container Healthchecks for Stack Images.

More points

And comparing Elastdocker and the popular deviantony/docker-elk

Automatic Docker Container Log Collection

Collect logs from all Docker containers on your host with a single command:

mise run collect-docker-logs

Filebeat automatically discovers containers, parses logs, and ships them to Elasticsearch. View and analyze everything in Kibana with zero configuration.

Requirements

• Docker 20.05 or higher with Docker Compose v2
• 4GB RAM (For Windows and MacOS make sure Docker's VM has more than 4GB+ memory.)

Setup

• Clone the Repository

git clone https://github.com/sherifabdlnaby/elastdocker.git

• Initialize Elasticsearch Keystore and TLS Self-Signed Certificates

mise run stack:setup

> For Linux's docker hosts only. By default virtual memory is not enough so run the next command as root sysctl -w vm.maxmap_count=262144

• Start Elastic Stack

mise run up        # <OR>  docker compose up -d

• Visit Kibana at https://localhost:5601 or https://<yourpublic_ip>:5601

> - Notice that Kibana is configured to use HTTPS, so you'll need to write https:// before localhost:5601 in the browser. > - Modify .env file for your needs, most importantly ELASTICPASSWORD that setup your superuser elastic's password, ELASTICSEARCHHEAP & LOGSTASH_HEAP for Elasticsearch & Logstash Heap Size.

Whatever your Host (e.g AWS EC2, Azure, DigitalOcean, or on-premise server), once you expose your host to the network, ELK component will be accessible on their respective ports. Since the enabled TLS uses a self-signed certificate, it is recommended to SSL-Terminate public traffic using your signed certificates.

🏃🏻‍♂️ To start ingesting logs, you can start by running mise run collect-docker-logs which will collect your host's container logs.

Additional Commands

mise run monitoring

mise run collect-docker-logs

mise run all

mise run nodes

mise run build

mise run down

mise run prune

Configuration

• Some Configuration are parameterized in the .env file.

• Elasticsearch Configuration in elasticsearch.yml at ./elasticsearch/config.
• Logstash Configuration in logstash.yml at ./logstash/config/logstash.yml.
• Logstash Pipeline in main.conf at ./logstash/pipeline/main.conf.
• Kibana Configuration in kibana.yml at ./kibana/config.
• Metricbeat Configuration in metricbeat.yml at ./metricbeat/config (for Stack Monitoring in ES 9+).

Setting Up Keystore

You can extend the Keystore generation script by adding keys to ./setup/keystore.sh script. (e.g Add S3 Snapshot Repository Credentials)

To Re-generate Keystore:

mise run keystore

Notes

• ⚠️ Elasticsearch HTTP layer is using SSL, thus mean you need to configure your elasticsearch clients with the CA in secrets/certs/ca/ca.crt, or configure client to ignore SSL Certificate Verification (e.g --insecure in curl).

• Adding Two Extra Nodes to the cluster will make the cluster depending on them and won't start without them again.

• The stack is driven by mise tasks; run mise tasks to list them. The Makefile keeps the same commands but is deprecated.

• Elasticsearch will save its data to a volume named elasticsearch-data

• Elasticsearch Keystore (that contains passwords and credentials) and SSL Certificate are generated in the ./secrets directory by the setup command.

• Make sure to run mise run stack:setup if you changed ELASTIC_PASSWORD and to restart the stack afterwards.

• For Linux Users it's recommended to set the following configuration (run as root)

sysctl -w vm.maxmapcount=262144

By default, Virtual Memory is not enough.

Working with Elastic APM

After completing the setup step, you will notice a container named apm-server which gives you deeper visibility into your applications and can help you to identify and resolve root cause issues with correlated traces, logs, and metrics.

Authenticating with Elastic APM

In order to authenticate with Elastic APM, you will need the following:

• The value of ELASTICAPMSECRETTOKEN defined in .env file as we have secret token enabled by default
• The ability to reach port 8200
• Install elastic apm client in your application e.g. for NodeJS based applications you need to install elastic-apm-node
• Import the package in your application and call the start function, In case of NodeJS based application you can do the following:

const apm = require('elastic-apm-node').start({
  serviceName: 'foobar',
  secretToken: process.env.ELASTICAPMSECRET_TOKEN,
// https is enabled by default as per elastdocker configuration   serverUrl: 'https://localhost:8200', })

8.17.0 → 8.19.x (run Upgrade Assistant) → 9.x

curl https://mise.run | sh          # or: brew install mise
echo 'eval "$(mise activate zsh)"' >> ~/.zshrc   # bash: mise activate bash
mise doctor                         # confirm the install is healthy

mise trust      # allow this repo's mise config to load
mise run setup  # install the pinned tools; git hooks self-install