Topics

#dataops #docker #kafka #kafka-rest-proxy #schema-registry

Language breakdown

Shell 46.7%

HTML 34.9%

Dockerfile 16.8%

JavaScript 1.6%

▸ Files click to expand

README

Fast Data Dev

Join the Slack Lenses.io Community!

Overview

Fast Data Dev is an Apache Kafka development environment packaged as a Docker container. It provides a Kafka ecosystem including a Kafka Broker, Confluent Schema Registry, and Kafka Connect with Lenses.io's Stream Reactor open source connectors (with optional enterprise support) pre-installed. On top of that there are data generators available, a web interface so you can access the various services logs and optionally TLS keystores and truststores, and a ton of settings so you can tailor it to your needs.

Quick Start

Our recommendation is to combine fast-data-dev with Lenses CE (community edition) using our docker-compose, available here. Lenses provides industry-leading Developer Experience for Apache Kafka and Kafka Connect and among many features, you may particularly enjoy our SQL Studio that allows you to run SQL queries on your Kafka topics and the SQL Processors for creating Streaming SQL queries.

Our docker-compose will setup fast-data-dev and Lenses for you.

Basic Usage

Start a complete Kafka environment with one command:

docker run --rm \
    -p 9092:9092 \
    -p 8081:8081 \
    -p 8083:8083 \
    -p 3030:3030 \
    lensesio/fast-data-dev

Access the web interface at: http://localhost:3030 The Kafka Broker will be accessible at localhost:9092, the Schema Registry at http://localhost:8081, and Kafka Connect at http://localhost:8083.

With a Custom Host for Remote Access

docker run --rm \
    -e ADVHOST=<YOURIP_ADDRESS or DNS name> \
    -p 9092:9092 \
    -p 8081:8081 \
    -p 8083:8083 \
    -p 3030:3030 \
    lensesio/fast-data-dev

This setup may require to allow connections to fast-data-dev from itself. For example if you are running in a Cloud VM, you may need to setup the firewall to not only accept connections from your computer, but also from the VM itself.

Within docker-compose

This setup allows for both access from your host and the rest of the docker-compose services. The Broker address from your host is localhost:9092, whilst for docker-compose services it is kafka:19092.

kafka:
    image: lensesio/fast-data-dev:3.9.0
    hostname: kafka
    environment:
      ADV_HOST: kafka
      RUNNING_SAMPLEDATA: 1
      RUNTESTS: 0
      # The next three variables are required if you want to have Kafka
      # available at the host for local development. They are tailored to KRaft
      # fast-data-dev (3.9.x or later). The broker will be available at localhost:9092.
      KAFKA_LISTENERS: PLAINTEXT://:9092,DOCKERCOMPOSE://:19092,CONTROLLER://:16062
      KAFKAADVERTISEDLISTENERS: PLAINTEXT://localhost:9092,DOCKERCOMPOSE://demo-kafka:19092
      KAFKALISTENERSECURITYPROTOCOLMAP: >
        DOCKERCOMPOSE:PLAINTEXT,CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,
        SSL:SSL,SASLPLAINTEXT:SASLPLAINTEXT,SASLSSL:SASLSSL
      # Makes things a little lighter
      DISABLE: debezium-mongodb,debezium-mysql,debezium-postgres,debezium-sqlserver,debezium-jdbc
    # These are the ports for Kafka and Schema Registry. You can disable them if
    # you don't need to access Kafka from your host.
    ports:
      - 9092:9092
      - 8081:8081
      - 8083:8083
      - 3030:3030
    # Optional, to allow to resume work (persistence)
    # volumes:
    #   - kafka-data-volume:/data

SSL/TLS Setup

# Enable SSL with automatic certificate generation
docker run --rm -p 9092:9092 -p 8081:8081 -p 3030:3030 -p 8083:8083 \
  -e ENABLE_SSL=1 \
  -e SSLEXTRAHOSTS=myhost.example.com,192.168.1.100 \
  lensesio/fast-data-dev
Access certificates at http://localhost:3030/certs/
Password for keystores: fastdata

No data generators and Kafka Connect

docker run --rm -p 9092:9092 -p 8081:8081 -p 3030:3030 \
  -e SAMPLEDATA=0 \
  -e RUNTESTS=0 \
  -e CONNECT_PORT=0 \
  lensesio/fast-data-dev

Connector Development

# Enable specific connectors only
docker run --rm --net=host \
  -e CONNECTORS=file \
  -e CONNECT_HEAP=2G \
  -v /path/to/custom/connector.jar:/connectors/connector.jar \
  lensesio/fast-data-dev

Architecture and Components

Fast Data Dev was first introduced in 2016, back when Kafka was getting traction but was not yet mainstream. We wanted to make it easy for everyone to get streaming within a couple minutes. People starting with Kafka and even experienced engineers hugged our effort and we keep maintaining it since then with upgrades and new features.

We do use vanilla Apache Kafka built from source, and Confluent's Schema Registry also built from source.

Components

The most reliable way to see the current components versions is to check /build.info:

docker run --rm lensesio/fast-data-dev:latest cat /build.info

Apache Kafka 3.9.0
Confluent Schema Registry 7.7.1
Stream Reactor Connectors 9.0.2

- AWS S3 - Azure Datalake - Azure DocumentDB - Azure EventHubs - Azure ServiceBus - Cassandra - ElasticSearch - FTP - Google Cloud Platform PubSub - Google Cloud Platform Cloud Storage - HTTP - JMS - MQTT - and more

Debezium Connectors 2.7.4

Services

Enabled by default services:

Kafka Broker

- Default address: PLAINTEXT://localhost:9092 - Setup in KRaft mode - JMX available at localhost:9581 - Optional TLS listener with -e ENABLE_TLS=1 - Optional advertised listener with -e ADV_HOST=[HOST]. If TLS is enabled, ADV_HOST will be added to the self-signed certificates. You can also add extra hosts with -e EXTRA_HOSTS=[HOST1],[HOST2],... - Configuration under /var/run/broker/server.properties - Restart with supervisorctl restart broker

Schema Registry

- Default address: http://localhost:8081 - Configuration under /var/run/schema-registry - Restart with supervisorctl restart schema-registry

Kafka Connect

- Default address: http://localhost:8083 - JMX available at localhost:9584 - Get a list of all available connectors:

docker run --rm lensesio/fast-data-dev:latest \       find /opt/lensesio/connectors -name "kafka-connect-*" -type d -exec basename '{}' \;

- Disable connectors to improve RAM usage or avoid conflicts with user-added plugins with -e DISABLE=aws-s3,debezium-jdbc - Explicitly enable connectors with -e CONNECTORS=aws-s3,file - Add your own connectors by mounting them under /connectors - Disable Connect entirely: -e CONNECT_PORT=0 - Configuration under /var/run/connect/connect-avro-distributed.properties - Restart with supervisorctl restart connect-distributed

Caddy Web Server

Serves a web interface where you can see logs, configuration files, and download self-signed keystore and truststore files. - Default address: http://localhost:3030 - Disable entirely with -e WEB_PORT=0 - Password protect by setting -e USER=[USER] -e PASSWORD=[PASSWORD]

Data Generators

Create a few topics with complex data in both AVRO and JSON formats. - By default will run once create and populate the topics, then exit. - To keep the generators running (loop over the datasets) set -e RUNNING_SAMPLEDATA=1 - To disable them set -e SAMPLEDATA=0 - If enabled, a demo file connector will also be created reading the broker's logs and producing them into a topic. To disable set -e FORWARDLOGS=0

Smoke Tests

Basic functionality tests to confirm the services started correctly - To disable set -e RUNTESTS=0

Disabled by default services:

Web terminal

- Enable with -e WEBTERMINALPORT=[PORT] - Default credentials: admin/admin - Change credentials with -e WEBTERMINALCREDS=[USER]:[PASS]

Supervisord Web UI

- Enable with -e SUPERVISORWEB=1 - Default address at http://localhost:9001 - If the web server is setup with credentials, they will also apply here

Advanced Configuration

Memory Configuration Options

-e C
-e BROKERHEAPOPTS="-Xmx320M -Xms320M"
-e C

Security and Authentication Options

-e USER=admin             # Basic auth username for Web UI (default: kafka)
-e PASSWORD=secret        # Basic auth password for Web UI
-e ENABLE_SSL=1           # Enable SSL/TLS for Broker
-e SSLEXTRAHOSTS=host1,host2  # Additional SSL certificate hosts
-e WEBTERMINALCREDS=admin:admin

Data and Testing

-e SAMPLEDATA=0           # Disable sample data generation
-e RUNNING_SAMPLEDATA=1   # Enable continuous sample data flow
-e RUNTESTS=0             # Disable integration tests
-e FORWARDLOGS=0          # Disable log forwarding to Kafka topics

Connector Management

-e CONNECTORS=jdbc,elastic,hbase     # Enable specific connectors only
-e DISABLE=hbase,mongodb             # Disable specific connectors

Debug and Development

-e DEBUG=1                # Enable debug logging
-e RUNASROOT=1          # Run services as root user
-e BROWSECONFIGS=1        # Expose service configs in web UI
-e SUPERVISORWEB=1        # Enable supervisor web interface (port 9001)
-e WEBTERMINALPORT=9002 # Enable web terminal

Kafka Component Configuration

Configure any Kafka component by converting properties to environment variables:

Replace dots with underscores
Convert to uppercase
Prefix with service name

Examples:

# Broker: log.retention.bytes -> KAFKALOGRETENTION_BYTES -e KAFKALOGRETENTION_BYTES=1073741824
Schema Registry: kafkastore.topic -> SCHEMAREGISTRYKAFKASTORE_TOPIC
-e SCHEMAREGISTRYKAFKASTORETOPIC=schemas
Connect: plugin.path -> CONNECTPLUGINPATH
-e CONNECTPLUGINPATH=/custom/connectors

Pre/Post Setup Scripts

Execute custom scripts during startup:

# Inline script -e PRE_SETUP="echo 'Pre-setup script'"
Script file
-e PRESETUPFILE="/path/to/script.sh"
-v /local/script.sh:/path/to/script.sh
Remote script
-e PRESETUPURL="https://example.com/setup.sh"

Working with the Environment

Command Line Tools

Access Kafka command line tools:

# Run commands directly docker run --rm -it --net=host lensesio/fast-data-dev \   kafka-topics --bootstrap-server localhost:9092 --list
Enter container for interactive use
docker run --rm -it --net=host lensesio/fast-data-dev bash

Troubleshooting

Common Issues

Container fails to start with hostname errors

- Solution: Add your hostname to /etc/hosts: 127.0.0.1 YourHostname localhost

Services not accessible from other machines

- Solution: Use -e ADVHOST=<YOURIP> and ensure firewall allows connections

Out of memory errors

- Solution: Increase Docker memory limit (minimum 4GB recommended) - Adjust heap sizes: -e CONNECTHEAP=2G -e BROKERHEAP_OPTS="-Xmx1G"

SSL connection issues

- Download certificates from http://localhost:3030/certs/ - Use password fastdata for all keystores

Memory Requirements

Minimum: 2GB RAM
Recommended: 4GB+ RAM
For heavy connector usage: 6GB+ RAM

Building from Source

Requirements

Docker with multi-stage build support
Optional: Docker buildx for multi-architecture builds

Build Commands

# Basic build
docker build -t local/fast-data-dev .
Multi-architecture build
docker buildx build --platform linux/amd64,linux/arm64 -t local/fast-data-dev .

🔗 More in this category