Kafka Producer for Efficient Data Streaming to Kafka

This Python Kafka producer facilitates high-performance data streaming from Ray DataFrame and Pandas DataFrame to Kafka. It is optimized to provide approximately 3-4x performance improvement compared to standard Kafka producers.

Installation

Install the package using pip:

pip3 install raykafkaproducer@git+https://github.com/ujjawal-khare-27/ray-kafka-producer@main --force-reinstall

Usage

• Import the package

from raykafkaproducer.producer_manager import KafkaProducerManager

• Create an instance of KafkaProducerManager

# actorpoolsize is the number of actors that will be created to send data to Kafka
num_cpu is the number of CPUs that will be allocated to each actor
kafkaproducermanager = KafkaProducerManager(bootstrapservers="localhost:9092", topic="test", actorpool_size=12,
            num_cpu=0.25)

• Send messages to Kafka (Ray DataFrame)

kafkaproducermanager.flushraydf(df = raydf, isactor=True)

• Send messages to Kafka (Pandas DataFrame)

kafkaproducermanager.flushpandasdf(df = pandas_df)