Learning Kafka - Configuring Kafka Producer for Idempotency

The following post covers the common configuration parameters in Kafka Producer and Kafka Cluster to achieve idempotency

Configuring Kafka Producer for idempotency involves various edge cases which should be considered. Refer librdkafka - Idempotent Producer for more information

Contents

Message duplication due to Network Error

kafka-producer-message-duplication

Idempotent Kafka Producer - Kafka Broker Setup

Kafka attaches a uniqueID to the messages being published which the Broker tests against before committing to its log. This prevents a message to be committed to the log twice due to network errors

kafka-producer-idempotent

Configuring Kafka Producer for Idempotency

For Kafka versions >1.1 an idempotent producer can be configured by setting the property enable.idempotence to true for the Kafka Producer.

enable.idempotence = true sets the following configuration parameters

  1. retries = Integet.MAX_VALUE

  2. max.in.flight.requests.per.connection = 5 (For Kafka>1.1)

  3. acks = all

Note that for Kafka versions>1.1 enabling idempotence also sets the configuration parameters to achieve message ordering (even with max.in.flight.requests.per.connection = 5)

As of writing this post node-rdkafka does not support idempotent configuration for Kafka Producer and hence there are no code samples for this post

Reference

  1. librdkafka - Idempotent Producer