This lesson explores the various configurations for Kafka producers and consumers for reliable message delivery.

We'll cover the following

Consumers

Explicit offset committing

Exactly Once Delivery

Even if we configure Kafka brokers with the most reliable configuration settings, we can still experience data loss across the system because of settings used on the producers or consumers. Consider the following scenario.

Say, we have three replicas running, and the option to have an unclean election is turned off. The producer uses the configuration setting acks=1 when sending messages to the partition leader. The producer sends a message to the partition leader which they successfully write. Before the message can be replicated by the in-sync replicas from the partition leader, it crashes. One of the in-sync followers which doesn’t have this latest message now becomes the leader (election is clean since the replica was in-sync). In this scenario, the producer thinks the message was successfully written, while in reality it was never committed since all the in-sync replicas apart from the partition leader were unable to copy it. The consumers will never see the message because it was never committed. Remember, consumers only read committed messages. The system is still consistent because the consumers never see the message, but from the producer’s perspective, the message is lost.

One response to the above scenario is to change the producer setting to acks=all. Now the producer receives a “message written successfully” notification only when the message has been replicated by all the in-sync replicas including the partition. But what if the producer sends a message in the midst of an election after a partition leader crashes? The producer will receive an exception LeaderNotAvailableException. The onus of handling that error and retrying to send the message is on the producer. The system remains consistent since the consumers never see this message. However, if the producer doesn’t handle retries, the message will be lost.