Conditional Produce in Kafka (KAFKA-2260)

[Niket Goel <ng...@confluent.io.INVALID>]

Hi all,

I am new to the Kafka project and while having a discussion with a colleague, I was pointed to the JIRA KAFKA-2260 <https://issues.apache.org/jira/browse/KAFKA-2260> [1] which talks about adding the ability to perform "conditional produce" requests to Kafka. This idea sounded very exciting to me and I can see it draw parallels with Optimistic Concurrency Control in Traditional Databases wherein Kafka could be used as a "system of record”, or as an arbiter for disparate systems working on the same data. The natural scale-out architecture of Kafka makes this ability even more interesting than the more monolithic databases that generally offer this. I found a system which is doing something like this - Waltz <https://wepay.github.io/waltz/docs/introduction>altz <https://wepay.github.io/waltz/docs/introduction> [2] today already.

Following the discussion in the JIRA [1] itself and then on KIP-27 <https://cwiki.apache.org/confluence/display/KAFKA/KIP-27+-+Conditional+Publish> [3], it seems like there was a lot of interest in this proposal, but it never materialized. Also IIUC, with the introduction of KIP-98 <https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging> [4], some of the use cases that were motivating the discussion seem to have been addressed (e.g. idempotent producer). 

In theory, introducing this ability to Kafka seems pretty great to me, but being new to Kafka I am not able to articulate real-life use cases where users of Kafka would utilize such an ability. This is where I would like help.

My questions to the group here today are:
1. Is there a use case that you are using/or want to use Kafka for, where having the ability to "conditionally produce” or "optimistically lock a key" would simplify your life?
2. If there are such scenarios today, how are people making it work? Is it through other Kafka features, or through application side logic.

Please feel to also add your thoughts on this feature’s usefulness, or even point me towards more suggested reading if I am missing things. :) 


Thanks
Niket Goel

[1] https://issues.apache.org/jira/browse/KAFKA-2260 <https://issues.apache.org/jira/browse/KAFKA-2260>
[2] https://wepay.github.io/waltz/docs/introduction <https://wepay.github.io/waltz/docs/introduction> 
[3] https://cwiki.apache.org/confluence/display/KAFKA/KIP-27+-+Conditional+Publish <https://cwiki.apache.org/confluence/display/KAFKA/KIP-27+-+Conditional+Publish> 
[4] https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging <https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging>