[GitHub] [nifi] ijokarumawak commented on issue #3439: NIFI-6218 Support setting transactional.id in PublishKafka/PublishKaf…

[GitBox <gi...@apache.org>]

ijokarumawak commented on issue #3439: NIFI-6218 Support setting transactional.id in PublishKafka/PublishKaf…
URL: https://github.com/apache/nifi/pull/3439#issuecomment-484794485
 
 
   @szaboferee Thanks for adding the `transactional.id` support.
   
   I didn't know about how that id is used at Kafka cluster, but looks interesting, so tried learning about it.
   My understandings:
   - `transactional.id` is used by Kafka cluster to identify a Producer, even across restart or reconnect
   - The id is used to address 'zombie instances'
   https://stackoverflow.com/questions/50335227/how-to-pick-a-kafka-transaction-id
   
   Based on that, I assume NiFi users should configure PublishKafka's transactional.id to make it having following characteristics:
   - Consistent across restart. So it shouldn't contain any UUID or Timestamp ... etc such values changing when the processor starts
       - We may want to store the generated value into NiFi Managed State (Local) if we support these type of values to be used as transactional.id
   - Unique per NiFi node (or even per Processor??). Having the same transactional.id with more than 1 NiFi node would cause issue
   
   If above understanding is correct, I would configure it as: `my-nifi-cluster-${hostname}`. Will it be a reasonable configuration for its purpose?
   If so, I think we should add that to the processor property's description as an good example value.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services