[GitHub] [nifi] ijokarumawak edited a comment on issue #3439: NIFI-6218 Support setting transactional.id in PublishKafka/PublishKaf…

[GitBox <gi...@apache.org>]

ijokarumawak edited a comment on issue #3439: NIFI-6218 Support setting transactional.id in PublishKafka/PublishKaf…
URL: https://github.com/apache/nifi/pull/3439#issuecomment-484794485
 
 
   @szaboferee Thanks for adding the `transactional.id` support.
   
   I didn't know about how that id is used at Kafka cluster, but looks interesting, so tried learning about it.
   My understandings:
   - `transactional.id` is used by Kafka cluster to identify a Producer, even across restart or reconnect
   - The id is used to address 'zombie instances'
   https://stackoverflow.com/questions/50335227/how-to-pick-a-kafka-transaction-id
   
   Based on that, I assume NiFi users should configure PublishKafka's transactional.id to make it having following characteristics:
   - Consistent across restart. So it shouldn't contain any UUID or Timestamp ... etc such values changing when the processor starts
       - We may want to store the generated value into NiFi Managed State (Local) if we support these type of values to be used as transactional.id
   - Unique per NiFi node (or even per Processor, if writing to the same topic from different processor instances). Having the same transactional.id with more than 1 NiFi node would cause issue.
   
   If above understanding is correct, I would configure it as: `my-nifi-cluster-${hostname}-processor-id`. Will it be a reasonable configuration for its purpose?
   If so, I think we should add that to the processor property's description as an good example value.
   
   Umm, it seems it's difficult to configure the value from a user's standing point.
   I prefer the processor generate a UUID when it starts at the first time, then store it into its local state, then reuse it when it restart later.
   
   How do you think??

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services