[GitHub] [druid] jihoonson commented on issue #9322: Kafka Indexing Service consumer properties not respected

[GitBox <gi...@apache.org>]

jihoonson commented on issue #9322: Kafka Indexing Service consumer properties not respected
URL: https://github.com/apache/druid/issues/9322#issuecomment-583048861
 
 
   Hi @abossert, I believe it is hard-coded by design. The Kafka indexing service is designed to guarantee the exactly-once ingestion which requires Druid to track valid offsets of partitions. If some offsets are expired and automatically reset in Kafka, Druid will find some gaps between the offsets from Kafka and the ones it is aware of, which will lead to an internal error. If you want to reset automatically with the Kafka indexing service, please consider setting `resetOffsetAutomatically` and `useEarliestOffset` properly in the tuningConfig. Also, please note that resetting automatically will lead to DATA LOSS (when `useEarliestOffset = false) or DATA DUPLICATION (when `useEarliestOffset = true).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org