You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Dong Lin (JIRA)" <ji...@apache.org> on 2018/07/25 17:53:00 UTC
[jira] [Comment Edited] (KAFKA-7190) Under low traffic conditions
purging repartition topics cause WARN statements about UNKNOWN_PRODUCER_ID
[ https://issues.apache.org/jira/browse/KAFKA-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16556053#comment-16556053 ]
Dong Lin edited comment on KAFKA-7190 at 7/25/18 5:52 PM:
----------------------------------------------------------
[~mjsax] I guess one question is, if LSO is defined for a partition, will consumer be able to consume beyond this offset, e.g. in read-uncommitted mode? If it is possible, then it seems that we should allow user to be able to delete messages beyond LSO. Since consumer may have already consumed it, it does not unnecessarily cause data loss if message beyond LSO is deleted. If it is impossible, then I agree we prevent user from deleting beyond LSO. What do you think?
was (Author: lindong):
[~mjsax] I guess one question is, if LSO is defined for a partition, will consumer be able to consume beyond this offset, e.g. in read-uncommitted mode? If it is possible, then it seems that we should allow user to be able to delete messages beyond LSO. Since consumer may have already consumed it, it does not unnecessarily cause data loss if message beyond LSO is deleted. If it is impossible, then I agree we prevent user from deleting beyond LSO. What do you think?
> Under low traffic conditions purging repartition topics cause WARN statements about UNKNOWN_PRODUCER_ID
> ---------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-7190
> URL: https://issues.apache.org/jira/browse/KAFKA-7190
> Project: Kafka
> Issue Type: Improvement
> Components: core, streams
> Affects Versions: 1.1.0, 1.1.1
> Reporter: Bill Bejeck
> Assignee: lambdaliu
> Priority: Major
>
> When a streams application has little traffic, then it is possible that consumer purging would delete
> even the last message sent by a producer (i.e., all the messages sent by
> this producer have been consumed and committed), and as a result, the broker
> would delete that producer's ID. The next time when this producer tries to
> send, it will get this UNKNOWN_PRODUCER_ID error code, but in this case,
> this error is retriable: the producer would just get a new producer id and
> retries, and then this time it will succeed.
>
> Possible fixes could be on the broker side, i.e., delaying the deletion of the produderIDs for a more extended period or on the streams side developing a more conservative approach to deleting offsets from repartition topics
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)