[jira] [Comment Edited] (KAFKA-10688) Handle accidental truncation of repartition topics as exceptional failure

["Guozhang Wang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-10688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17227080#comment-17227080 ] 

Guozhang Wang edited comment on KAFKA-10688 at 11/5/20, 11:50 PM:
------------------------------------------------------------------

Without KAFKA-3370, then we have to implement the desired behavior at the streams layer itself. That is:

1) Upon task assignment, explicitly set the starting offset for the main consumer based on the per-topic / global reset policy. For repartition topics, the reset policy would be using the global policy.

2) Upon task revive (for corrupted exception handling), do the same thing as 1).

3) During normal processing, if an InvalidOffsetException is thrown from main consumer, we differentiate these cases: 3.a) for source topics: log a warning and reset accordingly; 3.b) for repartition topics throw as fatal errors.

We can potentially be more strict that we require all topics contains committed offset, if only some of them have committed positions then fail. But for extensibility I'm going to hold on doing that for now.


was (Author: guozhang):
Without KAFKA-3370, then we have to implement the desired behavior at the streams layer itself. That is:

1) Upon task assignment, explicitly set the starting offset for the main consumer based on the per-topic / global reset policy. For repartition topics, the reset policy would be `latest`.

2) Upon task revive (for corrupted exception handling), do the same thing as 1).

3) During normal processing, if an InvalidOffsetException is thrown from main consumer, we differentiate these cases: 3.a) for source topics: log a warning and reset accordingly; 3.b) for repartition topics throw as fatal errors.

We can potentially be more strict that we require all topics contains committed offset, if only some of them have committed positions then fail. But for extensibility I'm going to hold on doing that for now.

> Handle accidental truncation of repartition topics as exceptional failure
> -------------------------------------------------------------------------
>
>                 Key: KAFKA-10688
>                 URL: https://issues.apache.org/jira/browse/KAFKA-10688
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Guozhang Wang
>            Assignee: Guozhang Wang
>            Priority: Major
>
> Today we always handle InvalidOffsetException from the main consumer by the resetting policy assuming they are for source topics. But repartition topics are also source topics and should never be truncated and hence cause InvalidOffsetException.
> We should differentiate these repartition topics from external source topics and treat the InvalidOffsetException from repartition topics as fatal and close the whole application.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)