You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Kamal Chandraprakash (Jira)" <ji...@apache.org> on 2023/09/04 10:41:00 UTC
[jira] [Updated] (KAFKA-15352) Ensure consistency while deleting the remote log segments
[ https://issues.apache.org/jira/browse/KAFKA-15352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kamal Chandraprakash updated KAFKA-15352:
-----------------------------------------
Description:
In Kafka-14888, the remote log segments are deleted which breaches the retention time/size before updating the log-start-offset. In middle of deletion, if the consumer starts to read from the beginning of the topic, then it will fail to read the messages and UNKNOWN_SERVER_ERROR will be thrown back to the consumer.
To ensure consistency, similar to local log segments where the actual segments are deleted after {{{}segment.delete.delay.ms{}}}, we should update the log-start-offset first before deleting the remote log segment.
See the [PR#13561|https://github.com/apache/kafka/pull/13561] and [comment|https://github.com/apache/kafka/pull/13561#discussion_r1293086543] for more details.
Case-2:
The old-leader (follower) can delete the remote log segment in the middle of leader election. We need to update the log-start-offset metadata for this case.
See this comment [https://github.com/apache/kafka/pull/13561#discussion_r1293081560]
was:
In Kafka-14888, the remote log segments are deleted which breaches the retention time/size before updating the log-start-offset. In middle of deletion, if the consumer starts to read from the beginning of the topic, then it will fail to read the messages and UNKNOWN_SERVER_ERROR will be thrown back to the consumer.
To ensure consistency, similar to local log segments where the actual segments are deleted after {{segment.delete.delay.ms}}, we should update the log-start-offset first before deleting the remote log segment.
See the [PR#13561|https://github.com/apache/kafka/pull/13561] and [comment|https://github.com/apache/kafka/pull/13561#discussion_r1293086543] for more details.
> Ensure consistency while deleting the remote log segments
> ---------------------------------------------------------
>
> Key: KAFKA-15352
> URL: https://issues.apache.org/jira/browse/KAFKA-15352
> Project: Kafka
> Issue Type: Sub-task
> Reporter: Kamal Chandraprakash
> Assignee: Kamal Chandraprakash
> Priority: Blocker
> Fix For: 3.6.0
>
>
> In Kafka-14888, the remote log segments are deleted which breaches the retention time/size before updating the log-start-offset. In middle of deletion, if the consumer starts to read from the beginning of the topic, then it will fail to read the messages and UNKNOWN_SERVER_ERROR will be thrown back to the consumer.
> To ensure consistency, similar to local log segments where the actual segments are deleted after {{{}segment.delete.delay.ms{}}}, we should update the log-start-offset first before deleting the remote log segment.
> See the [PR#13561|https://github.com/apache/kafka/pull/13561] and [comment|https://github.com/apache/kafka/pull/13561#discussion_r1293086543] for more details.
> Case-2:
> The old-leader (follower) can delete the remote log segment in the middle of leader election. We need to update the log-start-offset metadata for this case.
> See this comment [https://github.com/apache/kafka/pull/13561#discussion_r1293081560]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)