You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "hzh0425 (Jira)" <ji...@apache.org> on 2023/01/20 12:32:00 UTC
[jira] [Comment Edited] (KAFKA-14642) TopicBased RemoteLogMetadataManager can't support reassign user-partitions
[ https://issues.apache.org/jira/browse/KAFKA-14642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17679131#comment-17679131 ]
hzh0425 edited comment on KAFKA-14642 at 1/20/23 12:31 PM:
-----------------------------------------------------------
BTW, can I implement a Rocksdb based metadataCache for TopicBased RLMM? [~Hangleton] [~satish.duggana]
was (Author: JIRAUSER298236):
BTW, can I implement a Rocksdb based metadata storage for TopicBased RLMM? [~Hangleton] [~satish.duggana]
> TopicBased RemoteLogMetadataManager can't support reassign user-partitions
> --------------------------------------------------------------------------
>
> Key: KAFKA-14642
> URL: https://issues.apache.org/jira/browse/KAFKA-14642
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 3.3.1
> Reporter: hzh0425
> Assignee: hzh0425
> Priority: Blocker
>
> {*}Background{*}:
> In [KIP-405: Kafka Tiered Storage - Apache Kafka - Apache Software Foundation|https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage], kafka introduced the feature of hierarchical storage.
> Also, [KAFKA-9555] Topic-based implementation for the RemoteLogMetadataManager - ASF JIRA (apache.org) implements the default RLMM - 'TopicBased-RLMM'.
> {*}Problem{*}:
> TopicBased-RLMM will only subscribe to the Partitions where the current Broker is Leader or Follower. If the current Broker is not the Leader or Follower, then RLMM will directly skip the metadata records related to these Partitions.
> When reassign user-partitions occurs, rlmm will subscribe to new user-partitions, assuming that the metadata-partition to which the new user-partition belongs is 'metadata-partition0', and RLMM has consumed 'metadata-partition0' *to offset = 100* before the reassign partition occurs, then {*}after reassign{*}, RMLM will *not* consume 'metadata-partition0' \{*}from the beginning{*}, and finally cause the metadata records related to the new user-partition to *be lost with offset < 100.*
> *Solution*
> Let RLMM subscribe to all user-patitions, instead of only subscribing to partitions where the current broker is leader or follower.
> In this way, when reassign partition occurs, RLMM will have new partition's metadata records.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)