[jira] [Comment Edited] (KAFKA-13415) Track topic deletion state without ZK

["Omnia Ibrahim (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-13415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17452759#comment-17452759 ] 

Omnia Ibrahim edited comment on KAFKA-13415 at 12/3/21, 5:16 AM:
-----------------------------------------------------------------

Hey Colin, I don't believe this is resolved yet. If you check the thread on mailing list I explained that what I meant by "stuck" for deletion is regarding the state of LogDir deletion. When the cluster receives a topic deletion request, the KRAFT will delete the topic's metadata and eventually delete all LogDirs on any broker/disk at some point (correct me if I am wrong).
My question is how to confirm the deletion state of these LogDirs. For example, what will happen for the following case? * Topic-A has one partition with four replicas on broker1, 2, 3, 4
 * The owner of Topic-A fired a topic deletion request.
 * Broker1,2,3 have no issues; however, the disk on broker 4 isn't reachable.
 * Broker1,2,3 deleted any LogDir they had for Topic-A.
 * Broker 4 can't delete the LogDir (for example, the disk went to read-only mode).
 * Now Topic-A's data isn't entirely deleted until this last LogDir on broker-4 is deleted.
 * For data compliance, we need to confirm the deletion of the data.

 
Will KRAFT keeps listing the Topic-A until the last LogDir on broker-4 gets deleted? If this is the case, then this topic should be marked "Pending for deletion" for transparency; If KRAFT does not list the topic anymore, how will we confirm that all topic's LogDirs have been deleted?


was (Author: omnia_h_ibrahim):
Hey Colin, I don't believe this is resolved yet. If you check the thread on mailing list I explained what I meant by "stuck" for deletion is regarding the state of LogDir deletion. When the cluster receives a topic deletion request, the KRAFT will delete the topic's metadata and eventually delete all LogDirs on any broker/disk at some point (correct me if I am wrong).
My question is how to confirm the deletion state of these LogDirs. For example, what will happen for the following case? * Topic-A has one partition with four replicas on broker1, 2, 3, 4
 * The owner of Topic-A fired a topic deletion request.
 * Broker1,2,3 have no issues; however, the disk on broker 4 isn't reachable. 
 * Broker1,2,3 deleted any LogDir they had for Topic-A.
 * Broker 4 can't delete the LogDir (for example, the disk went to read-only mode).
 * Now Topic-A's data isn't entirely deleted until this last LogDir on broker-4 is deleted.
 * For data compliance, we need to confirm the deletion of the data.

 
Will KRAFT keeps listing the Topic-A until the last LogDir on broker-4 gets deleted? If this is the case, then this topic should be marked "Pending for deletion" for transparency; If KRAFT does not list the topic anymore, how will we confirm that all topic's LogDirs have been deleted?

> Track topic deletion state without ZK
> -------------------------------------
>
>                 Key: KAFKA-13415
>                 URL: https://issues.apache.org/jira/browse/KAFKA-13415
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 3.0.0
>            Reporter: Omnia Ibrahim
>            Priority: Major
>              Labels: kip-500
>
> Kafka topicCommand used to report which topic is marked for deletion by checking the znode on zookeeper; this feature has been silently deprecated without replacement as part of KAFKA-12596 Remove deprecated --zookeeper in topicCommands.
> Also as far as I can see, there's no equivalent for this with KIP-500 as well.
>  
> Is there any other way to know the state of deletion for Kafka with ZK and Without ZK?
> Is possible to leverage `RemoveTopicRecord` on the metadata topic to provide the same feature?



--
This message was sent by Atlassian Jira
(v8.20.1#820001)