You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "dgiri (Jira)" <ji...@apache.org> on 2023/08/10 12:07:00 UTC
[jira] [Updated] (KAFKA-15328) Inter broker partition reassignment triggers offline log dirs randomly
[ https://issues.apache.org/jira/browse/KAFKA-15328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dgiri updated KAFKA-15328:
--------------------------
Description:
When running the intra-broker partition reassignments from one disk to another, kafka triggers offline log dir. This happens randomly and can be reproduced only after running the intra-broker reassignments multiple times.
Observations made:
# After running the partition reassignment from (/data1 to /data2), kafka creates a *_topic-partition.uniqueId-future_* location on /data2 and starts copying the data.
# Partition reassignment completes as expected by renaming the _*topic-partition.uniqueId-future*_ location to _*topic-partition*_ on /data2.
# Within few milliseconds kafka still looks for _*topic-partition.uniqueId-future*_ location{_},{_} since it won't find that location anymore it marks the /data2 log dir as offline
Broker logs filtered with particular partition and error has been attached for more details.
was:
When running the intra-broker partition reassignments from one disk to another, kafka triggers offline log dir. This happens randomly and can be reproduced only after running the intra-broker reassignments multiple times.
Observations made:
# After running the partition reassignment from (/data1 to /data2), kafka creates a *_topic-partition.uniqueId-future_* location on /data2 and starts copying the data.
# Partition reassignment completes as expected by renaming the _*topic-partition.uniqueId-future*_ location __ to _*topic-partition*_ on /data2.
# Within few milliseconds kafka still looks for _*topic-partition.uniqueId-future*_ location{_},{_} since it won't find that location anymore it marks the /data2 log dir as offline
Broker logs filtered with particular partition and error has been attached for more details.
> Inter broker partition reassignment triggers offline log dirs randomly
> ----------------------------------------------------------------------
>
> Key: KAFKA-15328
> URL: https://issues.apache.org/jira/browse/KAFKA-15328
> Project: Kafka
> Issue Type: Bug
> Components: core, tools
> Affects Versions: 3.3.2
> Environment: RHEL 8.8 (Ootpa)
> Reporter: dgiri
> Priority: Major
> Labels: kafka-reassign-partitions
> Attachments: kafka_partition_reassignment_offline_lod_dir_errors.txt
>
>
> When running the intra-broker partition reassignments from one disk to another, kafka triggers offline log dir. This happens randomly and can be reproduced only after running the intra-broker reassignments multiple times.
>
> Observations made:
> # After running the partition reassignment from (/data1 to /data2), kafka creates a *_topic-partition.uniqueId-future_* location on /data2 and starts copying the data.
> # Partition reassignment completes as expected by renaming the _*topic-partition.uniqueId-future*_ location to _*topic-partition*_ on /data2.
> # Within few milliseconds kafka still looks for _*topic-partition.uniqueId-future*_ location{_},{_} since it won't find that location anymore it marks the /data2 log dir as offline
>
> Broker logs filtered with particular partition and error has been attached for more details.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)