[jira] [Resolved] (KAFKA-10314) KafkaStorageException on reassignment when offline log directories exist

["Mickael Maison (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-10314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mickael Maison resolved KAFKA-10314.
------------------------------------
    Fix Version/s: 2.7.0
       Resolution: Fixed

> KafkaStorageException on reassignment when offline log directories exist
> ------------------------------------------------------------------------
>
>                 Key: KAFKA-10314
>                 URL: https://issues.apache.org/jira/browse/KAFKA-10314
>             Project: Kafka
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 2.5.0
>            Reporter: Noa Resare
>            Assignee: Noa Resare
>            Priority: Minor
>             Fix For: 2.7.0
>
>
> If a reassignment of a partition is triggered to a broker with an offline directory, the new broker will fail to follow, instead raising a KafkaStorageException which causes the reassignment to stall indefinitely. The error message we see is the following:
> {{[2020-07-23 13:11:08,727] ERROR [Broker id=1] Skipped the become-follower state change with correlation id 14 from controller 1 epoch 1 for partition t2-0 (last update controller epoch 1) with leader 2 since the replica for the partition is offline due to disk error org.apache.kafka.common.errors.KafkaStorageException: Can not create log for t2-0 because log directories /tmp/kafka/d1 are offline (state.change.logger)}}
> It seems to me that unless the partition in question already existed on the offline log partition, a better behaviour would simply be to assign the partition to one of the available log directories.
> The conditional in [LogManager.scala:769|https://github.com/apache/kafka/blob/11f75691b87fcecc8b29bfd25c7067e054e408ea/core/src/main/scala/kafka/log/LogManager.scala#L769] was introduced to prevent the issue in [KAFKA-4763|https://issues.apache.org/jira/browse/KAFKA-4763] where partitions in offline logdirs would be re-created in an online directory as soon as a LeaderAndISR message gets processed. However, the semantics of isNew seems different in LogManager (the replica is new on this broker) compared to when isNew is set in [KafkaController.scala|https://github.com/apache/kafka/blob/11f75691b87fcecc8b29bfd25c7067e054e408ea/core/src/main/scala/kafka/controller/KafkaController.scala#L879] (where it seems to refer to whether the topic partition in itself is new, all followers gets {{isNew=false}})



--
This message was sent by Atlassian Jira
(v8.3.4#803005)