You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Colin McCabe (Jira)" <ji...@apache.org> on 2020/06/15 17:22:00 UTC
[jira] [Updated] (KAFKA-8362) LogCleaner gets stuck after partition
move between log directories
[ https://issues.apache.org/jira/browse/KAFKA-8362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin McCabe updated KAFKA-8362:
--------------------------------
Component/s: jbod
> LogCleaner gets stuck after partition move between log directories
> ------------------------------------------------------------------
>
> Key: KAFKA-8362
> URL: https://issues.apache.org/jira/browse/KAFKA-8362
> Project: Kafka
> Issue Type: Bug
> Components: jbod, log cleaner
> Reporter: Julio Ng
> Priority: Major
>
> When a partition is moved from one directory to another, their checkpoint entry in cleaner-offset-checkpoint file is not removed from the source directory.
> As a consequence when we read the last firstDirtyOffset, we might get a stale value from the old checkpoint file.
> Basically, we need clean up the entry from the check point file in the source directory when the move is completed
> The current issue is that the code in LogCleanerManager:
> {noformat}
> /**
> * @return the position processed for all logs.
> */
> def allCleanerCheckpoints: Map[TopicPartition, Long] = {
> inLock(lock) {
> checkpoints.values.flatMap(checkpoint => {
> try {
> checkpoint.read()
> } catch {
> case e: KafkaStorageException =>
> error(s"Failed to access checkpoint file ${checkpoint.file.getName} in dir ${checkpoint.file.getParentFile.getAbsolutePath}", e)
> Map.empty[TopicPartition, Long]
> }
> }).toMap
> }
> }{noformat}
> collapses the offsets when multiple entries exist for the topicPartition
--
This message was sent by Atlassian Jira
(v8.3.4#803005)