You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "evan.zhao (Jira)" <ji...@apache.org> on 2026/09/10 11:19:00 UTC
[jira] [Commented] (KAFKA-21061) AtomicMoveNotSupportedException(Invalid cross‑device link) when deleting topic on mdadm software‑RAID
[ https://issues.apache.org/jira/browse/KAFKA-21061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18113802#comment-18113802 ]
evan.zhao commented on KAFKA-21061:
-----------------------------------
/home/worker/kafka/kafka_data_dir/okp_unblock_topic-2 to /home/worker/kafka/kafka_data_dir/okp_unblock_topic-2.a286a15bf2014b1eb7fd15fbcfe1926e-delete retrying with a non-atomic move (org.apache.kafka.common.utils.Utils)
java.nio.file.AtomicMoveNotSupportedException: /home/worker/kafka/kafka_data_dir/okp_unblock_topic-2 -> /home/worker/kafka/kafka_data_dir/okp_unblock_topic-2.a286a15bf2014b1eb7fd15fbcfe1926e-delete: Invalid cross-device link
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:394)
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
at java.nio.file.Files.move(Files.java:1395)
at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:955)
at kafka.log.LocalLog.$anonfun$renameDir$2(LocalLog.scala:113)
at kafka.log.LocalLog.renameDir(LocalLog.scala:716)
at kafka.log.UnifiedLog.$anonfun$renameDir$2(UnifiedLog.scala:683)
at kafka.log.UnifiedLog.renameDir(UnifiedLog.scala:1993)
at kafka.log.LogManager.asyncDelete(LogManager.scala:1322)
at kafka.log.LogManager.$anonfun$asyncDelete$4(LogManager.scala:1359)
at scala.Option.foreach(Option.scala:437)
at kafka.log.LogManager.$anonfun$asyncDelete$3(LogManager.scala:1357)
at kafka.log.LogManager.$anonfun$asyncDelete$3$adapted(LogManager.scala:1355)
at scala.collection.mutable.HashSet$Node.foreach(HashSet.scala:450)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:376)
at kafka.log.LogManager.asyncDelete(LogManager.scala:1355)
at kafka.server.ReplicaManager.stopPartitions(ReplicaManager.scala:616)
at kafka.server.ReplicaManager.stopReplicas(ReplicaManager.scala:551)
at kafka.server.KafkaApis.handleStopReplicaRequest(KafkaApis.scala:335)
at kafka.server.KafkaApis.handle(KafkaApis.scala:194)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:159)
at java.lang.Thread.run(Thread.java:750)
[2026-06-14 19:21:26,515] INFO [GroupCoordinator 1]: Loading group metadata for engine_1_format_normal with generation 0 (kafka.coordinator.group.GroupCoordinator)
[2026-06-14 19:21:26,517] INFO [GroupMetadataManager brokerId=1] Finished loading offsets and group metadata from __consumer_offsets-18 in 140 milliseconds for epoch 0, of which 10 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
[2026-06-14 19:21:26,517] INFO [GroupMetadataManager brokerId=1] Finished loading offsets and group metadata from __consumer_offsets-33 in 140 milliseconds for epoch 0, of which 140 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
[2026-06-14 19:21:26,537] ERROR Error while renaming dir for okp_unblock_topic-2 in log dir /home/worker/kafka/kafka_data_dir (org.apache.kafka.storage.internals.log.LogDirFailureChannel)
java.nio.file.DirectoryNotEmptyException: /home/worker/kafka/kafka_data_dir/okp_unblock_topic-2
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:498)
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
at java.nio.file.Files.move(Files.java:1395)
at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:959)
at kafka.log.LocalLog.$anonfun$renameDir$2(LocalLog.scala:113)
at kafka.log.LocalLog.renameDir(LocalLog.scala:716)
at kafka.log.UnifiedLog.$anonfun$renameDir$2(UnifiedLog.scala:683)
at kafka.log.UnifiedLog.renameDir(UnifiedLog.scala:1993)
at kafka.log.LogManager.asyncDelete(LogManager.scala:1322)
at kafka.log.LogManager.$anonfun$asyncDelete$4(LogManager.scala:1359)
at scala.Option.foreach(Option.scala:437)
at kafka.log.LogManager.$anonfun$asyncDelete$3(LogManager.scala:1357)
at kafka.log.LogManager.$anonfun$asyncDelete$3$adapted(LogManager.scala:1355)
at scala.collection.mutable.HashSet$Node.foreach(HashSet.scala:450)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:376)
at kafka.log.LogManager.asyncDelete(LogManager.scala:1355)
at kafka.server.ReplicaManager.stopPartitions(ReplicaManager.scala:616)
at kafka.server.ReplicaManager.stopReplicas(ReplicaManager.scala:551)
at kafka.server.KafkaApis.handleStopReplicaRequest(KafkaApis.scala:335)
at kafka.server.KafkaApis.handle(KafkaApis.scala:194)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:159)
at java.lang.Thread.run(Thread.java:750)
> AtomicMoveNotSupportedException(Invalid cross‑device link) when deleting topic on mdadm software‑RAID
> -----------------------------------------------------------------------------------------------------
>
> Key: KAFKA-21061
> URL: https://issues.apache.org/jira/browse/KAFKA-21061
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 3.9.1
> Reporter: evan.zhao
> Priority: Major
> Attachments: image-2026-09-10-16-40-49-605.png, image-2026-09-10-16-41-21-375.png
>
>
> h3. Environment
> * OS: Linux
> * Kafka Version: {{3.9.1}}
> * Storage: Data directory resides on *mdadm software RAID array (/dev/md126)*
> * Filesystem: ext4 on md raid device
> !image-2026-09-10-16-41-21-375.png|width=463,height=299!
> h3. Problem Description
> When deleting a topic, Kafka tries to rename topic partition directory to a {{-delete}} suffixed marker directory via {{{}Utils.atomicMoveWithFallback(){}}}. It throws {{{}java.nio.file.AtomicMoveNotSupportedException: Invalid cross‑device link{}}}.
> After this exception, Kafka marks the entire log directory as {*}offline{*}, stops serving all replicas under this log dir. Internal {{__consumer_offsets}} partitions in the same log directory become unavailable, group metadata and offset loading fails, consumer groups stop working.
> Stack trace snippet:
> !image-2026-09-10-16-40-49-605.png|width=548,height=210!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)