You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Jay Lee (JIRA)" <ji...@apache.org> on 2019/06/13 06:06:00 UTC
[jira] [Created] (KAFKA-8533) Kafka 2.1.0 version crashes when deleting log

Jay Lee created KAFKA-8533:
------------------------------

             Summary: Kafka 2.1.0 version crashes when deleting log
                 Key: KAFKA-8533
                 URL: https://issues.apache.org/jira/browse/KAFKA-8533
             Project: Kafka
          Issue Type: Bug
          Components: log cleaner
    Affects Versions: 2.1.0
            Reporter: Jay Lee


We are running Kafka 2.1.0 version in RedHat Linux. 

On 28th-May, Kafka crashed with the log in the bottom. 

Please let me know what may be the issue. 

For your information, here below are the versions of our OS & kafka: 
 * *OS: RHEL 7.6*
 * *Kafka: 2.12-2.1.0*

======================================================

May 28 08:31:51 tsea-bc-db02 sh: [2019-05-28 08:31:51,291] ERROR Failed to clean up log for __consumer_offsets-46 in dir /tmp/kafka-logs due to IOException (kafka.server.LogDirFailureChannel)
May 28 08:31:51 tsea-bc-db02 sh: java.nio.file.NoSuchFileException: /tmp/kafka-logs/__consumer_offsets-46/00000000000000000000.log
May 28 08:31:51 tsea-bc-db02 sh: at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
May 28 08:31:51 tsea-bc-db02 sh: at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
May 28 08:31:51 tsea-bc-db02 sh: at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
May 28 08:31:51 tsea-bc-db02 sh: at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
May 28 08:31:51 tsea-bc-db02 sh: at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
May 28 08:31:51 tsea-bc-db02 sh: at java.nio.file.Files.move(Files.java:1395)
May 28 08:31:51 tsea-bc-db02 sh: at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:809)
May 28 08:31:51 tsea-bc-db02 sh: at org.apache.kafka.common.record.FileRecords.renameTo(FileRecords.java:222)
May 28 08:31:51 tsea-bc-db02 sh: at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:488)
May 28 08:31:51 tsea-bc-db02 sh: at kafka.log.Log.asyncDeleteSegment(Log.scala:1838)
May 28 08:31:51 tsea-bc-db02 sh: at kafka.log.Log.$anonfun$replaceSegments$6(Log.scala:1901)
May 28 08:31:51 tsea-bc-db02 sh: at kafka.log.Log.$anonfun$replaceSegments$6$adapted(Log.scala:1896)
May 28 08:31:51 tsea-bc-db02 sh: at scala.collection.immutable.List.foreach(List.scala:388)
May 28 08:31:51 tsea-bc-db02 sh: at kafka.log.Log.replaceSegments(Log.scala:1896)
May 28 08:31:51 tsea-bc-db02 sh: at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:583)
May 28 08:31:51 tsea-bc-db02 sh: at kafka.log.Cleaner.$anonfun$doClean$6(LogCleaner.scala:515)
May 28 08:31:51 tsea-bc-db02 sh: at kafka.log.Cleaner.$anonfun$doClean$6$adapted(LogCleaner.scala:514)
May 28 08:31:51 tsea-bc-db02 sh: at scala.collection.immutable.List.foreach(List.scala:388)
May 28 08:31:51 tsea-bc-db02 sh: at kafka.log.Cleaner.doClean(LogCleaner.scala:514)
May 28 08:31:51 tsea-bc-db02 sh: at kafka.log.Cleaner.clean(LogCleaner.scala:492)
May 28 08:31:51 tsea-bc-db02 sh: at kafka.log.LogCleaner$CleanerThread.cleanLog(LogCleaner.scala:353)
May 28 08:31:51 tsea-bc-db02 sh: at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:319)
May 28 08:31:51 tsea-bc-db02 sh: at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:300)
May 28 08:31:51 tsea-bc-db02 sh: at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
May 28 08:31:51 tsea-bc-db02 sh: Suppressed: java.nio.file.NoSuchFileException: /tmp/kafka-logs/__consumer_offsets-46/00000000000000000000.log -> /tmp/kafka-logs/__consumer_offsets-46/00000000000000000000.log.deleted
May 28 08:31:51 tsea-bc-db02 sh: at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
May 28 08:31:51 tsea-bc-db02 sh: at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
May 28 08:31:51 tsea-bc-db02 sh: at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:396)
May 28 08:31:51 tsea-bc-db02 sh: at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
May 28 08:31:51 tsea-bc-db02 sh: at java.nio.file.Files.move(Files.java:1395)
May 28 08:31:51 tsea-bc-db02 sh: at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:806)
May 28 08:31:51 tsea-bc-db02 sh: ... 17 more
May 28 08:31:51 tsea-bc-db02 sh: [2019-05-28 08:31:51,309] INFO [ReplicaManager broker=2] Stopping serving replicas in dir /tmp/kafka-logs (kafka.server.ReplicaManager)
May 28 08:31:51 tsea-bc-db02 sh: [2019-05-28 08:31:51,312] INFO [ReplicaFetcherManager on broker 2] Removed fetcher for partitions Set(__consumer_offsets-22, CONSUME_MESSAGE_TOPIC-0, __consumer_offsets-4, __consumer_offsets-7, __consumer_offsets-46, __consumer_offsets-25, __consumer_offsets-49, __consumer_offsets-16, __consumer_offsets-28, __consumer_offsets-31, __consumer_offsets-37, __consumer_offsets-19, __consumer_offsets-13, __consumer_offsets-43, __consumer_offsets-1, __consumer_offsets-34, __consumer_offsets-10, __consumer_offsets-40, PUBLISH_MESSAGE_TOPIC-0) (kafka.server.ReplicaFetcherManager)
May 28 08:31:51 tsea-bc-db02 sh: [2019-05-28 08:31:51,313] INFO [ReplicaAlterLogDirsManager on broker 2] Removed fetcher for partitions Set(__consumer_offsets-22, CONSUME_MESSAGE_TOPIC-0, __consumer_offsets-4, __consumer_offsets-7, __consumer_offsets-46, __consumer_offsets-25, __consumer_offsets-49, __consumer_offsets-16, __consumer_offsets-28, __consumer_offsets-31, __consumer_offsets-37, __consumer_offsets-19, __consumer_offsets-13, __consumer_offsets-43, __consumer_offsets-1, __consumer_offsets-34, __consumer_offsets-10, __consumer_offsets-40, PUBLISH_MESSAGE_TOPIC-0) (kafka.server.ReplicaAlterLogDirsManager)
May 28 08:31:51 tsea-bc-db02 sh: [2019-05-28 08:31:51,322] INFO [ReplicaManager broker=2] Broker 2 stopped fetcher for partitions __consumer_offsets-22,CONSUME_MESSAGE_TOPIC-0,__consumer_offsets-4,__consumer_offsets-7,__consumer_offsets-46,__consumer_offsets-25,__consumer_offsets-49,__consumer_offsets-16,__consumer_offsets-28,__consumer_offsets-31,__consumer_offsets-37,__consumer_offsets-19,__consumer_offsets-13,__consumer_offsets-43,__consumer_offsets-1,__consumer_offsets-34,__consumer_offsets-10,__consumer_offsets-40,PUBLISH_MESSAGE_TOPIC-0 and stopped moving logs for partitions because they are in the failed log directory /tmp/kafka-logs. (kafka.server.ReplicaManager)
May 28 08:31:51 tsea-bc-db02 sh: [2019-05-28 08:31:51,323] INFO Stopping serving logs in dir /tmp/kafka-logs (kafka.log.LogManager)
May 28 08:31:51 tsea-bc-db02 sh: [2019-05-28 08:31:51,328] ERROR Shutdown broker because all log dirs in /tmp/kafka-logs have failed (kafka.log.LogManager)
May 28 08:31:51 tsea-bc-db02 systemd: kafka.service: main process exited, code=exited, status=1/FAILURE



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)