[jira] [Created] (KAFKA-9493) Error after some time in Kafka on Kubernetes with PVC

["Rohit Agrawal (Jira)" <ji...@apache.org>]

Rohit Agrawal created KAFKA-9493:
------------------------------------

             Summary: Error after some time in Kafka on Kubernetes with PVC
                 Key: KAFKA-9493
                 URL: https://issues.apache.org/jira/browse/KAFKA-9493
             Project: Kafka
          Issue Type: Bug
            Reporter: Rohit Agrawal


Hi Team,

 

I keep on getting below error after some days on Kafka and after I start getting this error my Kafka pod is always in CrashBackLoopOff and keep on restarting. this is very critical for our project as we have data loss because of this error. please help me to rectify it.

 

[2020-02-01 19:05:45,257] ERROR Error while deleting the clean shutdown file in dir /var/lib/kafka/data (kafka.server.LogDirFailureChannel)

java.nio.file.FileSystemException: /var/lib/kafka/data/__consumer_offsets-29/leader-epoch-checkpoint: Input/output error

 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)

 at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)

 at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)

 at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)

 at java.nio.file.Files.newByteChannel(Files.java:361)

 at java.nio.file.Files.createFile(Files.java:632)

 at kafka.server.checkpoints.CheckpointFile.<init>(CheckpointFile.scala:45)

 at kafka.server.checkpoints.LeaderEpochCheckpointFile.<init>(LeaderEpochCheckpointFile.scala:60)

 at kafka.log.Log.newLeaderEpochFileCache$1(Log.scala:351)

 at kafka.log.Log.initializeLeaderEpochCache(Log.scala:367)

 at kafka.log.Log.<init>(Log.scala:283)

 at kafka.log.Log$.apply(Log.scala:2230)

 at kafka.log.LogManager.loadLog(LogManager.scala:275)

 at kafka.log.LogManager.$anonfun$loadLogs$12(LogManager.scala:345)

 at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:65)

 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

 at java.util.concurrent.FutureTask.run(FutureTask.java:266)

 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

 at java.lang.Thread.run(Thread.java:748)

[2020-02-01 19:05:45,257] ERROR Error while deleting the clean shutdown file in dir /var/lib/kafka/data (kafka.server.LogDirFailureChannel)

java.nio.file.FileSystemException: /var/lib/kafka/data/__consumer_offsets-35/leader-epoch-checkpoint: Input/output error

 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)

 at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)

 at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)

 at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)

 at java.nio.file.Files.newByteChannel(Files.java:361)

 at java.nio.file.Files.createFile(Files.java:632)

 at kafka.server.checkpoints.CheckpointFile.<init>(CheckpointFile.scala:45)

 at kafka.server.checkpoints.LeaderEpochCheckpointFile.<init>(LeaderEpochCheckpointFile.scala:60)

 at kafka.log.Log.newLeaderEpochFileCache$1(Log.scala:351)

 at kafka.log.Log.initializeLeaderEpochCache(Log.scala:367)

 at kafka.log.Log.<init>(Log.scala:283)

 at kafka.log.Log$.apply(Log.scala:2230)

 at kafka.log.LogManager.loadLog(LogManager.scala:275)

 at kafka.log.LogManager.$anonfun$loadLogs$12(LogManager.scala:345)

 at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:65)

 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

 at java.util.concurrent.FutureTask.run(FutureTask.java:266)

 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

 at java.lang.Thread.run(Thread.java:748)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)