You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Prasanna Gautam (JIRA)" <ji...@apache.org> on 2017/07/23 16:16:02 UTC
[jira] [Updated] (KAFKA-5628) Kafka Startup fails on corrupted
index files
[ https://issues.apache.org/jira/browse/KAFKA-5628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prasanna Gautam updated KAFKA-5628:
-----------------------------------
Priority: Minor (was: Major)
Description:
One of our kafka brokers shut down after a load test and while there are some corrupted index files , the broker is failing to start with a unsafe memory access error
{code:java}
[2017-07-23 15:52:32,019] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:53)
at org.apache.kafka.common.utils.Utils.readFully(Utils.java:854)
at org.apache.kafka.common.utils.Utils.readFullyOrFail(Utils.java:827)
at org.apache.kafka.common.record.FileLogInputStream$FileChannelLogEntry.loadRecord(FileLogInputStream.java:136)
at org.apache.kafka.common.record.FileLogInputStream$FileChannelLogEntry.record(FileLogInputStream.java:149)
at kafka.log.LogSegment$$anonfun$recover$1.apply(LogSegment.scala:225)
at kafka.log.LogSegment$$anonfun$recover$1.apply(LogSegment.scala:224)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at kafka.log.LogSegment.recover(LogSegment.scala:224)
at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:231)
at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:188)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at kafka.log.Log.loadSegments(Log.scala:188)
at kafka.log.Log.<init>(Log.scala:116)
at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$10$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:157)
at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
This doesn't seem to be same as https://issues.apache.org/jira/browse/KAFKA-1554 because these topics are actively in use and the other empty indices are recovered fine..
It seems the machine had died because the disk was full.
It seems to have resolved after the disk issue. Should kafka just check disk at startup and refuse to continue starting up?
was:
One of our kafka brokers shut down after a load test and while there are some corrupted index files , the broker is failing to start with a unsafe memory access error
{code:java}
[2017-07-23 15:52:32,019] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:53)
at org.apache.kafka.common.utils.Utils.readFully(Utils.java:854)
at org.apache.kafka.common.utils.Utils.readFullyOrFail(Utils.java:827)
at org.apache.kafka.common.record.FileLogInputStream$FileChannelLogEntry.loadRecord(FileLogInputStream.java:136)
at org.apache.kafka.common.record.FileLogInputStream$FileChannelLogEntry.record(FileLogInputStream.java:149)
at kafka.log.LogSegment$$anonfun$recover$1.apply(LogSegment.scala:225)
at kafka.log.LogSegment$$anonfun$recover$1.apply(LogSegment.scala:224)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at kafka.log.LogSegment.recover(LogSegment.scala:224)
at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:231)
at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:188)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at kafka.log.Log.loadSegments(Log.scala:188)
at kafka.log.Log.<init>(Log.scala:116)
at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$10$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:157)
at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
This doesn't seem to be same as https://issues.apache.org/jira/browse/KAFKA-1554 because these topics are actively in use and the other empty indices are recovered fine..
It seems the machine had died because the disk was full.
> Kafka Startup fails on corrupted index files
> --------------------------------------------
>
> Key: KAFKA-5628
> URL: https://issues.apache.org/jira/browse/KAFKA-5628
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.10.2.0
> Environment: Ubuntu 14.04, Java 8(1.8.0_65)
> Reporter: Prasanna Gautam
> Assignee: Jun Rao
> Priority: Minor
>
> One of our kafka brokers shut down after a load test and while there are some corrupted index files , the broker is failing to start with a unsafe memory access error
> {code:java}
> [2017-07-23 15:52:32,019] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
> java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
> at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:53)
> at org.apache.kafka.common.utils.Utils.readFully(Utils.java:854)
> at org.apache.kafka.common.utils.Utils.readFullyOrFail(Utils.java:827)
> at org.apache.kafka.common.record.FileLogInputStream$FileChannelLogEntry.loadRecord(FileLogInputStream.java:136)
> at org.apache.kafka.common.record.FileLogInputStream$FileChannelLogEntry.record(FileLogInputStream.java:149)
> at kafka.log.LogSegment$$anonfun$recover$1.apply(LogSegment.scala:225)
> at kafka.log.LogSegment$$anonfun$recover$1.apply(LogSegment.scala:224)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.log.LogSegment.recover(LogSegment.scala:224)
> at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:231)
> at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:188)
> at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
> at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
> at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
> at kafka.log.Log.loadSegments(Log.scala:188)
> at kafka.log.Log.<init>(Log.scala:116)
> at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$10$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:157)
> at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:57)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> This doesn't seem to be same as https://issues.apache.org/jira/browse/KAFKA-1554 because these topics are actively in use and the other empty indices are recovered fine..
> It seems the machine had died because the disk was full.
> It seems to have resolved after the disk issue. Should kafka just check disk at startup and refuse to continue starting up?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)