You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "K Zakee (JIRA)" <ji...@apache.org> on 2015/03/21 19:35:38 UTC
[jira] [Updated] (KAFKA-2038) Unable to restart brokers after it
went down with no space left on disk
[ https://issues.apache.org/jira/browse/KAFKA-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
K Zakee updated KAFKA-2038:
---------------------------
Description:
One of the 8 volumes on brokers disks were full and broker wen down with below error, Cannot restart this broker, as it costantly throws the following error on startup.
[2015-03-21 03:12:21,433] FATAL [app=broker] [ReplicaFetcherThread-6-3] [ReplicaFetcherThread-6-3], Disk error while replicating data. (kafka.server.ReplicaFetcherThread)
kafka.common.KafkaStorageException: I/O exception in append to log ‘Topic-11'
at kafka.log.Log.append(Log.scala:266)
at kafka.server.ReplicaFetcherThread.processPartitionData(ReplicaFetcherThread.scala:54)
at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:128)
at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:109)
at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:224)
at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403)
at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply$mcV$sp(AbstractFetcherThread.scala:109)
at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:109)
at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:109)
at kafka.utils.Utils$.inLock(Utils.scala:535)
at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:108)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:86)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
Caused by: java.io.IOException: No space left on device
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.FileDispatcher.write(FileDispatcher.java:39)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69)
at sun.nio.ch.IOUtil.write(IOUtil.java:40)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:198)
at kafka.message.ByteBufferMessageSet.writeTo(ByteBufferMessageSet.scala:133)
at kafka.log.FileMessageSet.append(FileMessageSet.scala:210)
at kafka.log.LogSegment.append(LogSegment.scala:85)
at kafka.log.Log.append(Log.scala:309)
... 12 more
=================
[2015-03-21 10:38:25,244] INFO [app=broker] [main] [Kafka Server 5], shut down completed (kafka.server.KafkaServer)
[2015-03-21 10:38:25,245] FATAL [app=broker] [main] Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
at kafka.log.FileMessageSet$$anon$1.makeNext(FileMessageSet.scala:188)
at kafka.log.FileMessageSet$$anon$1.makeNext(FileMessageSet.scala:165)
at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
at kafka.log.LogSegment.recover(LogSegment.scala:175)
at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:162)
at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:141)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
at kafka.log.Log.loadSegments(Log.scala:141)
at kafka.log.Log.<init>(Log.scala:67)
at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:142)
at kafka.utils.Utils$$anon$1.run(Utils.scala:54)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
was:
On of the 8 volumes on brokers disks were full and broker wen down with below error, Cannot restart this broker, as it costantly throws the following error on startup.
[2015-03-21 03:12:21,433] FATAL [app=broker] [ReplicaFetcherThread-6-3] [ReplicaFetcherThread-6-3], Disk error while replicating data. (kafka.server.ReplicaFetcherThread)
kafka.common.KafkaStorageException: I/O exception in append to log ‘Topic-11'
at kafka.log.Log.append(Log.scala:266)
at kafka.server.ReplicaFetcherThread.processPartitionData(ReplicaFetcherThread.scala:54)
at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:128)
at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:109)
at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:224)
at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403)
at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply$mcV$sp(AbstractFetcherThread.scala:109)
at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:109)
at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:109)
at kafka.utils.Utils$.inLock(Utils.scala:535)
at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:108)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:86)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
Caused by: java.io.IOException: No space left on device
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.FileDispatcher.write(FileDispatcher.java:39)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69)
at sun.nio.ch.IOUtil.write(IOUtil.java:40)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:198)
at kafka.message.ByteBufferMessageSet.writeTo(ByteBufferMessageSet.scala:133)
at kafka.log.FileMessageSet.append(FileMessageSet.scala:210)
at kafka.log.LogSegment.append(LogSegment.scala:85)
at kafka.log.Log.append(Log.scala:309)
... 12 more
=================
[2015-03-21 10:38:25,244] INFO [app=broker] [main] [Kafka Server 5], shut down completed (kafka.server.KafkaServer)
[2015-03-21 10:38:25,245] FATAL [app=broker] [main] Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
at kafka.log.FileMessageSet$$anon$1.makeNext(FileMessageSet.scala:188)
at kafka.log.FileMessageSet$$anon$1.makeNext(FileMessageSet.scala:165)
at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
at kafka.log.LogSegment.recover(LogSegment.scala:175)
at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:162)
at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:141)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
at kafka.log.Log.loadSegments(Log.scala:141)
at kafka.log.Log.<init>(Log.scala:67)
at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:142)
at kafka.utils.Utils$$anon$1.run(Utils.scala:54)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
> Unable to restart brokers after it went down with no space left on disk
> -----------------------------------------------------------------------
>
> Key: KAFKA-2038
> URL: https://issues.apache.org/jira/browse/KAFKA-2038
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8.2.1
> Reporter: K Zakee
> Priority: Blocker
>
> One of the 8 volumes on brokers disks were full and broker wen down with below error, Cannot restart this broker, as it costantly throws the following error on startup.
> [2015-03-21 03:12:21,433] FATAL [app=broker] [ReplicaFetcherThread-6-3] [ReplicaFetcherThread-6-3], Disk error while replicating data. (kafka.server.ReplicaFetcherThread)
> kafka.common.KafkaStorageException: I/O exception in append to log ‘Topic-11'
> at kafka.log.Log.append(Log.scala:266)
> at kafka.server.ReplicaFetcherThread.processPartitionData(ReplicaFetcherThread.scala:54)
> at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:128)
> at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:109)
> at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:224)
> at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403)
> at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply$mcV$sp(AbstractFetcherThread.scala:109)
> at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:109)
> at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:109)
> at kafka.utils.Utils$.inLock(Utils.scala:535)
> at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:108)
> at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:86)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
> Caused by: java.io.IOException: No space left on device
> at sun.nio.ch.FileDispatcher.write0(Native Method)
> at sun.nio.ch.FileDispatcher.write(FileDispatcher.java:39)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69)
> at sun.nio.ch.IOUtil.write(IOUtil.java:40)
> at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:198)
> at kafka.message.ByteBufferMessageSet.writeTo(ByteBufferMessageSet.scala:133)
> at kafka.log.FileMessageSet.append(FileMessageSet.scala:210)
> at kafka.log.LogSegment.append(LogSegment.scala:85)
> at kafka.log.Log.append(Log.scala:309)
> ... 12 more
> =================
> [2015-03-21 10:38:25,244] INFO [app=broker] [main] [Kafka Server 5], shut down completed (kafka.server.KafkaServer)
> [2015-03-21 10:38:25,245] FATAL [app=broker] [main] Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
> java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
> at kafka.log.FileMessageSet$$anon$1.makeNext(FileMessageSet.scala:188)
> at kafka.log.FileMessageSet$$anon$1.makeNext(FileMessageSet.scala:165)
> at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
> at kafka.log.LogSegment.recover(LogSegment.scala:175)
> at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:162)
> at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:141)
> at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
> at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
> at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
> at kafka.log.Log.loadSegments(Log.scala:141)
> at kafka.log.Log.<init>(Log.scala:67)
> at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:142)
> at kafka.utils.Utils$$anon$1.run(Utils.scala:54)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)