You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2016/12/07 02:36:58 UTC
[jira] [Commented] (SPARK-18756) Memory leak in Spark streaming
[ https://issues.apache.org/jira/browse/SPARK-18756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727466#comment-15727466 ]
Sean Owen commented on SPARK-18756:
-----------------------------------
CC [~zsxwing] is this related to the netty byte buffer stuff you've been dealing with for a while?
> Memory leak in Spark streaming
> ------------------------------
>
> Key: SPARK-18756
> URL: https://issues.apache.org/jira/browse/SPARK-18756
> Project: Spark
> Issue Type: Bug
> Components: Block Manager, DStreams
> Affects Versions: 2.0.0, 2.0.1, 2.0.2
> Reporter: Udit Mehrotra
>
> We have a Spark streaming application, that processes data from Kinesis.
> In our application we are observing a memory leak at the Executors with Netty buffers not being released properly, when the Spark BlockManager tries to replicate the input blocks received from Kinesis stream. The leak occurs, when we set Storage Level as MEMORY_AND_DISK_2 for the Kinesis input blocks. However, if we change the Storage level to use MEMORY_AND_DISK, which avoids creating a replica, we do not observe the leak any more. We were able to detect the leak, and obtain the stack trace by running the executors with an additional JVM option: -Dio.netty.leakDetectionLevel=advanced.
> Here is the stack trace of the leak:
> 16/12/06 22:30:12 ERROR ResourceLeakDetector: LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
> Recent access records: 0
> Created at:
> io.netty.buffer.CompositeByteBuf.<init>(CompositeByteBuf.java:103)
> io.netty.buffer.Unpooled.wrappedBuffer(Unpooled.java:335)
> io.netty.buffer.Unpooled.wrappedBuffer(Unpooled.java:247)
> org.apache.spark.util.io.ChunkedByteBuffer.toNetty(ChunkedByteBuffer.scala:69)
> org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$replicate(BlockManager.scala:1182)
> org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:997)
> org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:926)
> org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:866)
> org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:926)
> org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:702)
> org.apache.spark.streaming.receiver.BlockManagerBasedBlockHandler.storeBlock(ReceivedBlockHandler.scala:80)
> org.apache.spark.streaming.receiver.ReceiverSupervisorImpl.pushAndReportBlock(ReceiverSupervisorImpl.scala:158)
> org.apache.spark.streaming.receiver.ReceiverSupervisorImpl.pushArrayBuffer(ReceiverSupervisorImpl.scala:129)
> org.apache.spark.streaming.receiver.Receiver.store(Receiver.scala:133)
> org.apache.spark.streaming.kinesis.KinesisReceiver.org$apache$spark$streaming$kinesis$KinesisReceiver$$storeBlockWithRanges(KinesisReceiver.scala:282)
> org.apache.spark.streaming.kinesis.KinesisReceiver$GeneratedBlockHandler.onPushBlock(KinesisReceiver.scala:352)
> org.apache.spark.streaming.receiver.BlockGenerator.pushBlock(BlockGenerator.scala:297)
> org.apache.spark.streaming.receiver.BlockGenerator.org$apache$spark$streaming$receiver$BlockGenerator$$keepPushingBlocks(BlockGenerator.scala:269)
> org.apache.spark.streaming.receiver.BlockGenerator$$anon$1.run(BlockGenerator.scala:110)
> We also observe a continuous increase in off heap memory usage at the executors. Any help would be appreciated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org