You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (JIRA)" <ji...@apache.org> on 2019/02/10 20:48:00 UTC
[jira] [Commented] (SPARK-26684) Add logs when allocating large
memory for PooledByteBufAllocator
[ https://issues.apache.org/jira/browse/SPARK-26684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16764558#comment-16764558 ]
Dongjoon Hyun commented on SPARK-26684:
---------------------------------------
Hi, [~liupengcheng].
As I commented on your PR, please contribute to `Netty` project itself. Inside `Netty` project, it is one liner for each functions.
- https://github.com/netty/netty/blob/4.1/buffer/src/main/java/io/netty/buffer/PooledByteBufAllocator.java#L321
> Add logs when allocating large memory for PooledByteBufAllocator
> ----------------------------------------------------------------
>
> Key: SPARK-26684
> URL: https://issues.apache.org/jira/browse/SPARK-26684
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle
> Affects Versions: 2.4.0
> Reporter: liupengcheng
> Priority: Trivial
>
> Currently, Spark use `PooledByteBufAllocator` to allocate memory for channel reading. However, the allocated heap/offheap memory size is not tracked. Sometimes, this make it difficult to find out the cause of OOM failures(for instance, direct memory oom). we have to dump the heap and use more advanced tools like MAT to locate the cause.
> Actually, we can add some logs for `PooledByteBufAllocator` when allocating large memory, which can facilitate the debugging.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org