You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2020/12/08 09:41:00 UTC
[jira] [Assigned] (SPARK-33710) Shuffle Index缓存内存泄漏OOM
[ https://issues.apache.org/jira/browse/SPARK-33710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-33710:
------------------------------------
Assignee: Apache Spark
> Shuffle Index缓存内存泄漏OOM
> ----------------------
>
> Key: SPARK-33710
> URL: https://issues.apache.org/jira/browse/SPARK-33710
> Project: Spark
> Issue Type: Bug
> Components: Shuffle, YARN
> Affects Versions: 3.2.0
> Reporter: liangtianlun
> Assignee: Apache Spark
> Priority: Major
>
> h2. YARN NodeManger频繁GC, 随后内存溢出生成了dump文件
> !https://upload-images.jianshu.io/upload_images/18249296-24acecfcc46dc744.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240!
>
> h2. 使用MAT内存分析工具, 定位到Shuffle Index模块
> 使用guava缓存内存限制, 没有对缓存的key做限制, 导致内存中存在大量的路径信息;如果缓存中的ShuffleIndexInformation的size非常小的时候, 那么key的数量会非常多, 最终导致内存溢出; 我觉得这里是有缺陷的, key的容量也应该加入100mb的统计中去
> !https://upload-images.jianshu.io/upload_images/18249296-ed0cfee76b6f6bf2.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240!
> !https://upload-images.jianshu.io/upload_images/18249296-43ec91771f3c68b7.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240!
> !https://upload-images.jianshu.io/upload_images/18249296-f85e27a501605260.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240!
> !https://upload-images.jianshu.io/upload_images/18249296-87118ce13744c2ca.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org