You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "liangtianlun (Jira)" <ji...@apache.org> on 2020/12/09 07:15:00 UTC
[jira] [Updated] (SPARK-33710) Shuffle Index use Guava cache OOM,
Yarn NodeManage GC Alarm
[ https://issues.apache.org/jira/browse/SPARK-33710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
liangtianlun updated SPARK-33710:
---------------------------------
Description:
h2. CDH6.3 Yarn nodemanger frequently GC, and then the dump file is generated due to memory overflow
!https://upload-images.jianshu.io/upload_images/18249296-24acecfcc46dc744.png?imageMogr2/auto-orient/strip|imageView2/2/w/607/format/webp!
h2. Use the Memory Analyzer Tool to locate the shuffle index module
Using guava to cache the memory limit, there is no restriction on the cache key, resulting in a lot of path information in the memory. If the size of shuffleindexinformation in the cache is very small, the number of keys will be very large, and eventually lead to memory overflow. I think there is a defect here, and the capacity of key should be added to the statistics of 100MB
!https://upload-images.jianshu.io/upload_images/18249296-ed0cfee76b6f6bf2.png?imageMogr2/auto-orient/strip|imageView2/2/w/630/format/webp!
According to the MAT, the ExternalShuffleBlockHandler uses guava's local cache and takes up 82.88% of the heap memory
!https://upload-images.jianshu.io/upload_images/18249296-43ec91771f3c68b7.png?imageMogr2/auto-orient/strip|imageView2/2/w/760/format/webp!
!https://upload-images.jianshu.io/upload_images/18249296-f85e27a501605260.png?imageMogr2/auto-orient/strip|imageView2/2/w/1147/format/webp!
Through the analysis, it is found that there are a lot of shuffle index path information in the memory, which takes up more than 400 MB of memory, and the number is very large. This path is the key of shuffleindex cache in the external shufflebock resolver. After looking at the source code, we know that there may be some defects in the cache management, because the limited 100MB does not include the key statistics
!https://upload-images.jianshu.io/upload_images/18249296-87118ce13744c2ca.png?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp!
was:
h2. YARN NodeManger频繁GC, 随后内存溢出生成了dump文件
!https://upload-images.jianshu.io/upload_images/18249296-24acecfcc46dc744.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240!
h2. 使用MAT内存分析工具, 定位到Shuffle Index模块
使用guava缓存内存限制, 没有对缓存的key做限制, 导致内存中存在大量的路径信息;如果缓存中的ShuffleIndexInformation的size非常小的时候, 那么key的数量会非常多, 最终导致内存溢出; 我觉得这里是有缺陷的, key的容量也应该加入100mb的统计中去
!https://upload-images.jianshu.io/upload_images/18249296-ed0cfee76b6f6bf2.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240!
!https://upload-images.jianshu.io/upload_images/18249296-43ec91771f3c68b7.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240!
!https://upload-images.jianshu.io/upload_images/18249296-f85e27a501605260.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240!
!https://upload-images.jianshu.io/upload_images/18249296-87118ce13744c2ca.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240!
Summary: Shuffle Index use Guava cache OOM, Yarn NodeManage GC Alarm (was: Shuffle Index缓存内存泄漏OOM)
> Shuffle Index use Guava cache OOM, Yarn NodeManage GC Alarm
> -----------------------------------------------------------
>
> Key: SPARK-33710
> URL: https://issues.apache.org/jira/browse/SPARK-33710
> Project: Spark
> Issue Type: Bug
> Components: Shuffle, YARN
> Affects Versions: 3.2.0
> Reporter: liangtianlun
> Priority: Major
>
> h2. CDH6.3 Yarn nodemanger frequently GC, and then the dump file is generated due to memory overflow
> !https://upload-images.jianshu.io/upload_images/18249296-24acecfcc46dc744.png?imageMogr2/auto-orient/strip|imageView2/2/w/607/format/webp!
>
> h2. Use the Memory Analyzer Tool to locate the shuffle index module
>
> Using guava to cache the memory limit, there is no restriction on the cache key, resulting in a lot of path information in the memory. If the size of shuffleindexinformation in the cache is very small, the number of keys will be very large, and eventually lead to memory overflow. I think there is a defect here, and the capacity of key should be added to the statistics of 100MB
>
> !https://upload-images.jianshu.io/upload_images/18249296-ed0cfee76b6f6bf2.png?imageMogr2/auto-orient/strip|imageView2/2/w/630/format/webp!
> According to the MAT, the ExternalShuffleBlockHandler uses guava's local cache and takes up 82.88% of the heap memory
>
>
> !https://upload-images.jianshu.io/upload_images/18249296-43ec91771f3c68b7.png?imageMogr2/auto-orient/strip|imageView2/2/w/760/format/webp!
> !https://upload-images.jianshu.io/upload_images/18249296-f85e27a501605260.png?imageMogr2/auto-orient/strip|imageView2/2/w/1147/format/webp!
> Through the analysis, it is found that there are a lot of shuffle index path information in the memory, which takes up more than 400 MB of memory, and the number is very large. This path is the key of shuffleindex cache in the external shufflebock resolver. After looking at the source code, we know that there may be some defects in the cache management, because the limited 100MB does not include the key statistics
> !https://upload-images.jianshu.io/upload_images/18249296-87118ce13744c2ca.png?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp!
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org