You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/12/08 09:39:05 UTC

[GitHub] [spark] liangtianlun opened a new pull request #30672: [SPARK-33710][ Shuffle] [YARN] Shuffle index use guava cache OOM, Yarn NodeManager GC alarm

liangtianlun opened a new pull request #30672:
URL: https://github.com/apache/spark/pull/30672


   ### What changes were proposed in this pull request?
   Guava cache capacity limit join key size statistics
   
   ### Why are the changes needed?
   When using guava cache, the key size is not counted, resulting in memory overflow. If the value is infinitely small, then the heap memory can store countless File type keys. I think this is a defect
   
   ### Does this PR introduce _any_ user-facing change?
   No
   
   ### How was this patch tested?
   No
   
   ![Cache OutOfMemory](https://upload-images.jianshu.io/upload_images/18249296-ed0cfee76b6f6bf2.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)
   ![Many Paths](https://upload-images.jianshu.io/upload_images/18249296-f85e27a501605260.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)
   
   [ISSUE SPARK-33710](https://issues.apache.org/jira/browse/SPARK-33710)
   [YARN GC ALARM](https://www.jianshu.com/writer#/notebooks/37701793/notes/80461429/preview)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30672: [SPARK-33710][ Shuffle] [YARN] Shuffle index use guava cache OOM, Yarn NodeManager GC alarm

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30672:
URL: https://github.com/apache/spark/pull/30672#issuecomment-740506068


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] holdenk commented on pull request #30672: [SPARK-33710][ Shuffle] [YARN] Shuffle index use guava cache OOM, Yarn NodeManager GC alarm

Posted by GitBox <gi...@apache.org>.
holdenk commented on pull request #30672:
URL: https://github.com/apache/spark/pull/30672#issuecomment-918630555


   Hey @liangtianlun  would you be open to re-submitting this PR for 3.3 with a test?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] github-actions[bot] closed pull request #30672: [SPARK-33710][ Shuffle] [YARN] Shuffle index use guava cache OOM, Yarn NodeManager GC alarm

Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #30672:
URL: https://github.com/apache/spark/pull/30672


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] github-actions[bot] commented on pull request #30672: [SPARK-33710][ Shuffle] [YARN] Shuffle index use guava cache OOM, Yarn NodeManager GC alarm

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #30672:
URL: https://github.com/apache/spark/pull/30672#issuecomment-803216100


   We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
   If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org