You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Zhongshuai Pei (JIRA)" <ji...@apache.org> on 2016/06/18 01:48:05 UTC
[jira] [Comment Edited] (SPARK-15340) Limit the size of the map
used to cache JobConfs to void OOM
[ https://issues.apache.org/jira/browse/SPARK-15340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337421#comment-15337421 ]
Zhongshuai Pei edited comment on SPARK-15340 at 6/18/16 1:47 AM:
-----------------------------------------------------------------
[~clockfly]
1. I run in the cluster mode on YARN and use beeline
2. i run tpcds(500g and must be orc) and set driver.memory 30g
3. it is heap space OOM.you can run " jstat -gc pid" and will find the memory of old grow fast and will not be released
4. i run tpcds for 5 hours and OOM happened
was (Author: doingdone9):
[~clockfly]
1. I run in the cluster mode on YARN and use spark-sql
2. i run tpcds(500g and must be orc) and set driver.memory 30g
3. it is heap space OOM.you can run " jstat -gc pid" and will find the memory of old grow fast and will not be released
4. i run tpcds for 5 hours and OOM happened
> Limit the size of the map used to cache JobConfs to void OOM
> ------------------------------------------------------------
>
> Key: SPARK-15340
> URL: https://issues.apache.org/jira/browse/SPARK-15340
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.5.0, 1.6.0
> Reporter: Zhongshuai Pei
> Priority: Critical
>
> when i run tpcds (orc) by using JDBCServer, driver always OOM.
> i find tens of thousands of Jobconf from dump file and these JobConf can not be recycled, So we should limit the size of the map used to cache JobConfs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org