You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by DoingDone9 <gi...@git.apache.org> on 2016/05/16 09:13:30 UTC
[GitHub] spark pull request: [SPARK-15340][SQL]Limit the size of the map us...
GitHub user DoingDone9 opened a pull request:
https://github.com/apache/spark/pull/13130
[SPARK-15340][SQL]Limit the size of the map used to cache JobConfs to void OOM
# What changes were proposed in this pull request?
limit the size of the map used to cache JobConfs in SparkEnv.scala
# How was this patch tested?
Run tpcds(orc), the num of JobConf in dump file < 1000 and OOM not happened
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/DoingDone9/spark LimitNumOfMap
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13130.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13130
----
commit 6bba37cba181169d619c5defb127ad01c8cc944c
Author: Zhongshuai Pei <pe...@huawei.com>
Date: 2015-12-08T09:21:13Z
Merge pull request #1 from apache/master
pull
commit 1604d256741c5e21cff35fb1e1fc2ac69969b7bc
Author: Zhongshuai Pei <pe...@huawei.com>
Date: 2016-04-21T03:09:26Z
Merge pull request #2 from apache/master
pull
commit 0bde0b84dde93886e3bfd3fb9ae6a323c520807c
Author: Zhongshuai Pei <pe...@huawei.com>
Date: 2016-05-11T08:37:04Z
Merge pull request #3 from apache/master
pull
commit 29903c983fd18a0c41af422e9db3c445b08017bd
Author: Zhongshuai Pei <pe...@huawei.com>
Date: 2016-05-11T08:41:42Z
Limit the number of Map used for caching
commit 82d78a36161167c76aebf313ce9541ce51989948
Author: Zhongshuai Pei <pe...@huawei.com>
Date: 2016-05-16T08:45:55Z
Update HadoopRDD.scala
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13130: [SPARK-15340][SQL]Limit the size of the map used ...
Posted by DoingDone9 <gi...@git.apache.org>.
Github user DoingDone9 closed the pull request at:
https://github.com/apache/spark/pull/13130
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13130: [SPARK-15340][SQL]Limit the size of the map used to cach...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13130
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15340][SQL]Limit the size of the map us...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13130#issuecomment-219383243
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15340][SQL]Limit the size of the map us...
Posted by akohli <gi...@git.apache.org>.
Github user akohli commented on the pull request:
https://github.com/apache/spark/pull/13130#issuecomment-219998908
sure (i get that) but 1000 is an arbitrary number that may or may not sufficient. why not 10? why not 10000?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15340][SQL]Limit the size of the map us...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/13130#issuecomment-220497728
Do you know why it OOMs when it is softValues?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15340][SQL]Limit the size of the map us...
Posted by DoingDone9 <gi...@git.apache.org>.
Github user DoingDone9 commented on the pull request:
https://github.com/apache/spark/pull/13130#issuecomment-220273234
I refer to other codes using the maximumSize method @akohli
```
/** A cache of Spark SQL data source tables that have been accessed. */
protected[hive] val cachedDataSourceTables: LoadingCache[QualifiedTableName, LogicalPlan] = {
val cacheLoader = new CacheLoader[QualifiedTableName, LogicalPlan]() {
override def load(in: QualifiedTableName): LogicalPlan = {
logDebug(s"Creating new cached data source for $in")
getDatasourceRelation(in)
}
}
CacheBuilder.newBuilder().maximumSize(1000).build(cacheLoader)
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15340][SQL]Limit the size of the map us...
Posted by akohli <gi...@git.apache.org>.
Github user akohli commented on the pull request:
https://github.com/apache/spark/pull/13130#issuecomment-219750370
curious on a couple of things, firstly, you are using CacheBuilder with a cache size of 1000, is that sufficient? Wouldn't it be better to catch an exception or detect an error when a jobconf is added to the original map; alternatively is this masking an issue that proper GC is not happening for unreferenced cached configs?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15340][SQL]Limit the size of the map us...
Posted by DoingDone9 <gi...@git.apache.org>.
Github user DoingDone9 commented on the pull request:
https://github.com/apache/spark/pull/13130#issuecomment-222502337
The size of JobConf is about 124k and the memory of my dirver is 10g (> 124k * 10000 = 1.18g), so it is work. @rxin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13130: [SPARK-15340][SQL]Limit the size of the map used to cach...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/13130
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15340][SQL]Limit the size of the map us...
Posted by DoingDone9 <gi...@git.apache.org>.
Github user DoingDone9 commented on the pull request:
https://github.com/apache/spark/pull/13130#issuecomment-219905068
It's ok. If the size > 1000, the new values will replace the old values. @akohli
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15340][SQL]Limit the size of the map us...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/13130#discussion_r63334846
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -363,7 +363,7 @@ private[spark] object HadoopRDD extends Logging {
def containsCachedMetadata(key: String): Boolean = SparkEnv.get.hadoopJobMetadata.containsKey(key)
- private def putCachedMetadata(key: String, value: Any): Unit =
+ private def putCachedMetadata(key: String, value: Object): Unit =
--- End diff --
Could I ask why this should be `Object`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15340][SQL]Limit the size of the map us...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/13130#issuecomment-221161008
Does it work for you when you changed it to 10000 rather than 1000?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15340][SQL]Limit the size of the map us...
Posted by DoingDone9 <gi...@git.apache.org>.
Github user DoingDone9 commented on the pull request:
https://github.com/apache/spark/pull/13130#issuecomment-221146422
I'm confused about it too
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13130: [SPARK-15340][SQL]Limit the size of the map used to cach...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13130
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61142/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15340][SQL]Limit the size of the map us...
Posted by DoingDone9 <gi...@git.apache.org>.
Github user DoingDone9 commented on the pull request:
https://github.com/apache/spark/pull/13130#issuecomment-220497354
cc @yhuai @srowen
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13130: [SPARK-15340][SQL]Limit the size of the map used to cach...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13130
**[Test build #61142 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61142/consoleFull)** for PR 13130 at commit [`82d78a3`](https://github.com/apache/spark/commit/82d78a36161167c76aebf313ce9541ce51989948).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15340][SQL]Limit the size of the map us...
Posted by DoingDone9 <gi...@git.apache.org>.
Github user DoingDone9 commented on a diff in the pull request:
https://github.com/apache/spark/pull/13130#discussion_r63453073
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -363,7 +363,7 @@ private[spark] object HadoopRDD extends Logging {
def containsCachedMetadata(key: String): Boolean = SparkEnv.get.hadoopJobMetadata.containsKey(key)
- private def putCachedMetadata(key: String, value: Any): Unit =
+ private def putCachedMetadata(key: String, value: Object): Unit =
--- End diff --
CacheBuilder.newBuilder() return CacheBuilder<`Object`, `Object`>, i can only get map[String, `Object`] @HyukjinKwon
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13130: [SPARK-15340][SQL]Limit the size of the map used to cach...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13130
**[Test build #61142 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61142/consoleFull)** for PR 13130 at commit [`82d78a3`](https://github.com/apache/spark/commit/82d78a36161167c76aebf313ce9541ce51989948).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org