You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by dongjoon-hyun <gi...@git.apache.org> on 2018/08/20 06:26:01 UTC
[GitHub] spark pull request #22150: [SPARK-25144][SQL][TEST][BRANCH-2.3] Free aggrega...
GitHub user dongjoon-hyun opened a pull request:
https://github.com/apache/spark/pull/22150
[SPARK-25144][SQL][TEST][BRANCH-2.3] Free aggregate map when task ends
## What changes were proposed in this pull request?
[SPARK-25144](https://issues.apache.org/jira/browse/SPARK-25144) reports memory leaks on Apache Spark 2.1.3, 2.2.2, 2.3.2-RC5.
```scala
scala> case class Foo(bar: Option[String])
scala> val ds = List(Foo(Some("bar"))).toDS
scala> val result = ds.flatMap(_.bar).distinct
scala> result.rdd.isEmpty
18/08/19 23:01:54 WARN Executor: Managed memory leak detected; size = 8650752 bytes, TID = 125
res0: Boolean = false
```
This is a backport of @cloud-fan 's https://github.com/apache/spark/pull/21738 which is a single commit among 3 commits of SPARK-21743. In addition, I added a test case to prevent regressions in branch-2.3 and branch-2.2. Although SPARK-21743 is reverted due to regression, this subpatch can go to branch-2.3 and branch-2.2.
## How was this patch tested?
Pass the jenkins with a newly added test case.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjoon-hyun/spark SPARK-25144
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22150.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22150
----
commit 4355bf7f49b94dcfcfbe9df6be4c5c66cd0144e4
Author: Wenchen Fan <we...@...>
Date: 2018-07-10T15:07:10Z
[SPARK-25144][SQL] Free aggregate map when task ends
This is a backport of https://github.com/apache/spark/pull/21738.
Author: Wenchen Fan <we...@databricks.com>
Closes #21738 from cloud-fan/limit.
commit 5f7f4466fb6192bc1b45d76e7c3a514d75e482ab
Author: Dongjoon Hyun <do...@...>
Date: 2018-08-20T05:56:22Z
[SPARK-25144][SQL][TEST][BRANCH-2.3] Add a test case.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22150: [SPARK-25144][SQL][TEST][BRANCH-2.3] Free aggrega...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun closed the pull request at:
https://github.com/apache/spark/pull/22150
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22150: [SPARK-25144][SQL][TEST][BRANCH-2.3] Free aggregate map ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22150
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22150: [SPARK-25144][SQL][TEST][BRANCH-2.3] Free aggregate map ...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22150
Thank you all.
Sure, @HyukjinKwon !
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22150: [SPARK-25144][SQL][TEST][BRANCH-2.3] Free aggregate map ...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22150
LGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22150: [SPARK-25144][SQL][TEST][BRANCH-2.3] Free aggregate map ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22150
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94946/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22150: [SPARK-25144][SQL][TEST][BRANCH-2.3] Free aggregate map ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22150
**[Test build #94946 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94946/testReport)** for PR 22150 at commit [`5f7f446`](https://github.com/apache/spark/commit/5f7f4466fb6192bc1b45d76e7c3a514d75e482ab).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class Foo(bar: Option[String])`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22150: [SPARK-25144][SQL][TEST][BRANCH-2.3] Free aggregate map ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22150
**[Test build #94944 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94944/testReport)** for PR 22150 at commit [`5f7f446`](https://github.com/apache/spark/commit/5f7f4466fb6192bc1b45d76e7c3a514d75e482ab).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class Foo(bar: Option[String])`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22150: [SPARK-25144][SQL][TEST][BRANCH-2.3] Free aggregate map ...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22150
retest this please.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22150: [SPARK-25144][SQL][TEST][BRANCH-2.3] Free aggregate map ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22150
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22150: [SPARK-25144][SQL][TEST][BRANCH-2.3] Free aggregate map ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22150
LGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22150: [SPARK-25144][SQL][TEST][BRANCH-2.3] Free aggregate map ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22150
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22150: [SPARK-25144][SQL][TEST][BRANCH-2.3] Free aggregate map ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22150
Merged to branch-2.2 and branch-2.3.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22150: [SPARK-25144][SQL][TEST][BRANCH-2.3] Free aggregate map ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22150
**[Test build #94944 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94944/testReport)** for PR 22150 at commit [`5f7f446`](https://github.com/apache/spark/commit/5f7f4466fb6192bc1b45d76e7c3a514d75e482ab).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22150: [SPARK-25144][SQL][TEST][BRANCH-2.3] Free aggregate map ...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22150
@cloud-fan . Could you review this PR? This is effectively, a cherry-pick of your commit + a small test code.
Also, cc @gatorsmile , @HyukjinKwon , @viirya .
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22150: [SPARK-25144][SQL][TEST][BRANCH-2.3] Free aggregate map ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22150
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2313/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22150: [SPARK-25144][SQL][TEST][BRANCH-2.3] Free aggregate map ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22150
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2312/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22150: [SPARK-25144][SQL][TEST][BRANCH-2.3] Free aggregate map ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22150
Merged to branch-2.3 only. It has a conflict to branch-2.2. @dongjoon-hyun mind opening another PR for it too please?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22150: [SPARK-25144][SQL][TEST][BRANCH-2.3] Free aggregate map ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22150
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22150: [SPARK-25144][SQL][TEST][BRANCH-2.3] Free aggregate map ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22150
**[Test build #94946 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94946/testReport)** for PR 22150 at commit [`5f7f446`](https://github.com/apache/spark/commit/5f7f4466fb6192bc1b45d76e7c3a514d75e482ab).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22150: [SPARK-25144][SQL][TEST][BRANCH-2.3] Free aggregate map ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22150
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94944/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22150: [SPARK-25144][SQL][TEST][BRANCH-2.3] Free aggregate map ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22150
Also can you add the same test in master branch too?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org