You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by dongjoon-hyun <gi...@git.apache.org> on 2018/08/20 16:20:00 UTC

[GitHub] spark pull request #22156: [SPARK-25144][SQL][TEST][BRANCH-2.2] Free aggrega...

GitHub user dongjoon-hyun opened a pull request:

    https://github.com/apache/spark/pull/22156

    [SPARK-25144][SQL][TEST][BRANCH-2.2] Free aggregate map when task ends

    ## What changes were proposed in this pull request?
    
    [SPARK-25144](https://issues.apache.org/jira/browse/SPARK-25144) reports memory leaks on Apache Spark 2.0.2 ~ 2.3.2-RC5.
    
    ```scala
    scala> case class Foo(bar: Option[String])
    scala> val ds = List(Foo(Some("bar"))).toDS
    scala> val result = ds.flatMap(_.bar).distinct
    scala> result.rdd.isEmpty
    18/08/19 23:01:54 WARN Executor: Managed memory leak detected; size = 8650752 bytes, TID = 125
    res0: Boolean = false
    ```
    
    This is a backport of cloud-fan 's https://github.com/apache/spark/pull/21738 which is a single commit among 3 commits of SPARK-21743. In addition, I added a test case to prevent regressions in branch-2.3 and branch-2.2. Although SPARK-21743 is reverted due to regression, this subpatch can go to branch-2.3 and branch-2.2. This will be merged as cloud-fan 's commit.
    
    ## How was this patch tested?
    
    Pass the jenkins with a newly added test case.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/dongjoon-hyun/spark SPARK-25144-2.2

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/22156.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #22156
    
----
commit 27dea91f1126ae4b575246d1e17410e79042e9e1
Author: Wenchen Fan <we...@...>
Date:   2018-08-20T12:44:22Z

    [SPARK-25144][SQL][TEST][BRANCH-2.2] Free aggregate map when task ends
    
    [SPARK-25144](https://issues.apache.org/jira/browse/SPARK-25144) reports memory leaks on Apache Spark 2.0.2 ~ 2.3.2-RC5.
    
    ```scala
    scala> case class Foo(bar: Option[String])
    scala> val ds = List(Foo(Some("bar"))).toDS
    scala> val result = ds.flatMap(_.bar).distinct
    scala> result.rdd.isEmpty
    18/08/19 23:01:54 WARN Executor: Managed memory leak detected; size = 8650752 bytes, TID = 125
    res0: Boolean = false
    ```
    
    This is a backport of cloud-fan 's https://github.com/apache/spark/pull/21738 which is a single commit among 3 commits of SPARK-21743. In addition, I added a test case to prevent regressions in branch-2.3 and branch-2.2. Although SPARK-21743 is reverted due to regression, this subpatch can go to branch-2.3 and branch-2.2. This will be merged as cloud-fan 's commit.
    
    Pass the jenkins with a newly added test case.
    
    Closes #22150 from dongjoon-hyun/SPARK-25144.
    
    Lead-authored-by: Wenchen Fan <we...@databricks.com>
    Co-authored-by: Dongjoon Hyun <do...@apache.org>
    Signed-off-by: hyukjinkwon <gu...@apache.org>

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22156: [SPARK-25144][SQL][TEST][BRANCH-2.2] Free aggregate map ...

Posted by dongjoon-hyun <gi...@git.apache.org>.

Github user dongjoon-hyun commented on the issue:

    https://github.com/apache/spark/pull/22156
  
    Thank you, @HyukjinKwon .


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22156: [SPARK-25144][SQL][TEST][BRANCH-2.2] Free aggregate map ...

Posted by dongjoon-hyun <gi...@git.apache.org>.

Github user dongjoon-hyun commented on the issue:

    https://github.com/apache/spark/pull/22156
  
    cc @HyukjinKwon 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22156: [SPARK-25144][SQL][TEST][BRANCH-2.2] Free aggregate map ...

Posted by HyukjinKwon <gi...@git.apache.org>.

Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/22156
  
    Merged to branch-2.2.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22156: [SPARK-25144][SQL][TEST][BRANCH-2.2] Free aggregate map ...

Posted by dongjoon-hyun <gi...@git.apache.org>.

Github user dongjoon-hyun commented on the issue:

    https://github.com/apache/spark/pull/22156
  
    Thank you for review and approval, @HyukjinKwon .


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22156: [SPARK-25144][SQL][TEST][BRANCH-2.2] Free aggregate map ...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22156
  
    **[Test build #94963 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94963/testReport)** for PR 22156 at commit [`27dea91`](https://github.com/apache/spark/commit/27dea91f1126ae4b575246d1e17410e79042e9e1).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22156: [SPARK-25144][SQL][TEST][BRANCH-2.2] Free aggregate map ...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22156
  
    **[Test build #94963 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94963/testReport)** for PR 22156 at commit [`27dea91`](https://github.com/apache/spark/commit/27dea91f1126ae4b575246d1e17410e79042e9e1).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22156: [SPARK-25144][SQL][TEST][BRANCH-2.2] Free aggregate map ...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22156
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94963/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org