You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by hvanhovell <gi...@git.apache.org> on 2017/09/28 08:49:20 UTC

[GitHub] spark pull request #19378: [SPARK-22143][SQL][BRANCH-2.2] Fix memory leak in...

GitHub user hvanhovell opened a pull request:

    https://github.com/apache/spark/pull/19378

    [SPARK-22143][SQL][BRANCH-2.2] Fix memory leak in OffHeapColumnVector

    This is a backport of https://github.com/apache/spark/commit/02bb0682e68a2ce81f3b98d33649d368da7f2b3d.
    
    ## What changes were proposed in this pull request?
    `WriteableColumnVector` does not close its child column vectors. This can create memory leaks for `OffHeapColumnVector` where we do not clean up the memory allocated by a vectors children. This can be especially bad for string columns (which uses a child byte column vector).
    
    ## How was this patch tested?
    I have updated the existing tests to always use both on-heap and off-heap vectors. Testing and diagnosis was done locally.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/hvanhovell/spark SPARK-22143-2.2

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/19378.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #19378
    
----
commit c58a9e59a9bfb60ef85731b11213eba464c5076a
Author: Herman van Hovell <hv...@databricks.com>
Date:   2017-09-27T21:08:30Z

    [SPARK-22143][SQL] Fix memory leak in OffHeapColumnVector
    
    ## What changes were proposed in this pull request?
    `WriteableColumnVector` does not close its child column vectors. This can create memory leaks for `OffHeapColumnVector` where we do not clean up the memory allocated by a vectors children. This can be especially bad for string columns (which uses a child byte column vector).
    
    ## How was this patch tested?
    I have updated the existing tests to always use both on-heap and off-heap vectors. Testing and diagnoses was done locally.
    
    Author: Herman van Hovell <hv...@databricks.com>
    
    Closes #19367 from hvanhovell/SPARK-22143.
    
    # Conflicts:
    #	sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java
    #	sql/core/src/test/scala/org/apache/spark/sql/execution/vectorized/ColumnVectorSuite.scala
    #	sql/core/src/test/scala/org/apache/spark/sql/execution/vectorized/ColumnarBatchSuite.scala

commit 20540ecab64e367f3e30166545dc8725fb268a46
Author: Herman van Hovell <hv...@databricks.com>
Date:   2017-09-28T08:47:13Z

    Fix up ColumnarBatchSuite

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19378: [SPARK-22143][SQL][BRANCH-2.2] Fix memory leak in OffHea...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19378
  
    **[Test build #82273 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82273/testReport)** for PR 19378 at commit [`20540ec`](https://github.com/apache/spark/commit/20540ecab64e367f3e30166545dc8725fb268a46).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19378: [SPARK-22143][SQL][BRANCH-2.2] Fix memory leak in OffHea...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19378
  
    **[Test build #82279 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82279/testReport)** for PR 19378 at commit [`a82c494`](https://github.com/apache/spark/commit/a82c494e55c0d57875625c5cf72a9ca39bd608b8).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19378: [SPARK-22143][SQL][BRANCH-2.2] Fix memory leak in OffHea...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19378
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82273/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19378: [SPARK-22143][SQL][BRANCH-2.2] Fix memory leak in OffHea...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19378
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82279/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19378: [SPARK-22143][SQL][BRANCH-2.2] Fix memory leak in OffHea...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19378
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19378: [SPARK-22143][SQL][BRANCH-2.2] Fix memory leak in OffHea...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19378
  
    **[Test build #82279 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82279/testReport)** for PR 19378 at commit [`a82c494`](https://github.com/apache/spark/commit/a82c494e55c0d57875625c5cf72a9ca39bd608b8).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19378: [SPARK-22143][SQL][BRANCH-2.2] Fix memory leak in OffHea...

Posted by hvanhovell <gi...@git.apache.org>.

Github user hvanhovell commented on the issue:

    https://github.com/apache/spark/pull/19378
  
    Merging to 2.2


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19378: [SPARK-22143][SQL][BRANCH-2.2] Fix memory leak in OffHea...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19378
  
    **[Test build #82273 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82273/testReport)** for PR 19378 at commit [`20540ec`](https://github.com/apache/spark/commit/20540ecab64e367f3e30166545dc8725fb268a46).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org