You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by 10110346 <gi...@git.apache.org> on 2018/03/20 04:11:43 UTC
[GitHub] spark pull request #20862: [SPARK-23744][CORE]Fix memory leak in ReadableCha...
GitHub user 10110346 opened a pull request:
https://github.com/apache/spark/pull/20862
[SPARK-23744][CORE]Fix memory leak in ReadableChannelFileRegion
## What changes were proposed in this pull request?
In the class `ReadableChannelFileRegion`, the `buffer` is direct memory, we should modify `deallocate` to free it, and `deallocate` will be called by `release`
## How was this patch tested?
existing unit test
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/10110346/spark leakmem
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20862.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20862
----
commit 5b58c57607551328c893a3857717e4b159ecf841
Author: liuxian <li...@...>
Date: 2018-03-20T03:19:59Z
fix
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20862: [SPARK-23744][CORE]Fix memory leak in ReadableCha...
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/20862#discussion_r178623333
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -301,7 +301,10 @@ private class ReadableChannelFileRegion(source: ReadableByteChannel, blockSize:
written
}
- override def deallocate(): Unit = source.close()
+ override def deallocate() {
+ source.close()
+ StorageUtils.dispose(buffer)
--- End diff --
Sorry for being late since I overlooked.
While `ByteBuffer.allocateDirect` allocates memory in off-heap memory, the memory will be also reclaimed when the ByteBuffer object will be reclaimed.
Regarding `-XX:+DisableExplicitGC`, as described [here](http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html), this option just disables `System.gc()`. Other GC operations may reclaim the memory in off-heap allocated by `ByteBuffer.allocateDirect`.
>By default calls to System.gc() are enabled (-XX:-DisableExplicitGC). Use -XX:+DisableExplicitGC to disable calls to System.gc(). Note that the JVM still performs garbage collection when necessary.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20862: [SPARK-23744][CORE]Fix memory leak in ReadableCha...
Posted by 10110346 <gi...@git.apache.org>.
Github user 10110346 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20862#discussion_r177993186
--- Diff: core/src/test/scala/org/apache/spark/storage/DiskStoreSuite.scala ---
@@ -197,7 +197,7 @@ class DiskStoreSuite extends SparkFunSuite {
while (region.transfered() < region.count()) {
region.transferTo(byteChannel, region.transfered())
}
-
+ region.release()
--- End diff --
I think we should release the direct memory in unit test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20862: [SPARK-23744][CORE]Fix memory leak in ReadableCha...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/20862#discussion_r177986757
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -301,7 +301,10 @@ private class ReadableChannelFileRegion(source: ReadableByteChannel, blockSize:
written
}
- override def deallocate(): Unit = source.close()
+ override def deallocate() {
+ source.close()
+ StorageUtils.dispose(buffer)
--- End diff --
IIRC direct byte buffer can be GCed, cc @kiszk
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20862: [SPARK-23744][CORE]Fix memory leak in ReadableChannelFil...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20862
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20862: [SPARK-23744][CORE]Fix memory leak in ReadableChannelFil...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20862
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88410/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20862: [SPARK-23744][CORE]Fix memory leak in ReadableChannelFil...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20862
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1629/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20862: [SPARK-23744][CORE]Fix memory leak in ReadableChannelFil...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20862
**[Test build #88402 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88402/testReport)** for PR 20862 at commit [`5b58c57`](https://github.com/apache/spark/commit/5b58c57607551328c893a3857717e4b159ecf841).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20862: [SPARK-23744][CORE]Fix memory leak in ReadableChannelFil...
Posted by 10110346 <gi...@git.apache.org>.
Github user 10110346 commented on the issue:
https://github.com/apache/spark/pull/20862
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20862: [SPARK-23744][CORE]Fix memory leak in ReadableChannelFil...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20862
**[Test build #88402 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88402/testReport)** for PR 20862 at commit [`5b58c57`](https://github.com/apache/spark/commit/5b58c57607551328c893a3857717e4b159ecf841).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20862: [SPARK-23744][CORE]Fix memory leak in ReadableCha...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/20862#discussion_r177986789
--- Diff: core/src/test/scala/org/apache/spark/storage/DiskStoreSuite.scala ---
@@ -197,7 +197,7 @@ class DiskStoreSuite extends SparkFunSuite {
while (region.transfered() < region.count()) {
region.transferTo(byteChannel, region.transfered())
}
-
+ region.release()
--- End diff --
why this change?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20862: [SPARK-23744][CORE]Fix memory leak in ReadableChannelFil...
Posted by 10110346 <gi...@git.apache.org>.
Github user 10110346 commented on the issue:
https://github.com/apache/spark/pull/20862
cc @cloud-fan
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20862: [SPARK-23744][CORE]Fix memory leak in ReadableChannelFil...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20862
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20862: [SPARK-23744][CORE]Fix memory leak in ReadableChannelFil...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20862
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1635/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20862: [SPARK-23744][CORE]Fix memory leak in ReadableChannelFil...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20862
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20862: [SPARK-23744][CORE]Fix memory leak in ReadableCha...
Posted by 10110346 <gi...@git.apache.org>.
Github user 10110346 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20862#discussion_r177992858
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -301,7 +301,10 @@ private class ReadableChannelFileRegion(source: ReadableByteChannel, blockSize:
written
}
- override def deallocate(): Unit = source.close()
+ override def deallocate() {
+ source.close()
+ StorageUtils.dispose(buffer)
--- End diff --
In fact, direct memory is also off-heap memory,
if we config this parameter "-XX:+DisableExplicitGC",the memory will not be reclaimed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20862: [SPARK-23744][CORE]Fix memory leak in ReadableCha...
Posted by 10110346 <gi...@git.apache.org>.
Github user 10110346 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20862#discussion_r178688855
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -301,7 +301,10 @@ private class ReadableChannelFileRegion(source: ReadableByteChannel, blockSize:
written
}
- override def deallocate(): Unit = source.close()
+ override def deallocate() {
+ source.close()
+ StorageUtils.dispose(buffer)
--- End diff --
I see,thanks
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20862: [SPARK-23744][CORE]Fix memory leak in ReadableChannelFil...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20862
**[Test build #88410 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88410/testReport)** for PR 20862 at commit [`5b58c57`](https://github.com/apache/spark/commit/5b58c57607551328c893a3857717e4b159ecf841).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20862: [SPARK-23744][CORE]Fix memory leak in ReadableCha...
Posted by 10110346 <gi...@git.apache.org>.
Github user 10110346 closed the pull request at:
https://github.com/apache/spark/pull/20862
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20862: [SPARK-23744][CORE]Fix memory leak in ReadableCha...
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/20862#discussion_r178626148
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -301,7 +301,10 @@ private class ReadableChannelFileRegion(source: ReadableByteChannel, blockSize:
written
}
- override def deallocate(): Unit = source.close()
+ override def deallocate() {
+ source.close()
+ StorageUtils.dispose(buffer)
--- End diff --
In other words, I think that the off-heap in `buffer` will not be leaked if `buffer` will be reclaimed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20862: [SPARK-23744][CORE]Fix memory leak in ReadableChannelFil...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20862
**[Test build #88410 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88410/testReport)** for PR 20862 at commit [`5b58c57`](https://github.com/apache/spark/commit/5b58c57607551328c893a3857717e4b159ecf841).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20862: [SPARK-23744][CORE]Fix memory leak in ReadableChannelFil...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20862
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20862: [SPARK-23744][CORE]Fix memory leak in ReadableChannelFil...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20862
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88402/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org