You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/17 14:57:38 UTC

[GitHub] [spark] tomvanbussel opened a new pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

tomvanbussel opened a new pull request #29787:
URL: https://github.com/apache/spark/pull/29787


   ### What changes were proposed in this pull request?
   
   This PR changes `UnsafeExternalSorter.SpillableIterator` to free its memory (except for the page holding the last record) if it is forced to spill after all of its records have been read. It also makes sure that `lastPage` is freed if `loadNext` is never called the again. The latter was necessary to get my test case to succeed (otherwise it would complain about a leak).
   
   ### Why are the changes needed?
   
   No memory is freed after calling `UnsafeExternalSorter.SpillableIterator.spill()` when all records have been read, even though it is still holding onto some memory. This may cause a `SparkOutOfMemoryError` to be thrown, even though we could have just freed the memory instead.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No
   
   ### How was this patch tested?
   
   A test was added to `UnsafeExternalSorterSuite`.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan closed pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #29787:
URL: https://github.com/apache/spark/pull/29787


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29787:
URL: https://github.com/apache/spark/pull/29787#discussion_r490734570



##########
File path: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java
##########
@@ -527,19 +527,23 @@ public long getCurrentPageNumber() {
 
     public long spill() throws IOException {
       synchronized (this) {
-        if (inMemSorter == null || numRecords <= 0) {
+        if (inMemSorter == null) {
           return 0L;
         }
 
         long currentPageNumber = upstream.getCurrentPageNumber();
 
         ShuffleWriteMetrics writeMetrics = new ShuffleWriteMetrics();
-        // Iterate over the records that have not been returned and spill them.
-        final UnsafeSorterSpillWriter spillWriter =
-          new UnsafeSorterSpillWriter(blockManager, fileBufferSizeBytes, writeMetrics, numRecords);
-        spillIterator(upstream, spillWriter);
-        spillWriters.add(spillWriter);
-        upstream = spillWriter.getReader(serializerManager);
+        if (numRecords > 0) {
+          // Iterate over the records that have not been returned and spill them.
+          final UnsafeSorterSpillWriter spillWriter = new UnsafeSorterSpillWriter(
+                  blockManager, fileBufferSizeBytes, writeMetrics, numRecords);
+          spillIterator(upstream, spillWriter);
+          spillWriters.add(spillWriter);
+          upstream = spillWriter.getReader(serializerManager);
+        } else {
+          upstream = null;

Review comment:
       when all records have been read, the `numRecords` will be 0, and we will reach the else branch here, which doesn't do spill either?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29787:
URL: https://github.com/apache/spark/pull/29787#issuecomment-694822480


   cc @Ngone51 for the flaky test above: `org.apache.spark.scheduler.BarrierTaskContextSuite.support multiple barrier() call within a single task`


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29787:
URL: https://github.com/apache/spark/pull/29787#issuecomment-694696583


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128859/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29787:
URL: https://github.com/apache/spark/pull/29787#issuecomment-694682863






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] tomvanbussel commented on pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
tomvanbussel commented on pull request #29787:
URL: https://github.com/apache/spark/pull/29787#issuecomment-694295729


   cc @hvanhovell @cloud-fan 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29787:
URL: https://github.com/apache/spark/pull/29787#issuecomment-694796366


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128868/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29787:
URL: https://github.com/apache/spark/pull/29787#issuecomment-694724254






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29787:
URL: https://github.com/apache/spark/pull/29787#issuecomment-694696478






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29787:
URL: https://github.com/apache/spark/pull/29787#issuecomment-694294426


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29787:
URL: https://github.com/apache/spark/pull/29787#issuecomment-694682339


   **[Test build #128859 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128859/testReport)** for PR 29787 at commit [`8598c09`](https://github.com/apache/spark/commit/8598c0987c345ff0e72e28555addd325ae9ed762).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29787:
URL: https://github.com/apache/spark/pull/29787#issuecomment-694680501


   ok to test


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29787:
URL: https://github.com/apache/spark/pull/29787#issuecomment-694682190


   add to whitelist


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29787:
URL: https://github.com/apache/spark/pull/29787#issuecomment-694298119


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29787:
URL: https://github.com/apache/spark/pull/29787#issuecomment-694682339






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29787:
URL: https://github.com/apache/spark/pull/29787#issuecomment-694696549






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29787:
URL: https://github.com/apache/spark/pull/29787#issuecomment-694696549






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29787:
URL: https://github.com/apache/spark/pull/29787#issuecomment-694724254






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29787:
URL: https://github.com/apache/spark/pull/29787#issuecomment-694796359


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] tomvanbussel commented on a change in pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
tomvanbussel commented on a change in pull request #29787:
URL: https://github.com/apache/spark/pull/29787#discussion_r490774220



##########
File path: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java
##########
@@ -527,19 +527,23 @@ public long getCurrentPageNumber() {
 
     public long spill() throws IOException {
       synchronized (this) {
-        if (inMemSorter == null || numRecords <= 0) {
+        if (inMemSorter == null) {
           return 0L;
         }
 
         long currentPageNumber = upstream.getCurrentPageNumber();
 
         ShuffleWriteMetrics writeMetrics = new ShuffleWriteMetrics();
-        // Iterate over the records that have not been returned and spill them.
-        final UnsafeSorterSpillWriter spillWriter =
-          new UnsafeSorterSpillWriter(blockManager, fileBufferSizeBytes, writeMetrics, numRecords);
-        spillIterator(upstream, spillWriter);
-        spillWriters.add(spillWriter);
-        upstream = spillWriter.getReader(serializerManager);
+        if (numRecords > 0) {
+          // Iterate over the records that have not been returned and spill them.
+          final UnsafeSorterSpillWriter spillWriter = new UnsafeSorterSpillWriter(
+                  blockManager, fileBufferSizeBytes, writeMetrics, numRecords);
+          spillIterator(upstream, spillWriter);
+          spillWriters.add(spillWriter);
+          upstream = spillWriter.getReader(serializerManager);
+        } else {
+          upstream = null;

Review comment:
       There is nothing to spill to disk here, as all records have already been read, but we still need to free the memory. Before this PR we would keep the memory allocated.
   Added a comment to the code to clarify this.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29787:
URL: https://github.com/apache/spark/pull/29787#issuecomment-694691012


   **[Test build #128860 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128860/testReport)** for PR 29787 at commit [`8598c09`](https://github.com/apache/spark/commit/8598c0987c345ff0e72e28555addd325ae9ed762).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29787:
URL: https://github.com/apache/spark/pull/29787#issuecomment-694723708


   **[Test build #128868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128868/testReport)** for PR 29787 at commit [`1728eb3`](https://github.com/apache/spark/commit/1728eb32621ee9fcf1c05fd4825f5bf2623bf2f0).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29787:
URL: https://github.com/apache/spark/pull/29787#issuecomment-694795728


   **[Test build #128868 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128868/testReport)** for PR 29787 at commit [`1728eb3`](https://github.com/apache/spark/commit/1728eb32621ee9fcf1c05fd4825f5bf2623bf2f0).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29787:
URL: https://github.com/apache/spark/pull/29787#issuecomment-694298119


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29787:
URL: https://github.com/apache/spark/pull/29787#issuecomment-694822827


   github action passed, thanks, merging to master!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29787:
URL: https://github.com/apache/spark/pull/29787#issuecomment-694294426


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29787:
URL: https://github.com/apache/spark/pull/29787#issuecomment-694723708


   **[Test build #128868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128868/testReport)** for PR 29787 at commit [`1728eb3`](https://github.com/apache/spark/commit/1728eb32621ee9fcf1c05fd4825f5bf2623bf2f0).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29787:
URL: https://github.com/apache/spark/pull/29787#issuecomment-694682863






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29787:
URL: https://github.com/apache/spark/pull/29787#issuecomment-694796359






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] tomvanbussel commented on a change in pull request #29787: [SPARK-32911][CORE] Free memory in UnsafeExternalSorter.SpillableIterator.spill() when all records have been read.

Posted by GitBox <gi...@apache.org>.
tomvanbussel commented on a change in pull request #29787:
URL: https://github.com/apache/spark/pull/29787#discussion_r490322471



##########
File path: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java
##########
@@ -581,14 +590,15 @@ public void loadNext() throws IOException {
       try {
         synchronized (this) {
           loaded = true;
-          // Just consumed the last record from in memory iterator
+          // Just consumed the last record from the in-memory iterator.
           if (lastPage != null) {
             // Do not free the page here, while we are locking `SpillableIterator`. The `freePage`
             // method locks the `TaskMemoryManager`, and it's a bad idea to lock 2 objects in
             // sequence. We may hit dead lock if another thread locks `TaskMemoryManager` and
             // `SpillableIterator` in sequence, which may happen in
             // `TaskMemoryManager.acquireExecutionMemory`.
             pageToFree = lastPage;
+            allocatedPages.clear();

Review comment:
       `spill()` synchronizes on `UnsafeExternalSorter.this` when accessing `allocatedPages`. I presume this is to avoid concurrency issues with `cleanupResources`. However, I think we're pretty much screwed here anyway if there's a concurrent call to `cleanupResources` as that will free the page of the record that we are loading here.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org