You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by heary-cao <gi...@git.apache.org> on 2017/05/05 06:20:02 UTC
[GitHub] spark pull request #17869: [CORE]Run the SortShuffleSuite unit tests have re...
GitHub user heary-cao opened a pull request:
https://github.com/apache/spark/pull/17869
[CORE]Run the SortShuffleSuite unit tests have residual spark_* system directory
## What changes were proposed in this pull request?
This PR solution to run the SortShuffleSuite unit tests have residual spark_* system directory
For example:
OS:Windows 7
After the running SortShuffleSuite unit tests,
the system of TMP directory have '..\AppData\Local\Temp\spark-f64121f9-11b4-4ffd-a4f0-cfca66643503' not deleted
## How was this patch tested?
Run SortShuffleSuite unit test.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/heary-cao/spark SortShuffleSuite
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17869.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17869
----
commit 5effbeb99d05c93a71f78452c793aca0be62ae89
Author: caoxuewen <ca...@zte.com.cn>
Date: 2017-05-05T06:18:03Z
Run the SortShuffleSuite unit tests have residual spark_* system directory
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit ...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on a diff in the pull request:
https://github.com/apache/spark/pull/17869#discussion_r115424961
--- Diff: mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala ---
@@ -774,6 +774,7 @@ class ALSCleanerSuite extends SparkFunSuite {
} finally {
Utils.deleteRecursively(localDir)
Utils.deleteRecursively(checkpointDir)
+ Utils.clearLocalRootDirs()
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on the issue:
https://github.com/apache/spark/pull/17869
@HyukjinKwon
I suggest to add to the beforeAll。
If the added beforeEach, Most of the unit tests will run the Utils.clearLocalRootDirs() twice.
What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17869
@heary-cao to make sure, are few test cases in `SortShuffleSuite` being failed in your local environment because these were unable to be deleted, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17869
@heary-cao, it seems AppVeyor tests were fine before/after this PR. Is this a flaky test fix or did I misunderstand what this PR targets?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/17869#discussion_r115120038
--- Diff: core/src/test/scala/org/apache/spark/SortShuffleSuite.scala ---
@@ -39,12 +39,12 @@ class SortShuffleSuite extends ShuffleSuite with BeforeAndAfterAll {
override def beforeAll() {
super.beforeAll()
conf.set("spark.shuffle.manager", "sort")
+ conf.set("spark.local.dir", Utils.createTempDir().getAbsolutePath)
}
override def beforeEach(): Unit = {
super.beforeEach()
tempDir = Utils.createTempDir()
- conf.set("spark.local.dir", tempDir.getAbsolutePath)
--- End diff --
@heary-cao I am sorry, probably I didn't understand yet. Here it looks we create a temp directory, `tempDir`, before a test and remove `tempDir` after a test recusively. How come a directory could leave undeleted?
Or do you mean actually the test is failed because it fails to remove the temp directory as they have exclusive lock only on Windows?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17869
Test results are fine. I think my question is why this happens and how the current change solves the issue above. Another question is, does second run of the test cases fail due to this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on the issue:
https://github.com/apache/spark/pull/17869
@HyukjinKwon
You are right,
add Utils.clearLocalRootDirs() in the first of beforeEach or in the last of afterEach, The results are correct.
I choose to add Utils.clearLocalRootDirs() in the last of afterEach. It looks much more logical.
In addition, ALSCleanerSuite test set of the same problems.
so modify it.
please review again.
thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17869
**[Test build #3726 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3726/testReport)** for PR 17869 at commit [`e901625`](https://github.com/apache/spark/commit/e9016258978908dcc317700330ec61eae6d5439d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17869
I think that's fine. It should be safe.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/17869#discussion_r115417588
--- Diff: mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala ---
@@ -774,6 +774,7 @@ class ALSCleanerSuite extends SparkFunSuite {
} finally {
Utils.deleteRecursively(localDir)
Utils.deleteRecursively(checkpointDir)
+ Utils.clearLocalRootDirs()
--- End diff --
Could we add before/after for each likewise?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17869
@heary-cao, could you try to add `Utils.clearLocalRootDirs()` in the first of `beforeEach` and in the last of `afterEach` and check out the results?
It looks initial local directory is cached in `Utils.localRootDirs` and then it keeps returning this directory set before even after setting `conf.set("spark.local.dir", tempDir.getAbsolutePath)`.
Also, I quickly skimmed after running `grep -r "spark.local.dir" . | grep Suite | grep ".scala"` and I see two suspicious instances as below:
```
./mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala: conf.set("spark.local.dir", localDir.getAbsolutePath)
./mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala: conf.set("spark.local.dir", localDir.getAbsolutePath)
```
Could you check out both if you have some time?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17869
**[Test build #3744 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3744/testReport)** for PR 17869 at commit [`e901625`](https://github.com/apache/spark/commit/e9016258978908dcc317700330ec61eae6d5439d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17869
**[Test build #3740 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3740/testReport)** for PR 17869 at commit [`e901625`](https://github.com/apache/spark/commit/e9016258978908dcc317700330ec61eae6d5439d).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17869
**[Test build #3740 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3740/testReport)** for PR 17869 at commit [`e901625`](https://github.com/apache/spark/commit/e9016258978908dcc317700330ec61eae6d5439d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/17869#discussion_r115402836
--- Diff: core/src/test/scala/org/apache/spark/SortShuffleSuite.scala ---
@@ -50,6 +50,7 @@ class SortShuffleSuite extends ShuffleSuite with BeforeAndAfterAll {
override def afterEach(): Unit = {
try {
Utils.deleteRecursively(tempDir)
+ Utils.clearLocalRootDirs()
--- End diff --
Let's add this in `beforeEach`. This might already be initialised in previous tests.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17869
Merged to master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17869
**[Test build #3747 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3747/testReport)** for PR 17869 at commit [`e901625`](https://github.com/apache/spark/commit/e9016258978908dcc317700330ec61eae6d5439d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/17869
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit ...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on a diff in the pull request:
https://github.com/apache/spark/pull/17869#discussion_r115425030
--- Diff: mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala ---
@@ -774,6 +774,7 @@ class ALSCleanerSuite extends SparkFunSuite {
} finally {
Utils.deleteRecursively(localDir)
Utils.deleteRecursively(checkpointDir)
+ Utils.clearLocalRootDirs()
--- End diff --
ok,
thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17869
I guess these were unable to be deleted on Windows as some files are not closed. Shouldn't we find where the resources are not being closed rather leaving them undeleted?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on the issue:
https://github.com/apache/spark/pull/17869
@srowen ,
I commit to modify the PR.
Can you help me to run `test build` again.
thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on the issue:
https://github.com/apache/spark/pull/17869
@HyukjinKwon
Run the ShuffleNettySuite unit tests, no residual Temp directory.
Run the SortShuffleSuite unit tests ,Will remain a temp directory not deleted.
thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17869
**[Test build #3747 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3747/testReport)** for PR 17869 at commit [`e901625`](https://github.com/apache/spark/commit/e9016258978908dcc317700330ec61eae6d5439d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17869
**[Test build #3705 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3705/testReport)** for PR 17869 at commit [`11e7f21`](https://github.com/apache/spark/commit/11e7f2133a66a5f14c32d01401d62f5a4f735ae4).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17869
Hi @srowen, this looks reasonable to me and I remember my similar PR was reviewed by you. Please let me cc you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/17869#discussion_r115428147
--- Diff: mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala ---
@@ -774,6 +774,8 @@ class ALSCleanerSuite extends SparkFunSuite {
} finally {
Utils.deleteRecursively(localDir)
Utils.deleteRecursively(checkpointDir)
+ // Clear the local cache directory
+ Utils.clearLocalRootDirs()
--- End diff --
I am sorry, I had to be clear ... I meant adding `afterEach` and `beforeAll` similarly ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17869
**[Test build #3744 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3744/testReport)** for PR 17869 at commit [`e901625`](https://github.com/apache/spark/commit/e9016258978908dcc317700330ec61eae6d5439d).
* This patch passes all tests.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on the issue:
https://github.com/apache/spark/pull/17869
@HyukjinKwon
I suggest to add to the beforeAll。
If the added beforeEach, Most of the unit tests will run twice.
What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17869
**[Test build #3736 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3736/testReport)** for PR 17869 at commit [`e901625`](https://github.com/apache/spark/commit/e9016258978908dcc317700330ec61eae6d5439d).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on the issue:
https://github.com/apache/spark/pull/17869
@HyukjinKwon
does second run of the test cases not fail,
all test cases is ok.
Different is all use cases after the operation, will recusively a spark_ TMP directory is not delete.
Here actually I also not understand, why to perform after the first test case, we remove the spark_ directory,
```
remove:C:\Users\Default\AppData\Local\Temp\spark-6d46f183-bb26-4f98-a50a-e19b2068c15a\spark-c0db2905-02f1-4bbb-811b-3cf910dee7cf
remove:C:\Users\Default\AppData\Local\Temp\spark-6d46f183-bb26-4f98-a50a-e19b2068c15a
```
but after the second run test cases, continue to delete the directory
`remove:C:\Users\Default\AppData\Local\Temp\spark-6d46f183-bb26-4f98-a50a-e19b2068c15a\spark-c0db2905-02f1-4bbb-811b-3cf910dee7cf\userFiles-76f038fe-4e25-4622-b8c1-74481ffb6eb9`
Because the caused by `userFiles-76f038fe-4e25-4622-b8c1-74481ffb6eb9` directory ?
thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/17869#discussion_r115120048
--- Diff: core/src/test/scala/org/apache/spark/SortShuffleSuite.scala ---
@@ -39,12 +39,12 @@ class SortShuffleSuite extends ShuffleSuite with BeforeAndAfterAll {
override def beforeAll() {
super.beforeAll()
conf.set("spark.shuffle.manager", "sort")
+ conf.set("spark.local.dir", Utils.createTempDir().getAbsolutePath)
--- End diff --
And I didn't understand yet how this solves this issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17869
**[Test build #3705 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3705/testReport)** for PR 17869 at commit [`11e7f21`](https://github.com/apache/spark/commit/11e7f2133a66a5f14c32d01401d62f5a4f735ae4).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17869
Hi @heary-cao, I sent a PR to your branch - https://github.com/heary-cao/spark/pull/1 for minor cleanup. Please review that one and merge it so that the commits are pushed into this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17869
Build started: [CORE] `org.apache.spark.SortShuffleSuite` [](https://ci.appveyor.com/project/spark-test/spark/branch/1E2C6759-E0C7-40D4-AAA2-9FFC1934AC62)
Diff: https://github.com/apache/spark/compare/master...spark-test:1E2C6759-E0C7-40D4-AAA2-9FFC1934AC62
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on the issue:
https://github.com/apache/spark/pull/17869
@HyukjinKwon
I would like to suggest that
modify ALSCleanerSuite with another PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17869
(it looks the test failure is unrelated to me)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on the issue:
https://github.com/apache/spark/pull/17869
To run again?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on the issue:
https://github.com/apache/spark/pull/17869
@HyukjinKwon
ok
thanks.
I wish you the best
A happy holiday.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on the issue:
https://github.com/apache/spark/pull/17869
@HyukjinKwon
I agree with your point of view. So I again tested.
When running the first unit test, will set the spark. Local. Dir. At the end, this directory will be deleted.
Delete` ..\spark-dd832eb6-0c82-4354-b48e-917d82b7f3be\spark-3487446f-0d8d-46cc-8ff5-b3115ded595d\userFiles-cc48f5f4-b6a3-4e31-a6ff-a0175df1d2c0` file at the same time.
but Run the second unit tests, the reset will set the spark. Local. Dir. At the end, this directory will be deleted again. at the same time delete `..\spark-dd832eb6-0c82-4354-b48e-917d82b7f3be\spark-3487446f-0d8d-46cc-8ff5-b3115ded595d\userFiles-d07214c1-f04c-4e08-a256-c1f2f6f9b529` file.
The file directory is created the first unit test run time.
So the residual directory is `..\spark-dd832eb6-0c82-4354-b48e-917d82b7f3be\spark-3487446f-0d8d-46cc-8ff5-b3115ded595d.`
I modify the code again. please review it again.
thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17869
**[Test build #3736 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3736/testReport)** for PR 17869 at commit [`e901625`](https://github.com/apache/spark/commit/e9016258978908dcc317700330ec61eae6d5439d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [CORE]Run the SortShuffleSuite unit tests have residual ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17869
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on the issue:
https://github.com/apache/spark/pull/17869
@HyukjinKwon
this changes before and after deleting files log information:
**Before changes**
> Run the unit test: SortShuffleManager properly cleans up files for shuffles that use the serialized path
```
remove:C:\Users\Default\AppData\Local\Temp\spark-6d46f183-bb26-4f98-a50a-e19b2068c15a\blockmgr-9c553fa3-dfc9-435d-8624-ab48961c8248\0c
remove:C:\Users\Default\AppData\Local\Temp\spark-6d46f183-bb26-4f98-a50a-e19b2068c15a\blockmgr-9c553fa3-dfc9-435d-8624-ab48961c8248\0d
...
remove:C:\Users\Default\AppData\Local\Temp\spark-6d46f183-bb26-4f98-a50a-e19b2068c15a\blockmgr-9c553fa3-dfc9-435d-8624-ab48961c8248
remove:C:\Users\Default\AppData\Local\Temp\spark-6d46f183-bb26-4f98-a50a-e19b2068c15a\spark-c0db2905-02f1-4bbb-811b-3cf910dee7cf\userFiles-d8906c18-17ab-41e7-917e-11cd7c71243e
remove:C:\Users\Default\AppData\Local\Temp\spark-6d46f183-bb26-4f98-a50a-e19b2068c15a\spark-c0db2905-02f1-4bbb-811b-3cf910dee7cf
remove:C:\Users\Default\AppData\Local\Temp\spark-6d46f183-bb26-4f98-a50a-e19b2068c15a
```
> Run the unit test: SortShuffleManager properly cleans up files for shuffles that use the deserialized path
```
remove:C:\Users\Default\AppData\Local\Temp\spark-371eb96b-738f-4f6f-97c9-7a694afc34cc\blockmgr-0691abbf-61b8-41eb-9476-92479d9209d2\0c
remove:C:\Users\Default\AppData\Local\Temp\spark-371eb96b-738f-4f6f-97c9-7a694afc34cc\blockmgr-0691abbf-61b8-41eb-9476-92479d9209d2\0d
...
remove:C:\Users\Default\AppData\Local\Temp\spark-371eb96b-738f-4f6f-97c9-7a694afc34cc\blockmgr-0691abbf-61b8-41eb-9476-92479d9209d2
remove:C:\Users\Default\AppData\Local\Temp\spark-371eb96b-738f-4f6f-97c9-7a694afc34cc
remove:C:\Users\Default\AppData\Local\Temp\spark-6d46f183-bb26-4f98-a50a-e19b2068c15a\spark-c0db2905-02f1-4bbb-811b-3cf910dee7cf\userFiles-76f038fe-4e25-4622-b8c1-74481ffb6eb9
```
_residual system directory is_ `C:\Users\Default\AppData\Local\Temp\spark-6d46f183-bb26-4f98-a50a-e19b2068c15a\spark-c0db2905-02f1-4bbb-811b-3cf910dee7cf`
**After changes**
> Run the unit test: SortShuffleManager properly cleans up files for shuffles that use the serialized path
```
remove:C:\Users\Default\AppData\Local\Temp\spark-c2a131f1-08a5-428c-bed4-e29cfcacbbb9
remove:C:\Users\Default\AppData\Local\Temp\spark-bda0e806-6394-4180-8579-05a14c9aa83b\spark-3a7708e1-56fb-40ed-890f-6a8cc80d7473\userFiles-68974f6c-d4fc-4f8a-b66f-01a25ed12c44
```
> Run the unit test: SortShuffleManager properly cleans up files for shuffles that use the deserialized path
```
remove:C:\Users\Default\AppData\Local\Temp\spark-07ef44ee-c165-4637-a52d-daf2f8780a50
remove:C:\Users\Default\AppData\Local\Temp\spark-bda0e806-6394-4180-8579-05a14c9aa83b\spark-3a7708e1-56fb-40ed-890f-6a8cc80d7473\userFiles-19e1cd23-077d-4d05-a676-6d3f8f2cfa39
remove:C:\Users\Default\AppData\Local\Temp\spark-bda0e806-6394-4180-8579-05a14c9aa83b\spark-3a7708e1-56fb-40ed-890f-6a8cc80d7473
remove:C:\Users\Default\AppData\Local\Temp\spark-bda0e806-6394-4180-8579-05a14c9aa83b\blockmgr-73e600e1-0ff7-4d79-8da0-ffac572be0d3\0c
...
remove:C:\Users\Default\AppData\Local\Temp\spark-bda0e806-6394-4180-8579-05a14c9aa83b\blockmgr-73e600e1-0ff7-4d79-8da0-ffac572be0d3
remove:C:\Users\Default\AppData\Local\Temp\spark-bda0e806-6394-4180-8579-05a14c9aa83b\blockmgr-815208b6-c080-4707-8553-a5cec9709e1e\0c
...
remove:C:\Users\Default\AppData\Local\Temp\spark-bda0e806-6394-4180-8579-05a14c9aa83b\blockmgr-815208b6-c080-4707-8553-a5cec9709e1e
remove:C:\Users\Default\AppData\Local\Temp\spark-bda0e806-6394-4180-8579-05a14c9aa83b
```
So, after changes, remove file sequence has also changed.
thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17869
I think this does not look too urgent to merge it without investigating the reason and it is inappropriate to submit a PR without knowing/explaining how it fixes.
I suspect this is related with local dir cache and my guts say this is not a proper way to deal with it.
If you are unable to explain why it happens and how the current change fixes it, I would rather suggest to close this.
I can take a look maybe within tonight or tomorrow instead (I have been on holiday and don't have an access to a computer).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17869
I think I ran `SortShuffleSuite` correctly.
**Before**
```
[00:16:24] [info] SortShuffleSuite:
...
[00:18:53] [info] - SortShuffleManager properly cleans up files for shuffles that use the serialized path (297 milliseconds)
[00:18:53] [info] - SortShuffleManager properly cleans up files for shuffles that use the deserialized path (94 milliseconds)
...
```
**After**
```
[00:15:49] [info] SortShuffleSuite:
...
[00:18:18] [info] - SortShuffleManager properly cleans up files for shuffles that use the serialized path (500 milliseconds)
[00:18:18] [info] - SortShuffleManager properly cleans up files for shuffles that use the deserialized path (359 milliseconds)
...
```
Also, even if it fails, we should at least find the reason and fix if possible. Probably, there should be somewhere the resources are not being closed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org