You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by srowen <gi...@git.apache.org> on 2014/05/11 17:26:08 UTC
[GitHub] spark pull request: SPARK-1798. Tests should clean up temp files
GitHub user srowen opened a pull request:
https://github.com/apache/spark/pull/732
SPARK-1798. Tests should clean up temp files
Three issues related to temp files that tests generate – these should be touched up for hygiene but are not urgent.
Modules have a log4j.properties which directs the unit-test.log output file to a directory like `[module]/target/unit-test.log`. But this ends up creating `[module]/[module]/target/unit-test.log` instead of former.
The `work/` directory is not deleted by "mvn clean", in the parent and in modules. Neither is the `checkpoint/` directory created under the various external modules.
Many tests create a temp directory, which is not usually deleted. This can be largely resolved by calling `deleteOnExit()` at creation and trying to call `Utils.deleteRecursively` consistently to clean up, sometimes in an `@After` method.
_If anyone seconds the motion, I can create a more significant change that introduces a new test trait along the lines of `LocalSparkContext`, which provides management of temp directories for subclasses to take advantage of._
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/srowen/spark SPARK-1798
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/732.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #732
----
commit bdd0f410e17f69c5463dbb3014240e7f552dccda
Author: Sean Owen <so...@cloudera.com>
Date: 2014-05-06T21:19:07Z
Remove duplicate module dir in log4j.properties output path for tests
commit b21b3566fa51b5375ca809449004a873cb5feefb
Author: Sean Owen <so...@cloudera.com>
Date: 2014-05-06T21:21:48Z
Remove work/ and checkpoint/ dirs with mvn clean
commit 5af578e5be2450371ec14aa33d1a679f029fb720
Author: Sean Owen <so...@cloudera.com>
Date: 2014-05-06T21:22:23Z
Try to consistently delete test temp dirs and files, and set deleteOnExit() for each
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1798. Tests should clean up temp files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/732#issuecomment-42774629
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1798. Tests should clean up temp files
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/732
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1798. Tests should clean up temp files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/732#issuecomment-42773573
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1798. Tests should clean up temp files
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/732#discussion_r12552796
--- Diff: mllib/src/test/scala/org/apache/spark/mllib/util/MLUtilsSuite.scala ---
@@ -90,7 +92,7 @@ class MLUtilsSuite extends FunSuite with LocalSparkContext {
assert(multiclassPoints(1).label === -1.0)
assert(multiclassPoints(2).label === -1.0)
- deleteQuietly(tempDir)
+ Utils.deleteRecursively(tempDir)
--- End diff --
That's true. I suppose the idea is that tests shouldn't fail merely because cleanup failed. But other tests that do cleanup don't swallow exceptions. I had thought to both be consistent, and err on the side of reporting such an error (in case it weirdly indicates a problem? like files being held open incorrectly somewhere)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1798. Tests should clean up temp files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/732#issuecomment-42773567
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1798. Tests should clean up temp files
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/732#issuecomment-42882250
Looks good to me pending one small question. This is a nice clean-up of various test cod.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1798. Tests should clean up temp files
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/732#issuecomment-42889334
Okay, i'll pull this in. If there was a reason to swallow exceptions then we can add that back easily.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1798. Tests should clean up temp files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/732#issuecomment-42774630
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14887/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1798. Tests should clean up temp files
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/732#discussion_r12550787
--- Diff: mllib/src/test/scala/org/apache/spark/mllib/util/MLUtilsSuite.scala ---
@@ -90,7 +92,7 @@ class MLUtilsSuite extends FunSuite with LocalSparkContext {
assert(multiclassPoints(1).label === -1.0)
assert(multiclassPoints(2).label === -1.0)
- deleteQuietly(tempDir)
+ Utils.deleteRecursively(tempDir)
--- End diff --
This changes the behavior to not swallow exceptions. This was added originally by @mengxr... is there a reason this squashes exceptions?
https://github.com/jegonzal/spark/commit/98750a74#diff-006677f6b8222b96d21bc3e46ac9fe77R161
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1798. Tests should clean up temp files
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/732#discussion_r12553138
--- Diff: mllib/src/test/scala/org/apache/spark/mllib/util/MLUtilsSuite.scala ---
@@ -90,7 +92,7 @@ class MLUtilsSuite extends FunSuite with LocalSparkContext {
assert(multiclassPoints(1).label === -1.0)
assert(multiclassPoints(2).label === -1.0)
- deleteQuietly(tempDir)
+ Utils.deleteRecursively(tempDir)
--- End diff --
Yeah, I don't quite see a reason why we'd want to squash these. Okay perhaps we can standardize on "not squashing" exceptions and if it turns out this is necessary here we can fix it up later. It does seem to be passing fine in your patch
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---