You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by pierre-borckmans <gi...@git.apache.org> on 2015/12/19 18:02:20 UTC
[GitHub] spark pull request: [SPARK-12440] - [core] - Avoid setCheckpoint w...
GitHub user pierre-borckmans opened a pull request:
https://github.com/apache/spark/pull/10392
[SPARK-12440] - [core] - Avoid setCheckpoint warning when directory is not local
In SparkContext method `setCheckpointDir`, a warning is issued when spark master is not local and the passed directory for the checkpoint dir appears to be local.
In practice, when relying on HDFS configuration file and using relative path (incomplete URI without HDFS scheme, ...), this warning should not be issued and might be confusing.
This PR creates a `FileSystem` with the given directory, and check whether it is local or not.
The warning is only issued if the directory is not local.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pierre-borckmans/spark SPARK-12440_CheckpointDir_Warning_NonLocal
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10392.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10392
----
commit bace5b98d20a9f7d6876b65120e6668aa064c948
Author: pierre-borckmans <pi...@realimpactanalytics.com>
Date: 2015-12-19T16:55:00Z
AVOID setCheckpoint warning when file system is not local
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by pierre-borckmans <gi...@git.apache.org>.
Github user pierre-borckmans commented on a diff in the pull request:
https://github.com/apache/spark/pull/10392#discussion_r48103410
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -2072,14 +2072,15 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli
// Otherwise, the driver may attempt to reconstruct the checkpointed RDD from
// its own local file system, which is incorrect because the checkpoint files
// are actually on the executor machines.
- if (!isLocal && Utils.nonLocalPaths(directory).isEmpty) {
+ val path = new Path(directory, UUID.randomUUID().toString)
+ val fs = path.getFileSystem(hadoopConfiguration)
+ val isDirLocal = fs.isInstanceOf[LocalFileSystem]
+ if (!isLocal && Utils.nonLocalPaths(directory).isEmpty && !isDirLocal) {
--- End diff --
Indeed, should be ```if (!isLocal && Utils.nonLocalPaths(directory).isEmpty && isDirLocal) {```.
Checking LocalFileSystem is brittle indeed.
The point is the following. We had a use case where we were deploying on a Yarn cluster, with Kerberos. We could not access parquet files on hdfs, nor set the checkpoint dir using a full URI (`hdfs://host:port/path/xxx`). Instead, we had to use the path only `path/xxx`, and the `FileSystem` used by spark both for the checkpoint dir and the parquet file loading would then correctly access hdfs, relying on the system hadoop configuration.
With this use case, the warning is misleading since spark master is not local, but the path we are using is not local.
I agree with you, changing the warning message could suffice, but I'm not sure how to rephrase it for this use case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10392#issuecomment-166605772
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48186/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10392#issuecomment-166568588
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10392#issuecomment-166411099
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10392#issuecomment-166410688
**[Test build #48126 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48126/consoleFull)** for PR 10392 at commit [`bace5b9`](https://github.com/apache/spark/commit/bace5b98d20a9f7d6876b65120e6668aa064c948).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/10392#issuecomment-166646614
Eh, I don't think anyone wants flaky tests of course. Search JIRA. They get improved by people improving them -- please have a go if you'd like to help.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10392#issuecomment-166605771
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10392#issuecomment-166605662
**[Test build #48186 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48186/consoleFull)** for PR 10392 at commit [`707ce5b`](https://github.com/apache/spark/commit/707ce5b628625d5455b9ff4d6cc637bde453021b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/10392#issuecomment-166640491
Yes, flaky tests. There are so many, some really complex integration tests, that unfortunately something fails half the time
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10392#issuecomment-166411102
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48126/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/10392#issuecomment-166407388
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10392#discussion_r48133110
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -2072,14 +2072,15 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli
// Otherwise, the driver may attempt to reconstruct the checkpointed RDD from
// its own local file system, which is incorrect because the checkpoint files
// are actually on the executor machines.
- if (!isLocal && Utils.nonLocalPaths(directory).isEmpty) {
+ val path = new Path(directory, UUID.randomUUID().toString)
+ val fs = path.getFileSystem(hadoopConfiguration)
+ val isDirLocal = fs.isInstanceOf[LocalFileSystem]
+ if (!isLocal && Utils.nonLocalPaths(directory).isEmpty && !isDirLocal) {
--- End diff --
Is the "host:port" the issue? then write `hdfs:///path/xxx`.
My suggestion is to modify the warning message to something like `s"If Spark is not running in local mode, then the checkpoint directory must not be on the local filesystem. Directory '$directory' appears to be on the local filesystem."` It still causes a warning in your case, but I believe that warning is avoidable (?) with the right `hdfs` URI in this case. Hence it's still useful.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10392#issuecomment-166583582
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48184/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10392#issuecomment-166568251
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/10392#issuecomment-167114552
Merged to master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by pierre-borckmans <gi...@git.apache.org>.
Github user pierre-borckmans commented on the pull request:
https://github.com/apache/spark/pull/10392#issuecomment-166637885
@srowen Thanks. Just for my curiosity, the failing tests were completely unrelated, right? Does that mean master was in an unstable state? Or are the unit tests somewhat flaky?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10392#issuecomment-166568388
**[Test build #48181 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48181/consoleFull)** for PR 10392 at commit [`6e21b23`](https://github.com/apache/spark/commit/6e21b235c996861f775a2202ba8a3cce7690b525).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10392#issuecomment-166583580
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10392#issuecomment-166568584
**[Test build #48181 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48181/consoleFull)** for PR 10392 at commit [`6e21b23`](https://github.com/apache/spark/commit/6e21b235c996861f775a2202ba8a3cce7690b525).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10392#discussion_r48101284
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -2072,14 +2072,15 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli
// Otherwise, the driver may attempt to reconstruct the checkpointed RDD from
// its own local file system, which is incorrect because the checkpoint files
// are actually on the executor machines.
- if (!isLocal && Utils.nonLocalPaths(directory).isEmpty) {
+ val path = new Path(directory, UUID.randomUUID().toString)
+ val fs = path.getFileSystem(hadoopConfiguration)
+ val isDirLocal = fs.isInstanceOf[LocalFileSystem]
+ if (!isLocal && Utils.nonLocalPaths(directory).isEmpty && !isDirLocal) {
--- End diff --
Wait, you're checking here whether the directory is _not_ local here -- the warning happens when it is local right? Checking `LocalFileSystem` is a little bit brittle too.
I think the right answer here is to disambiguate the checkpoint dir with an `hdfs:` scheme. This avoids ambiguity too. If it's just to otherwise save a warning, I don't think this is worth it. At best, clarify the warning message.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/10392
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10392#issuecomment-166411093
**[Test build #48126 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48126/consoleFull)** for PR 10392 at commit [`bace5b9`](https://github.com/apache/spark/commit/bace5b98d20a9f7d6876b65120e6668aa064c948).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/10392#issuecomment-166624346
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by pierre-borckmans <gi...@git.apache.org>.
Github user pierre-borckmans commented on the pull request:
https://github.com/apache/spark/pull/10392#issuecomment-166565536
@srowen Agreed. I reverted my changes, and updated the warning message.
Thanks for your review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10392#discussion_r48237938
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -2073,8 +2073,9 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli
// its own local file system, which is incorrect because the checkpoint files
// are actually on the executor machines.
if (!isLocal && Utils.nonLocalPaths(directory).isEmpty) {
- logWarning("Checkpoint directory must be non-local " +
- "if Spark is running on a cluster: " + directory)
+ logWarning(s"Spark is not running in local mode, therefore the checkpoint directory " +
--- End diff --
Interpolation here is on the wrong string of these 3. Also looks like some style check error
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10392#issuecomment-166573828
**[Test build #48186 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48186/consoleFull)** for PR 10392 at commit [`707ce5b`](https://github.com/apache/spark/commit/707ce5b628625d5455b9ff4d6cc637bde453021b).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10392#issuecomment-166568589
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48181/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by pierre-borckmans <gi...@git.apache.org>.
Github user pierre-borckmans commented on the pull request:
https://github.com/apache/spark/pull/10392#issuecomment-166645424
Thanks, good to know. Is there work planned on that matter? Or do you guys just specialise in "seeing that it was a flaky test, just restart them" ? :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440] - [core] - Avoid setCheckpoint w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10392#issuecomment-166004451
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10392#issuecomment-166568253
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48180/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by pierre-borckmans <gi...@git.apache.org>.
Github user pierre-borckmans commented on the pull request:
https://github.com/apache/spark/pull/10392#issuecomment-166572991
@srowen I fixed the scala style issues and the string interpolator.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10392#issuecomment-166583532
**[Test build #48184 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48184/consoleFull)** for PR 10392 at commit [`c316479`](https://github.com/apache/spark/commit/c316479b5ab0a9ec55e4cc62ca3e6974a585b304).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12440][Core] - Avoid setCheckpoint warn...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10392#issuecomment-166571605
**[Test build #48184 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48184/consoleFull)** for PR 10392 at commit [`c316479`](https://github.com/apache/spark/commit/c316479b5ab0a9ec55e4cc62ca3e6974a585b304).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org