You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by merlintang <gi...@git.apache.org> on 2017/12/04 23:59:53 UTC
[GitHub] spark pull request #19885: [SPARK-22587] Spark job fails if fs.defaultFS and...
GitHub user merlintang opened a pull request:
https://github.com/apache/spark/pull/19885
[SPARK-22587] Spark job fails if fs.defaultFS and application jar are d…
…ifferent url
## What changes were proposed in this pull request?
Two filesystems comparing does not consider the authority of URI. Therefore, we have to add the authority to compare two filesystem, and two filesystem with different authority can not be the same FS.
Please review http://spark.apache.org/contributing.html before opening a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/merlintang/spark EAR-7377
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19885.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19885
----
commit 3675f0a41fc0715d3d7122bbff3ab6d8fbe057c9
Author: Mingjie Tang <mt...@hortonworks.com>
Date: 2017-12-04T23:31:31Z
SPARK-22587 Spark job fails if fs.defaultFS and application jar are different url
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19885
**[Test build #85889 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85889/testReport)** for PR 19885 at commit [`31cfb88`](https://github.com/apache/spark/commit/31cfb88acfd17a3ce4481fcf32f6f2470a932bc1).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by merlintang <gi...@git.apache.org>.
Github user merlintang commented on the issue:
https://github.com/apache/spark/pull/19885
My local test is ok. I would set up a system test and update this soon.
sorry about this delay.
On Tue, Jan 2, 2018 at 3:42 PM, Marcelo Vanzin <no...@github.com>
wrote:
> Any updates?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/spark/pull/19885#issuecomment-354905082>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/ABXY-RQZs7FMQzEUSsq4qiej6xlpU2g8ks5tGr72gaJpZM4Q1hI9>
> .
>
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/19885
I still have a question about it, URIs for HDFS like `hdfs://user1@nn1.com:8020` and `hdfs://user2@nn1.com:8020` , do we honor userInfo for HDFS filesystems, are they two HDFS clusters, or just two FS cache object in local client side? @merlintang did you try this in your local test?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19885
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by merlintang <gi...@git.apache.org>.
Github user merlintang commented on the issue:
https://github.com/apache/spark/pull/19885
@jerryshao yes, hdfs://user1@nn1.com:8020 and hdfs://user2@nn1.com:8020 would consider as two filesystem, since the authority information should be taken into consideration. that is why need to add the authority to check two FS.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19885: [SPARK-22587] Spark job fails if fs.defaultFS and...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/19885#discussion_r154891674
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -1428,6 +1428,12 @@ private object Client extends Logging {
return false
}
+ val srcAuthority = srcUri.getAuthority()
+ val dstAuthority = dstUri.getAuthority()
+ if (srcAuthority != null && !srcAuthority.equalsIgnoreCase(dstAuthority)) {
--- End diff --
Now, if src is null and dst is not, this fails to detect they're different. Please see my previous comment again.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/19885
I see. Thanks for the explanation @steveloughran . My concern is that current changes will affect all the filesystems, but we only saw this issue in wasb. So limiting authority comparison to only wasb will be safer.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/19885
Is this assumption based on the implementation of Hadoop `FileSystem`? I was thinking that wasb is an exception, for other we still keep the original code.
@steveloughran would you please comment.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19885
**[Test build #85902 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85902/testReport)** for PR 19885 at commit [`8c5f029`](https://github.com/apache/spark/commit/8c5f029b0b2e4dbb9912db6ce44eb6ff0ec31f6c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19885
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19885
**[Test build #85897 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85897/testReport)** for PR 19885 at commit [`778e1ef`](https://github.com/apache/spark/commit/778e1ef903d46a46f3a389fc9b5bf8038ac7cb71).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19885
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85897/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/19885
@vanzin please help to review, thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19885
**[Test build #84455 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84455/testReport)** for PR 19885 at commit [`8bbbf04`](https://github.com/apache/spark/commit/8bbbf046a8f706add8430dc6d34c551d87c4d5a1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19885
**[Test build #84594 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84594/testReport)** for PR 19885 at commit [`dfec0ac`](https://github.com/apache/spark/commit/dfec0ac812b690b79f416806409d96a65f4d9fe7).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/19885
Hi.
If the comparision is isolated to a method testing URIs, rather than filesystems, it should be straightforward to write a suite of tests for this, with lists of URIs expected to match, as separate one of those to fail
That way we can review those combinations which people expect to match/don't match & see they meet our expectations, plus have somewhere to put new variants over time.
So: do that test, then we can see if the code does what's needed. Once that's done I'll use it as a basis for defining what Path is meant to do in the Hadoop FS spec & tests.
Things to check
```
file:///file1 file:///file 2 : match; no auth
file:///c:file1 file://c:file2 match, windows cruft. This is the bit of Path which is most trouble
file://host/file1 file://host/file2
wasb://bucket1@user wasb://bucket1@user/
hdfs:/path1 hdfs:/path2 -- "default" FS; may be patched by the time you get to FileSystem.getURI
hdfs://namenode1/path1 hdfs://namenode1:8020/path2 -using default port. I think by the time you ask the filesystem for this (FileSystem.getURI() this may have been patched up)
```
no match:
```
file:///file1 file://host/file2 :no auth in src URI (sean's problem)
file://host/file1 file:///file2
file://host/file1 file://host2/file2
wasb://bucket1@user wasb://bucket2@user/
wasb://bucket1@user wasb://bucket1@user2/
s3a://user@pass:bucket1/ s3a://user2@pass2:bucket1/ (we do a bit of secret stripping in S3A, so this may end up working in real life. Could relax that to retaining user@ though, if we retain it at all)
hdfs:/path1 hdfs:/path2
hdfs://namenode1/path1 hdfs://namenode1:8080/path2
hdfs://namenode1:8020/path1 hdfs://namenode1:8080/path2
```
See? It's complex. Add the parameterised test and then it becomes easier to review/maintain & be confident those corner cases are being handled
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by merlintang <gi...@git.apache.org>.
Github user merlintang commented on the issue:
https://github.com/apache/spark/pull/19885
@jerryshao can you backport this to branch 2.2 as well.
thanks
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/19885
Let me merge to master and branch 2.3. Thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19885
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19885
**[Test build #84455 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84455/testReport)** for PR 19885 at commit [`8bbbf04`](https://github.com/apache/spark/commit/8bbbf046a8f706add8430dc6d34c551d87c4d5a1).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19885
Switching to comparing URIs should be ok if there's an easy way to canonicalize them. There's `FileSystem.getCanonicalUri`, but it's a protected method.
Otherwise it might be tricky to support all combinations (e.g. "hdfs://somehost/path" should be equal to "hdfs://somehost.mydomain.com/path").
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/19885
ok to test.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19885: [SPARK-22587] Spark job fails if fs.defaultFS and...
Posted by merlintang <gi...@git.apache.org>.
Github user merlintang commented on a diff in the pull request:
https://github.com/apache/spark/pull/19885#discussion_r154827513
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -1428,6 +1428,12 @@ private object Client extends Logging {
return false
}
+ val srcAuthority = srcUri.getAuthority()
+ val detAuthority = dstUri.getAuthority()
+ if (srcAuthority != detAuthority || (srcAuthority != null && !srcAuthority.equalsIgnoreCase(detAuthority))) {
--- End diff --
thanks all, I would update the PR soon.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/19885
LGTM. @merlintang please fix the PR title, thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19885
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by merlintang <gi...@git.apache.org>.
Github user merlintang commented on the issue:
https://github.com/apache/spark/pull/19885
@jerryshao can you review this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19885: [SPARK-22587] Spark job fails if fs.defaultFS and...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/19885#discussion_r160617569
--- Diff: resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala ---
@@ -357,6 +357,41 @@ class ClientSuite extends SparkFunSuite with Matchers {
sparkConf.get(SECONDARY_JARS) should be (Some(Seq(new File(jar2.toURI).getName)))
}
+ private val matching = Seq(
+ ("files URI match test1", "file:///file1", "file:///file2"),
+ ("files URI match test2", "file:///c:file1", "file://c:file2"),
+ ("files URI match test3", "file://host/file1", "file://host/file2"),
+ ("wasb URI match test", "wasb://bucket1@user", "wasb://bucket1@user/"),
+ ("hdfs URI match test", "hdfs:/path1", "hdfs:/path1")
+ )
+
+ matching.foreach {
+ t =>
+ test(t._1) {
+ assert(Client.compareUri(new URI(t._2), new URI(t._3)),
+ s"No match between ${t._2} and ${t._3}")
+ }
+ }
+
+ private val unmatching = Seq(
+ ("files URI unmatch test1", "file:///file1", "file://host/file2"),
+ ("files URI unmatch test2", "file://host/file1", "file:///file2"),
+ ("files URI unmatch test3", "file://host/file1", "file://host2/file2"),
+ ("wasb URI unmatch test1", "wasb://bucket1@user", "wasb://bucket2@user/"),
+ ("wasb URI unmatch test2", "wasb://bucket1@user", "wasb://bucket1@user2/"),
+ ("s3 URI unmatch test", "s3a://user@pass:bucket1/", "s3a://user2@pass2:bucket1/"),
+ ("hdfs URI unmatch test1", "hdfs://namenode1/path1", "hdfs://namenode1:8080/path2"),
+ ("hdfs URI unmatch test2", "hdfs://namenode1:8020/path1", "hdfs://namenode1:8080/path2")
+ )
+
+ unmatching.foreach {
+ t =>
--- End diff --
ditto.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19885
**[Test build #85931 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85931/testReport)** for PR 19885 at commit [`296a19f`](https://github.com/apache/spark/commit/296a19fc5b1881959c7cf52b3c6e33eb7fa12b57).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19885
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by merlintang <gi...@git.apache.org>.
Github user merlintang commented on the issue:
https://github.com/apache/spark/pull/19885
@steveloughran can you review the added system test cases?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/19885
LGTM. Effective use of parameterization
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19885: [SPARK-22587] Spark job fails if fs.defaultFS and...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/19885#discussion_r160617532
--- Diff: resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala ---
@@ -357,6 +357,41 @@ class ClientSuite extends SparkFunSuite with Matchers {
sparkConf.get(SECONDARY_JARS) should be (Some(Seq(new File(jar2.toURI).getName)))
}
+ private val matching = Seq(
+ ("files URI match test1", "file:///file1", "file:///file2"),
+ ("files URI match test2", "file:///c:file1", "file://c:file2"),
+ ("files URI match test3", "file://host/file1", "file://host/file2"),
+ ("wasb URI match test", "wasb://bucket1@user", "wasb://bucket1@user/"),
+ ("hdfs URI match test", "hdfs:/path1", "hdfs:/path1")
+ )
+
+ matching.foreach {
--- End diff --
nit:
```
matching.foreach { t =>
...
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/19885
I'd recommend the tests are parameterized, generating a separate test for each URI pair, and including the values on a failure. Plan for a future where all you have is a stack trace from jenkins and you want to work out what failed.
Because Scalatest's `test(name: String)(testFun: => Any)` is not a function definition, just a method which registers a test, you can apply it to a sequence of path pairs, so generate a unique test for each one, something like
```scala
val matching = Seq(
("files", "file:///file1", "file:///file2"),
("hdfs", "hdfs:/path1", "hdfs:/path1")
)
matching.foreach {
t =>
test(t._1) {
assert(Client.compareUri(new URI(t._2), new URI(t._3)),
s"No match between ${t.2} and ${t._3}")
}
}
```
+same for non-matching sets, asserting that the comparison is false
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19885: [SPARK-22587] Spark job fails if fs.defaultFS and...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/19885
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/19885
if you make a path of each of these and call getFileSystem() on them, you will end up with two different FS instances in the same JVM. But they'll both be talking to the same namenode using the UGI of whoever was the current user at the time getFileSystem() was called. That is: one cluster
* Nobody should be using user@ in an HDFS URL, it doesn't do anything
* for better or worse, it does in wasb (I wish they'd just used the hostname the way the others do)
* S3 does accept user:pass to put your full credentials in, but you get told off for doing this (it gets logged in too many places), and at some point we'll turn it off. Users shouldn't be doing it.
If someone really does refer to a source JAR with a user1@hdfs:// and the HDFS filesystem doesn't have a user --why not treat the FS as different and don't worry about how the filesystem interprets it. It's a special case you aren't normally going to see. And the moment you try to go fs.makeQualified() between the two, you'll get a get a stack trace.
That is, this is not valid:
```
new Path("hdfs://user1@nn1.com:8020").getFileSystem(conf).open(new Path("hdfs://user2@nn1.com:8020"))
```
You'll inevitably get a stack trace in makeQualified. (non normative statement, try it and see)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19885
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84455/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/19885
@vanzin its too late for this, but I don't see any reason why `FileSystem.getCanonicalUri` should be kept protected. If someone wants to volunteer with the spec changes to filesystem.md & contract tests, they'll get support.
Looking at what HDFS does there, it calls out HA support as special: you can't do DNS resolution
```java
protected URI canonicalizeUri(URI uri) {
if (HAUtilClient.isLogicalUri(getConf(), uri)) {
// Don't try to DNS-resolve logical URIs, since the 'authority'
// portion isn't a proper hostname
return uri;
} else {
return NetUtils.getCanonicalUri(uri, getDefaultPort());
}
}
```
where `NetUtils.getCanonicalUri()` does some DNS lookup with caching of previously canonicalized hosts via {{SecurityUtil.getByName}}. SecurityUtil is tagged as `@Public`; NetUtils isn't, but that could be relaxed while nobody is looking. But it doesn't address the big issue: different filesystems clearly have different rules about "canonical", and you don't want to try and work them out and re-replicate, as it is a moving-maintenance-target.
I'm stuck at this point. Created [HADOOP-15094](https://issues.apache.org/jira/browse/HADOOP-15094).
Looking at {{Filesystem.CACHE}}; that compares on: (scheme, authority, ugi), so it will actually return different FS instances for unqualified and qualified hosts. Maybe for this specific problem it's simplest to say "if you do that, don't expect things to work"
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19885: [SPARK-22587] Spark job fails if fs.defaultFS and...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19885#discussion_r155110589
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -1428,6 +1428,12 @@ private object Client extends Logging {
return false
}
+ val srcAuthority = srcUri.getAuthority()
+ val dstAuthority = dstUri.getAuthority()
+ if (srcAuthority != null && !srcAuthority.equalsIgnoreCase(dstAuthority)) {
--- End diff --
This very method uses `Objects.equal` to solve that problem for hosts below.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19885
**[Test build #85902 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85902/testReport)** for PR 19885 at commit [`8c5f029`](https://github.com/apache/spark/commit/8c5f029b0b2e4dbb9912db6ce44eb6ff0ec31f6c).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/19885
>User info isn't picked up from the URL, it's taken off your Kerberos credentials. If you are running HDFS unkerberized, then UGI takes it from the environment variable HADOOP_USER_NAME.
I understand that userInfo is not picked from URL. But here in this PR it tries to use authority to compare to filesystems, and authority includes userInfo. So based on the code here, `hdfs://user1@nn1.com:8020` and `hdfs://user2@nn1.com:8020` are obviously two filesystems, but is that true this two URIs belongs to two actual hdfs clusters? Or they're just two local filesystem objects?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by merlintang <gi...@git.apache.org>.
Github user merlintang commented on the issue:
https://github.com/apache/spark/pull/19885
I have added this test case for the URI comparing based on Steve's comments. I have tested this in my local vm, it pass the test.
meanwhile, for the hdfs://namenode1/path1 hdfs://namenode1:8020/path2 , the default port number of hdfs can be got. thus, they also matched.
below is the test case:
test("compare URI for filesystem") {
//case 1
var srcUri = new URI("file:///file1")
var dstUri = new URI("file:///file2")
assert(Client.compareUri(srcUri, dstUri) == true)
//case 2
srcUri = new URI("file:///c:file1")
dstUri = new URI("file://c:file2")
assert(Client.compareUri(srcUri, dstUri) == true)
//case 3
srcUri = new URI("file://host/file1")
dstUri = new URI("file://host/file2")
assert(Client.compareUri(srcUri, dstUri) == true)
//case 4
srcUri = new URI("wasb://bucket1@user")
dstUri = new URI("wasb://bucket1@user/")
assert(Client.compareUri(srcUri, dstUri) == true)
//case 5
srcUri = new URI("hdfs:/path1")
dstUri = new URI("hdfs:/path2")
assert(Client.compareUri(srcUri, dstUri) == true)
//case 6
srcUri = new URI("file:///file1")
dstUri = new URI("file://host/file2")
assert(Client.compareUri(srcUri, dstUri) == false)
//case 7
srcUri = new URI("file://host/file1")
dstUri = new URI("file:///file2")
assert(Client.compareUri(srcUri, dstUri) == false)
//case 8
srcUri = new URI("file://host/file1")
dstUri = new URI("file://host2/file2")
assert(Client.compareUri(srcUri, dstUri) == false)
//case 9
srcUri = new URI("wasb://bucket1@user")
dstUri = new URI("wasb://bucket2@user/")
assert(Client.compareUri(srcUri, dstUri) == false)
//case 10
srcUri = new URI("wasb://bucket1@user")
dstUri = new URI("wasb://bucket1@user2/")
assert(Client.compareUri(srcUri, dstUri) == false)
//case 11
srcUri = new URI("s3a://user@pass:bucket1/")
dstUri = new URI("s3a://user2@pass2:bucket1/")
assert(Client.compareUri(srcUri, dstUri) == false)
//case 12
srcUri = new URI("hdfs://namenode1/path1")
dstUri = new URI("hdfs://namenode1:8080/path2")
assert(Client.compareUri(srcUri, dstUri) == false)
//case 13
srcUri = new URI("hdfs://namenode1:8020/path1")
dstUri = new URI("hdfs://namenode1:8080/path2")
assert(Client.compareUri(srcUri, dstUri) == false)
}
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19885
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19885
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19885: [SPARK-22587] Spark job fails if fs.defaultFS and...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/19885#discussion_r154822603
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -1428,6 +1428,12 @@ private object Client extends Logging {
return false
}
+ val srcAuthority = srcUri.getAuthority()
+ val detAuthority = dstUri.getAuthority()
+ if (srcAuthority != detAuthority || (srcAuthority != null && !srcAuthority.equalsIgnoreCase(detAuthority))) {
--- End diff --
I guess this line is too long.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19885: [SPARK-22587] Spark job fails if fs.defaultFS and...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/19885#discussion_r154819072
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -1428,6 +1428,12 @@ private object Client extends Logging {
return false
}
+ val srcAuthority = srcUri.getAuthority()
+ val detAuthority = dstUri.getAuthority()
+ if (srcAuthority != detAuthority || (srcAuthority != null && !srcAuthority.equalsIgnoreCase(detAuthority))) {
--- End diff --
This compares two strings in a case-sensitive and case-insensitive way; I presume it's supposed to be case-insensitive but this will cause authority "foo" and "Foo" to return false though they're the same string ignoring case. Did this mean a null-safe comparison like ...
```
if ((srcAuthority == null && dstAuthority != null) ||
(srcAuthority != null && !srcAuthority.equalsIgnoreCase(dstAuthority)) {
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by merlintang <gi...@git.apache.org>.
Github user merlintang commented on the issue:
https://github.com/apache/spark/pull/19885
@jerryshao and @steveloughran thanks for your comments and review.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19885
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84594/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19885
**[Test build #85931 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85931/testReport)** for PR 19885 at commit [`296a19f`](https://github.com/apache/spark/commit/296a19fc5b1881959c7cf52b3c6e33eb7fa12b57).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/19885
@steveloughran @vanzin please help to review again.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19885: [SPARK-22587] Spark job fails if fs.defaultFS and...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/19885#discussion_r154816425
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -1428,6 +1428,12 @@ private object Client extends Logging {
return false
}
+ val srcAuthority = srcUri.getAuthority()
+ val detAuthority = dstUri.getAuthority()
--- End diff --
dst? instead of det?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19885
**[Test build #84594 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84594/testReport)** for PR 19885 at commit [`dfec0ac`](https://github.com/apache/spark/commit/dfec0ac812b690b79f416806409d96a65f4d9fe7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19885
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85931/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19885
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85889/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19885
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85902/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/19885
User info isn't picked up from the URL, it's taken off your Kerberos credentials. If you are running HDFS unkerberized, then UGI takes it from the environment variable `HADOOP_USER_NAME`.
Looking at the `DFSClient` code, it's done in
```
this.ugi = UserGroupInformation.getCurrentUser();
```
Also looking at that code, I'm not convinced that using user@hdfs wouldn't up causing confusion, especially in the world of HA HDFS. You'd really need to experiment to see if you could break things.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19885
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19885
Any updates?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by merlintang <gi...@git.apache.org>.
Github user merlintang commented on the issue:
https://github.com/apache/spark/pull/19885
I am so sorry for the late of testing function, I would update it soon.
On Thu, Dec 14, 2017 at 12:55 PM, UCB AMPLab <no...@github.com>
wrote:
> Can one of the admins verify this patch?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/spark/pull/19885#issuecomment-351832769>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/ABXY-dnjtJZHPD-2OulAGPdSSASXOKCJks5tAYsugaJpZM4Q1hI9>
> .
>
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19885
**[Test build #85897 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85897/testReport)** for PR 19885 at commit [`778e1ef`](https://github.com/apache/spark/commit/778e1ef903d46a46f3a389fc9b5bf8038ac7cb71).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/19885
@merlintang would you please add the problem to your PR description, currently it is a WASB problem in which userInfo is honored to differentiate filesystems. Please add the scenario to the description.
Besides the changes here will also affect all other filesystems like HDFS/S3, do they still have same behaviors? What will be happened if `hdfs://user1@nn1.com:8020` and `hdfs://user2@nn1.com:8020`, are they two filesystems? Would you please clarify?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19885
I think everybody is still waiting for the tests to be added to the PR.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19885
**[Test build #85889 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85889/testReport)** for PR 19885 at commit [`31cfb88`](https://github.com/apache/spark/commit/31cfb88acfd17a3ce4481fcf32f6f2470a932bc1).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org