You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by HyukjinKwon <gi...@git.apache.org> on 2016/11/12 19:44:19 UTC
[GitHub] spark pull request #15866: [SPARK-18422][CORE] Fix wholeTextFiles test to pa...
GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/15866
[SPARK-18422][CORE] Fix wholeTextFiles test to pass on Windows in JavaAPISuite
## What changes were proposed in this pull request?
This PR fixes the test `wholeTextFiles` in `JavaAPISuite.java`. This is failed due to the different path format on Windows.
For example, the path in `container` was
```
C:\projects\spark\target\tmp\1478967560189-0/part-00000
```
whereas `new URI(res._1()).getPath()` was as below:
/C:/projects/spark/target/tmp/1478967560189-0/part-00000
## How was this patch tested?
Tests in `JavaAPISuite.java`.
Tested via AppVeyor https://ci.appveyor.com/project/spark-test/spark
Diff : https://github.com/apache/spark/compare/master...spark-test:JavaAPISuite
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/spark SPARK-18422
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15866.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15866
----
commit 9c258a5bbe19f1ea84aa6b9bab53f02093449a38
Author: hyukjinkwon <gu...@gmail.com>
Date: 2016-11-12T19:40:13Z
Fix wholeTextFile tests in JavaAPISuite
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15866: [SPARK-18422][CORE] Fix wholeTextFiles test to pass on W...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/15866
Looks OK to me @HyukjinKwon but does the appveyor result in your comment above mean that it still fails on Windows?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15866: [SPARK-18422][CORE] Fix wholeTextFiles test to pass on W...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/15866
Thank you Sean. Actually, this is a bit annoying.
Here is what happens in the original test.
1. Writes a file to read back by `wholeTextFiles`.
```scala
scala> new File("C:\\a", "/part-0000")
res10: java.io.File = C:\a\part-0000
```
2. and then save the path in a map, `container`.
```scala
scala> val container = Map("C:\\a/part-0000" -> "contents")
res1: scala.collection.immutable.Map[String,String] = Map(C:\a/part-0000 -> contents)
```
3. then it compares the key in `container` to the collected paths from `wholeTextFiles`. The path should be
```scala
scala> val path = "file:/C:/a/part-0000"
path: String = file:/C:/a/part-0000
scala> val actualPath = new URI(path).getPath
res2: String = /C:/a/part-0000
scala>
```
4. it ends up with..
```scala
scala> container.contains(actualPath)
res4: Boolean = false
```
Just FWIW, (at least for myself to note this problem somewhere and re-visit sometime), it seems both back-slash and slash are fine with `java.io.File`.
```scala
scala> new File("C:/a")
res9: java.io.File = C:\a
scala> new File("C:\\a", "/part-0000")
res10: java.io.File = C:\a\part-0000
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15866: [SPARK-18422][CORE] Fix wholeTextFiles test to pass on W...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/15866
Yes please. I am pretty sure that this is a right fix without any other side effects.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15866: [SPARK-18422][CORE] Fix wholeTextFiles test to pa...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/15866
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15866: [SPARK-18422][CORE] Fix wholeTextFiles test to pass on W...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15866
**[Test build #68615 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68615/consoleFull)** for PR 15866 at commit [`e6b2a03`](https://github.com/apache/spark/commit/e6b2a035ef63cc8b2d75769cbd32fedc9908b763).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15866: [SPARK-18422][CORE] Fix wholeTextFiles test to pass on W...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/15866
Oh, the test this PR tries to deal with, `org.apache.spark.JavaAPISuite.wholeTextFiles` is passed. However, another test in here `org.apache.spark.JavaAPISuite.writeWithNewAPIHadoopFile` is being failed for a different reason which I am investigating this.
**Before**
```
[info] Test org.apache.spark.JavaAPISuite.wholeTextFiles started
[error] Test org.apache.spark.JavaAPISuite.wholeTextFiles failed: java.lang.AssertionError: expected:<spark is easy to use.
[error] > but was:<null>, took 0.578 sec
[error] at org.apache.spark.JavaAPISuite.wholeTextFiles(JavaAPISuite.java:1089)
[error] ...
```
**After**
```
[info] Test org.apache.spark.JavaAPISuite.wholeTextFiles started
...
[info] Test org.apache.spark.JavaAPISuite.writeWithNewAPIHadoopFile started
[error] Test org.apache.spark.JavaAPISuite.writeWithNewAPIHadoopFile failed: org.apache.spark.SparkException: Job aborted., took 0.407 sec
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15866: [SPARK-18422][CORE] Fix wholeTextFiles test to pass on W...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15866
**[Test build #68569 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68569/consoleFull)** for PR 15866 at commit [`9c258a5`](https://github.com/apache/spark/commit/9c258a5bbe19f1ea84aa6b9bab53f02093449a38).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15866: [SPARK-18422][CORE] Fix wholeTextFiles test to pass on W...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15866
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68615/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15866: [SPARK-18422][CORE] Fix wholeTextFiles test to pass on W...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15866
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68569/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15866: [SPARK-18422][CORE] Fix wholeTextFiles test to pass on W...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/15866
Merged to master/2.1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15866: [SPARK-18422][CORE] Fix wholeTextFiles test to pass on W...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/15866
@srowen It seems `org.apache.spark.JavaAPISuite.writeWithNewAPIHadoopFile` is also being failed but the root cause seems different. That is being failed with the exception as below:
```
java.io.IOException: Could not rename file:
...
```
And, other failures specified in the parent JIRA of this seems caused by the problems in `ReceiverTracker` we discussed in https://github.com/apache/spark/pull/15618
Will try to make another PR soon for this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15866: [SPARK-18422][CORE] Fix wholeTextFiles test to pass on W...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15866
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15866: [SPARK-18422][CORE] Fix wholeTextFiles test to pass on W...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15866
**[Test build #68615 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68615/consoleFull)** for PR 15866 at commit [`e6b2a03`](https://github.com/apache/spark/commit/e6b2a035ef63cc8b2d75769cbd32fedc9908b763).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15866: [SPARK-18422][CORE] Fix wholeTextFiles test to pass on W...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/15866
OK if it makes the test in question here pass, let's merge it, as a fix for that test. That much is OK right @HyukjinKwon ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15866: [SPARK-18422][CORE] Fix wholeTextFiles test to pass on W...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/15866
Build started: [CORE] `org.apache.spark.JavaAPISuite` [](https://ci.appveyor.com/project/spark-test/spark/branch/EAC49351-734B-4218-A2CF-13B3C3CB83E8)
Diff: https://github.com/apache/spark/compare/master...spark-test:EAC49351-734B-4218-A2CF-13B3C3CB83E8
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15866: [SPARK-18422][CORE] Fix wholeTextFiles test to pass on W...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/15866
Let me add a comment here and will try to clean up more.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15866: [SPARK-18422][CORE] Fix wholeTextFiles test to pass on W...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15866
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15866: [SPARK-18422][CORE] Fix wholeTextFiles test to pass on W...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/15866
Build started: [CORE] `org.apache.spark.JavaAPISuite` [](https://ci.appveyor.com/project/spark-test/spark/branch/198DDA52-F201-4D2B-BE2F-244E0C1725B2)
Diff: https://github.com/apache/spark/compare/master...spark-test:198DDA52-F201-4D2B-BE2F-244E0C1725B2
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15866: [SPARK-18422][CORE] Fix wholeTextFiles test to pass on W...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15866
**[Test build #68569 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68569/consoleFull)** for PR 15866 at commit [`9c258a5`](https://github.com/apache/spark/commit/9c258a5bbe19f1ea84aa6b9bab53f02093449a38).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org