You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by DoingDone9 <gi...@git.apache.org> on 2015/03/12 08:25:22 UTC
[GitHub] spark pull request: [SPARK-6300][Spark Core] sc.addFile(path) does...
GitHub user DoingDone9 opened a pull request:
https://github.com/apache/spark/pull/4993
[SPARK-6300][Spark Core] sc.addFile(path) does not support the relative path.
when i run cmd like that sc.addFile("../test.txt"), it did not work and throwed an exception:
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: file:../test.txt
at org.apache.hadoop.fs.Path.initialize(Path.java:206)
at org.apache.hadoop.fs.Path.<init>(Path.java:172)
........
.......
Caused by: java.net.URISyntaxException: Relative path in absolute URI: file:../test.txt
at java.net.URI.checkPath(URI.java:1804)
at java.net.URI.<init>(URI.java:752)
at org.apache.hadoop.fs.Path.initialize(Path.java:203)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/DoingDone9/spark relativePath
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4993.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4993
----
commit c3f046f8de7c418d4aa7e74afea9968a8baf9231
Author: DoingDone9 <79...@qq.com>
Date: 2015-03-02T02:11:18Z
Merge pull request #1 from apache/master
merge lastest spark
commit cb1852d14f62adbd194b1edda4ec639ba942a8ba
Author: DoingDone9 <79...@qq.com>
Date: 2015-03-05T07:05:10Z
Merge pull request #2 from apache/master
merge lastest spark
commit c87e8b6d8cb433376a7d14778915006c31f6c01c
Author: DoingDone9 <79...@qq.com>
Date: 2015-03-10T07:46:12Z
Merge pull request #3 from apache/master
merge lastest spark
commit 161cae3a29951d793ce721f9904888bd9529de72
Author: DoingDone9 <79...@qq.com>
Date: 2015-03-12T06:46:28Z
Merge pull request #4 from apache/master
merge lastest spark
commit e4a13fe302c73140731fc04b03728f4bc2f6be98
Author: DoingDone9 <79...@qq.com>
Date: 2015-03-12T06:48:24Z
getCanonicalPath
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6300][Spark Core] sc.addFile(path) does...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/4993
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6300][Spark Core] sc.addFile(path) does...
Posted by DoingDone9 <gi...@git.apache.org>.
Github user DoingDone9 commented on the pull request:
https://github.com/apache/spark/pull/4993#issuecomment-78783053
I have changed it. Please test it. @srowen
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6300][Spark Core] sc.addFile(path) does...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/4993#issuecomment-78500066
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6300][Spark Core] sc.addFile(path) does...
Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/4993#discussion_r26402600
--- Diff: core/src/test/scala/org/apache/spark/SparkContextSuite.scala ---
@@ -79,26 +79,49 @@ class SparkContextSuite extends FunSuite with LocalSparkContext {
val byteArray2 = converter.convert(bytesWritable)
assert(byteArray2.length === 0)
}
-
+
test("addFile works") {
- val file = File.createTempFile("someprefix", "somesuffix")
- val absolutePath = file.getAbsolutePath
+ val file1 = File.createTempFile("someprefix1", "somesuffix1")
+ val absolutePath1 = file1.getAbsolutePath
+
+ val pluto = Utils.createTempDir()
+ val file2 = File.createTempFile("someprefix2", "somesuffix2", pluto)
+ val relativePath = file2.getParent + "/../" + file2.getParentFile.getName + "/" + file2.getName
+ val absolutePath2 = file2.getAbsolutePath
+
try {
- Files.write("somewords", file, UTF_8)
- val length = file.length()
+ Files.write("somewords1", file1, UTF_8)
+ Files.write("somewords2", file2, UTF_8)
+ val length1 = file1.length()
+ val length2 = file2.length()
+
sc = new SparkContext(new SparkConf().setAppName("test").setMaster("local"))
- sc.addFile(file.getAbsolutePath)
+ sc.addFile(file1.getAbsolutePath)
+ sc.addFile(relativePath)
sc.parallelize(Array(1), 1).map(x => {
- val gotten = new File(SparkFiles.get(file.getName))
- if (!gotten.exists()) {
- throw new SparkException("file doesn't exist")
+ val gotten1 = new File(SparkFiles.get(file1.getName))
+ val gotten2 = new File(SparkFiles.get(file2.getName))
+ if (!gotten1.exists()) {
--- End diff --
`assert` ends up with screwy effects because the code is running inside a map function.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6300][Spark Core] sc.addFile(path) does...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4993#issuecomment-78807753
[Test build #28546 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28546/consoleFull) for PR 4993 at commit [`d594e16`](https://github.com/apache/spark/commit/d594e16c179934ab64d339a490b125572628de3b).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6300][Spark Core] sc.addFile(path) does...
Posted by DoingDone9 <gi...@git.apache.org>.
Github user DoingDone9 commented on the pull request:
https://github.com/apache/spark/pull/4993#issuecomment-78462887
I have added the test, please test it. @sryza
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6300][Spark Core] sc.addFile(path) does...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4993#issuecomment-78520588
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28516/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6300][Spark Core] sc.addFile(path) does...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4993#discussion_r26386701
--- Diff: core/src/test/scala/org/apache/spark/SparkContextSuite.scala ---
@@ -79,26 +79,49 @@ class SparkContextSuite extends FunSuite with LocalSparkContext {
val byteArray2 = converter.convert(bytesWritable)
assert(byteArray2.length === 0)
}
-
+
test("addFile works") {
- val file = File.createTempFile("someprefix", "somesuffix")
- val absolutePath = file.getAbsolutePath
+ val file1 = File.createTempFile("someprefix1", "somesuffix1")
--- End diff --
This doesn't use the temp dir you created below.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6300][Spark Core] sc.addFile(path) does...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4993#discussion_r26402950
--- Diff: core/src/test/scala/org/apache/spark/SparkContextSuite.scala ---
@@ -79,26 +79,49 @@ class SparkContextSuite extends FunSuite with LocalSparkContext {
val byteArray2 = converter.convert(bytesWritable)
assert(byteArray2.length === 0)
}
-
+
test("addFile works") {
- val file = File.createTempFile("someprefix", "somesuffix")
- val absolutePath = file.getAbsolutePath
+ val file1 = File.createTempFile("someprefix1", "somesuffix1")
+ val absolutePath1 = file1.getAbsolutePath
+
+ val pluto = Utils.createTempDir()
+ val file2 = File.createTempFile("someprefix2", "somesuffix2", pluto)
+ val relativePath = file2.getParent + "/../" + file2.getParentFile.getName + "/" + file2.getName
+ val absolutePath2 = file2.getAbsolutePath
+
try {
- Files.write("somewords", file, UTF_8)
- val length = file.length()
+ Files.write("somewords1", file1, UTF_8)
+ Files.write("somewords2", file2, UTF_8)
+ val length1 = file1.length()
+ val length2 = file2.length()
+
sc = new SparkContext(new SparkConf().setAppName("test").setMaster("local"))
- sc.addFile(file.getAbsolutePath)
+ sc.addFile(file1.getAbsolutePath)
+ sc.addFile(relativePath)
sc.parallelize(Array(1), 1).map(x => {
- val gotten = new File(SparkFiles.get(file.getName))
- if (!gotten.exists()) {
- throw new SparkException("file doesn't exist")
+ val gotten1 = new File(SparkFiles.get(file1.getName))
+ val gotten2 = new File(SparkFiles.get(file2.getName))
+ if (!gotten1.exists()) {
--- End diff --
Oh I see. Well at least this could be refactored into a little helper.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6300][Spark Core] sc.addFile(path) does...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/4993#issuecomment-81617680
I think this is OK to go actually. The refactoring of the common code isn't a big deal, and, the temp dir issue is something I'm addressing in bulk in SPARK-6338 right now anyway, so I could touch it up along with that logical change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6300][Spark Core] sc.addFile(path) does...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4993#issuecomment-78807769
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28546/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6300][Spark Core] sc.addFile(path) does...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4993#discussion_r26325092
--- Diff: core/src/test/scala/org/apache/spark/SparkContextSuite.scala ---
@@ -106,6 +106,35 @@ class SparkContextSuite extends FunSuite with LocalSparkContext {
sc.stop()
}
}
+
+ test("addFile works with relative path") {
--- End diff --
Rather than copy and paste the test I wonder if this can just be injected into the "addFile works" test. Like, make the test file a relative path there. Also I see that test doesn't use the framework provided temp dir which you did correctly here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6300][Spark Core] sc.addFile(path) does...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4993#issuecomment-78808767
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28547/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6300][Spark Core] sc.addFile(path) does...
Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/4993#discussion_r26325517
--- Diff: core/src/test/scala/org/apache/spark/SparkContextSuite.scala ---
@@ -106,6 +106,35 @@ class SparkContextSuite extends FunSuite with LocalSparkContext {
sc.stop()
}
}
+
+ test("addFile works with relative path") {
--- End diff --
I think it would be good to test both absolute and relative paths. Maybe we can break the meat of the test out into a separate function and have both tests call it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6300][Spark Core] sc.addFile(path) does...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4993#issuecomment-78784388
[Test build #28547 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28547/consoleFull) for PR 4993 at commit [`ee375cd`](https://github.com/apache/spark/commit/ee375cd197602a2e21d5ff81c57db63de4e940a6).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6300][Spark Core] sc.addFile(path) does...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4993#issuecomment-78434742
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6300][Spark Core] sc.addFile(path) does...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4993#discussion_r26386832
--- Diff: core/src/test/scala/org/apache/spark/SparkContextSuite.scala ---
@@ -79,26 +79,49 @@ class SparkContextSuite extends FunSuite with LocalSparkContext {
val byteArray2 = converter.convert(bytesWritable)
assert(byteArray2.length === 0)
}
-
+
test("addFile works") {
- val file = File.createTempFile("someprefix", "somesuffix")
- val absolutePath = file.getAbsolutePath
+ val file1 = File.createTempFile("someprefix1", "somesuffix1")
+ val absolutePath1 = file1.getAbsolutePath
+
+ val pluto = Utils.createTempDir()
+ val file2 = File.createTempFile("someprefix2", "somesuffix2", pluto)
+ val relativePath = file2.getParent + "/../" + file2.getParentFile.getName + "/" + file2.getName
+ val absolutePath2 = file2.getAbsolutePath
+
try {
- Files.write("somewords", file, UTF_8)
- val length = file.length()
+ Files.write("somewords1", file1, UTF_8)
+ Files.write("somewords2", file2, UTF_8)
+ val length1 = file1.length()
+ val length2 = file2.length()
+
sc = new SparkContext(new SparkConf().setAppName("test").setMaster("local"))
- sc.addFile(file.getAbsolutePath)
+ sc.addFile(file1.getAbsolutePath)
+ sc.addFile(relativePath)
sc.parallelize(Array(1), 1).map(x => {
- val gotten = new File(SparkFiles.get(file.getName))
- if (!gotten.exists()) {
- throw new SparkException("file doesn't exist")
+ val gotten1 = new File(SparkFiles.get(file1.getName))
+ val gotten2 = new File(SparkFiles.get(file2.getName))
+ if (!gotten1.exists()) {
+ throw new SparkException("file doesn't exist : " + absolutePath1)
+ }
+ if (!gotten2.exists()) {
--- End diff --
Same; feels like you can wrap up all these checks in a local function that you can just apply to both files rather than repeating code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6300][Spark Core] sc.addFile(path) does...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4993#issuecomment-78520562
[Test build #28516 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28516/consoleFull) for PR 4993 at commit [`0ff3fa8`](https://github.com/apache/spark/commit/0ff3fa88ae1599bd8ce67bcec7500f4647572267).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6300][Spark Core] sc.addFile(path) does...
Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/4993#discussion_r26284149
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1092,8 +1092,9 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli
*/
def addFile(path: String, recursive: Boolean): Unit = {
val uri = new URI(path)
+ val file = new File(path)
val schemeCorrectedPath = uri.getScheme match {
- case null | "local" => "file:" + uri.getPath
+ case null | "local" => "file:" + file.getCanonicalPath
--- End diff --
For scoping, probably better to move `new File(path)` into the match block.
Also, we should include the "file" scheme in the case as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6300][Spark Core] sc.addFile(path) does...
Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/4993#issuecomment-78437965
Thanks for taking this up @DoingDone9. Would it be possible to add a test for this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6300][Spark Core] sc.addFile(path) does...
Posted by DoingDone9 <gi...@git.apache.org>.
Github user DoingDone9 commented on the pull request:
https://github.com/apache/spark/pull/4993#issuecomment-78838440
/cc @sryza @srowen
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6300][Spark Core] sc.addFile(path) does...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4993#discussion_r26324850
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1093,7 +1093,7 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli
def addFile(path: String, recursive: Boolean): Unit = {
val uri = new URI(path)
val schemeCorrectedPath = uri.getScheme match {
- case null | "local" => "file:" + uri.getPath
+ case null | "local" => "file:" + new File(path).getCanonicalPath
--- End diff --
I think this might be slightly more robust as `new File(path).getCanonicalFile.toURI`. Then the `"file:"` can go away too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6300][Spark Core] sc.addFile(path) does...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4993#issuecomment-78500699
[Test build #28516 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28516/consoleFull) for PR 4993 at commit [`0ff3fa8`](https://github.com/apache/spark/commit/0ff3fa88ae1599bd8ce67bcec7500f4647572267).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6300][Spark Core] sc.addFile(path) does...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4993#issuecomment-78782198
[Test build #28546 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28546/consoleFull) for PR 4993 at commit [`d594e16`](https://github.com/apache/spark/commit/d594e16c179934ab64d339a490b125572628de3b).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6300][Spark Core] sc.addFile(path) does...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4993#discussion_r26386776
--- Diff: core/src/test/scala/org/apache/spark/SparkContextSuite.scala ---
@@ -79,26 +79,49 @@ class SparkContextSuite extends FunSuite with LocalSparkContext {
val byteArray2 = converter.convert(bytesWritable)
assert(byteArray2.length === 0)
}
-
+
test("addFile works") {
- val file = File.createTempFile("someprefix", "somesuffix")
- val absolutePath = file.getAbsolutePath
+ val file1 = File.createTempFile("someprefix1", "somesuffix1")
+ val absolutePath1 = file1.getAbsolutePath
+
+ val pluto = Utils.createTempDir()
+ val file2 = File.createTempFile("someprefix2", "somesuffix2", pluto)
+ val relativePath = file2.getParent + "/../" + file2.getParentFile.getName + "/" + file2.getName
+ val absolutePath2 = file2.getAbsolutePath
+
try {
- Files.write("somewords", file, UTF_8)
- val length = file.length()
+ Files.write("somewords1", file1, UTF_8)
+ Files.write("somewords2", file2, UTF_8)
+ val length1 = file1.length()
+ val length2 = file2.length()
+
sc = new SparkContext(new SparkConf().setAppName("test").setMaster("local"))
- sc.addFile(file.getAbsolutePath)
+ sc.addFile(file1.getAbsolutePath)
+ sc.addFile(relativePath)
sc.parallelize(Array(1), 1).map(x => {
- val gotten = new File(SparkFiles.get(file.getName))
- if (!gotten.exists()) {
- throw new SparkException("file doesn't exist")
+ val gotten1 = new File(SparkFiles.get(file1.getName))
+ val gotten2 = new File(SparkFiles.get(file2.getName))
+ if (!gotten1.exists()) {
--- End diff --
Shouldn't these be `assert`? I know they weren't before but it would be simpler.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6300][Spark Core] sc.addFile(path) does...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4993#discussion_r26386724
--- Diff: core/src/test/scala/org/apache/spark/SparkContextSuite.scala ---
@@ -79,26 +79,49 @@ class SparkContextSuite extends FunSuite with LocalSparkContext {
val byteArray2 = converter.convert(bytesWritable)
assert(byteArray2.length === 0)
}
-
+
test("addFile works") {
- val file = File.createTempFile("someprefix", "somesuffix")
- val absolutePath = file.getAbsolutePath
+ val file1 = File.createTempFile("someprefix1", "somesuffix1")
+ val absolutePath1 = file1.getAbsolutePath
+
+ val pluto = Utils.createTempDir()
--- End diff --
Heh, "pluto" isn't a good name, though I know it was copied from another test, where names were after planets.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6300][Spark Core] sc.addFile(path) does...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4993#discussion_r26386849
--- Diff: core/src/test/scala/org/apache/spark/SparkContextSuite.scala ---
@@ -79,26 +79,49 @@ class SparkContextSuite extends FunSuite with LocalSparkContext {
val byteArray2 = converter.convert(bytesWritable)
assert(byteArray2.length === 0)
}
-
+
test("addFile works") {
- val file = File.createTempFile("someprefix", "somesuffix")
- val absolutePath = file.getAbsolutePath
+ val file1 = File.createTempFile("someprefix1", "somesuffix1")
+ val absolutePath1 = file1.getAbsolutePath
+
+ val pluto = Utils.createTempDir()
+ val file2 = File.createTempFile("someprefix2", "somesuffix2", pluto)
+ val relativePath = file2.getParent + "/../" + file2.getParentFile.getName + "/" + file2.getName
+ val absolutePath2 = file2.getAbsolutePath
+
try {
- Files.write("somewords", file, UTF_8)
- val length = file.length()
+ Files.write("somewords1", file1, UTF_8)
+ Files.write("somewords2", file2, UTF_8)
+ val length1 = file1.length()
+ val length2 = file2.length()
+
sc = new SparkContext(new SparkConf().setAppName("test").setMaster("local"))
- sc.addFile(file.getAbsolutePath)
+ sc.addFile(file1.getAbsolutePath)
+ sc.addFile(relativePath)
sc.parallelize(Array(1), 1).map(x => {
- val gotten = new File(SparkFiles.get(file.getName))
- if (!gotten.exists()) {
- throw new SparkException("file doesn't exist")
+ val gotten1 = new File(SparkFiles.get(file1.getName))
+ val gotten2 = new File(SparkFiles.get(file2.getName))
+ if (!gotten1.exists()) {
+ throw new SparkException("file doesn't exist : " + absolutePath1)
+ }
+ if (!gotten2.exists()) {
+ throw new SparkException("file doesn't exist : " + absolutePath2)
--- End diff --
These and others can now use string interpolation, while we're here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6300][Spark Core] sc.addFile(path) does...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4993#issuecomment-78808747
[Test build #28547 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28547/consoleFull) for PR 4993 at commit [`ee375cd`](https://github.com/apache/spark/commit/ee375cd197602a2e21d5ff81c57db63de4e940a6).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org