You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by james64 <gi...@git.apache.org> on 2017/11/28 15:17:08 UTC
[GitHub] spark pull request #19834: [SPARK-22585][Core] Path in addJar is not url enc...
GitHub user james64 opened a pull request:
https://github.com/apache/spark/pull/19834
[SPARK-22585][Core] Path in addJar is not url encoded
## What changes were proposed in this pull request?
This updates a behavior of `addJar` method of `sparkContext` class. If path without any scheme is passed as input it is used literally without url encoding/decoding it.
## How was this patch tested?
A unit test is added for this.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/james64/spark SPARK-22585-encode-add-jar
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19834.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19834
----
commit a312107761a8de4779c1d1401d7099beb01190b6
Author: Jakub Dubovsky <ja...@seznam.cz>
Date: 2017-11-28T12:39:17Z
Do not decode path without schema in addJar
commit 1fc5db37fb6cd9f554a9651a04d9775c9b9a475a
Author: Jakub Dubovsky <ja...@seznam.cz>
Date: 2017-11-28T15:10:28Z
Add unit test
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19834: [SPARK-22585][Core] Path in addJar is not url encoded
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19834
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19834: [SPARK-22585][Core] Path in addJar is not url encoded
Posted by james64 <gi...@git.apache.org>.
Github user james64 commented on the issue:
https://github.com/apache/spark/pull/19834
Thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19834: [SPARK-22585][Core] Path in addJar is not url encoded
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19834
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84302/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19834: [SPARK-22585][Core] Path in addJar is not url encoded
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19834
**[Test build #84266 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84266/testReport)** for PR 19834 at commit [`bd667d9`](https://github.com/apache/spark/commit/bd667d93758515a39f9fea44c5edf92eff8a5898).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19834: [SPARK-22585][Core] Path in addJar is not url encoded
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19834
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84262/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19834: [SPARK-22585][Core] Path in addJar is not url encoded
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19834
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19834: [SPARK-22585][Core] Path in addJar is not url encoded
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19834
**[Test build #84295 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84295/testReport)** for PR 19834 at commit [`bd667d9`](https://github.com/apache/spark/commit/bd667d93758515a39f9fea44c5edf92eff8a5898).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19834: [SPARK-22585][Core] Path in addJar is not url encoded
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19834
Merged to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19834: [SPARK-22585][Core] Path in addJar is not url encoded
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19834
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19834: [SPARK-22585][Core] Path in addJar is not url enc...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/19834
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19834: [SPARK-22585][Core] Path in addJar is not url encoded
Posted by james64 <gi...@git.apache.org>.
Github user james64 commented on the issue:
https://github.com/apache/spark/pull/19834
So I have removed those three commented lines. Is there anything else to do before merge?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19834: [SPARK-22585][Core] Path in addJar is not url encoded
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19834
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84266/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19834: [SPARK-22585][Core] Path in addJar is not url encoded
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19834
**[Test build #84262 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84262/testReport)** for PR 19834 at commit [`1fc5db3`](https://github.com/apache/spark/commit/1fc5db37fb6cd9f554a9651a04d9775c9b9a475a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19834: [SPARK-22585][Core] Path in addJar is not url encoded
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19834
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19834: [SPARK-22585][Core] Path in addJar is not url encoded
Posted by james64 <gi...@git.apache.org>.
Github user james64 commented on the issue:
https://github.com/apache/spark/pull/19834
@srowen Let's continue our discussion here.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19834: [SPARK-22585][Core] Path in addJar is not url encoded
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19834
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19834: [SPARK-22585][Core] Path in addJar is not url encoded
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19834
**[Test build #84295 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84295/testReport)** for PR 19834 at commit [`bd667d9`](https://github.com/apache/spark/commit/bd667d93758515a39f9fea44c5edf92eff8a5898).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19834: [SPARK-22585][Core] Path in addJar is not url encoded
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19834
**[Test build #84302 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84302/testReport)** for PR 19834 at commit [`bd667d9`](https://github.com/apache/spark/commit/bd667d93758515a39f9fea44c5edf92eff8a5898).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19834: [SPARK-22585][Core] Path in addJar is not url encoded
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19834
**[Test build #84262 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84262/testReport)** for PR 19834 at commit [`1fc5db3`](https://github.com/apache/spark/commit/1fc5db37fb6cd9f554a9651a04d9775c9b9a475a).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19834: [SPARK-22585][Core] Path in addJar is not url encoded
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/19834
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19834: [SPARK-22585][Core] Path in addJar is not url encoded
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19834
**[Test build #84266 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84266/testReport)** for PR 19834 at commit [`bd667d9`](https://github.com/apache/spark/commit/bd667d93758515a39f9fea44c5edf92eff8a5898).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19834: [SPARK-22585][Core] Path in addJar is not url encoded
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19834
Another LGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19834: [SPARK-22585][Core] Path in addJar is not url encoded
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19834
**[Test build #84302 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84302/testReport)** for PR 19834 at commit [`bd667d9`](https://github.com/apache/spark/commit/bd667d93758515a39f9fea44c5edf92eff8a5898).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19834: [SPARK-22585][Core] Path in addJar is not url enc...
Posted by james64 <gi...@git.apache.org>.
Github user james64 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19834#discussion_r153524308
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1846,7 +1846,16 @@ class SparkContext(config: SparkConf) extends Logging {
Utils.validateURL(uri)
uri.getScheme match {
// A JAR file which exists only on the driver node
- case null | "file" => addJarFile(new File(uri.getPath))
+ case null =>
+ // SPARK-22585 path without schema is not url encoded
+ addJarFile(new File(uri.getRawPath))
+
+// val encoded = java.net.URLEncoder.encode(path, "UTF-8")
+// val newUri = new URI(encoded)
+// addJarFile(new File(newUri.getPath)
--- End diff --
These three commented lines shows solution which would url encode the path and then pass it to URI once again. If we can identify any advantage in this approach we can use it. Otherwise I am for using `getRawPath` one liner.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19834: [SPARK-22585][Core] Path in addJar is not url encoded
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/19834
LGTM.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19834: [SPARK-22585][Core] Path in addJar is not url encoded
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19834
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84295/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org