You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by va...@apache.org on 2017/08/01 20:42:45 UTC
spark git commit: [SPARK-21339][CORE] spark-shell --packages option
does not add jars to classpath on windows
Repository: spark
Updated Branches:
refs/heads/master b1d59e60d -> 58da1a245
[SPARK-21339][CORE] spark-shell --packages option does not add jars to classpath on windows
The --packages option jars are getting added to the classpath with the scheme as "file:///", in Unix it doesn't have problem with this since the scheme contains the Unix Path separator which separates the jar name with location in the classpath. In Windows, the jar file is not getting resolved from the classpath because of the scheme.
Windows : file:///C:/Users/<user>/.ivy2/jars/<jar-name>.jar
Unix : file:///home/<user>/.ivy2/jars/<jar-name>.jar
With this PR, we are avoiding the 'file://' scheme to get added to the packages jar files.
I have verified manually in Windows and Unix environments, with the change it adds the jar to classpath like below,
Windows : C:\Users\<user>\.ivy2\jars\<jar-name>.jar
Unix : /home/<user>/.ivy2/jars/<jar-name>.jar
Author: Devaraj K <de...@apache.org>
Closes #18708 from devaraj-kavali/SPARK-21339.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/58da1a24
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/58da1a24
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/58da1a24
Branch: refs/heads/master
Commit: 58da1a2455258156fe8ba57241611eac1a7928ef
Parents: b1d59e6
Author: Devaraj K <de...@apache.org>
Authored: Tue Aug 1 13:38:55 2017 -0700
Committer: Marcelo Vanzin <va...@cloudera.com>
Committed: Tue Aug 1 13:39:23 2017 -0700
----------------------------------------------------------------------
.../scala-2.11/src/main/scala/org/apache/spark/repl/Main.scala | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/58da1a24/repl/scala-2.11/src/main/scala/org/apache/spark/repl/Main.scala
----------------------------------------------------------------------
diff --git a/repl/scala-2.11/src/main/scala/org/apache/spark/repl/Main.scala b/repl/scala-2.11/src/main/scala/org/apache/spark/repl/Main.scala
index b8b38e8..9702a1e 100644
--- a/repl/scala-2.11/src/main/scala/org/apache/spark/repl/Main.scala
+++ b/repl/scala-2.11/src/main/scala/org/apache/spark/repl/Main.scala
@@ -18,6 +18,7 @@
package org.apache.spark.repl
import java.io.File
+import java.net.URI
import java.util.Locale
import scala.tools.nsc.GenericRunnerSettings
@@ -56,7 +57,10 @@ object Main extends Logging {
// Visible for testing
private[repl] def doMain(args: Array[String], _interp: SparkILoop): Unit = {
interp = _interp
- val jars = Utils.getUserJars(conf, isShell = true).mkString(File.pathSeparator)
+ val jars = Utils.getUserJars(conf, isShell = true)
+ // Remove file:///, file:// or file:/ scheme if exists for each jar
+ .map { x => if (x.startsWith("file:")) new File(new URI(x)).getPath else x }
+ .mkString(File.pathSeparator)
val interpArguments = List(
"-Yrepl-class-based",
"-Yrepl-outdir", s"${outputDir.getAbsolutePath}",
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org