You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2017/07/21 22:00:00 UTC
[jira] [Assigned] (SPARK-21339) spark-shell --packages option does
not add jars to classpath on windows
[ https://issues.apache.org/jira/browse/SPARK-21339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-21339:
------------------------------------
Assignee: Apache Spark
> spark-shell --packages option does not add jars to classpath on windows
> -----------------------------------------------------------------------
>
> Key: SPARK-21339
> URL: https://issues.apache.org/jira/browse/SPARK-21339
> Project: Spark
> Issue Type: Bug
> Components: Spark Shell, Windows
> Affects Versions: 2.1.1
> Environment: Windows 10 Enterprise x64
> Reporter: Goran Blankendal
> Assignee: Apache Spark
>
> I am unable to import symbols from dependencies specified with the packages option:
> {code}
> spark-shell --packages "com.datastax.spark:spark-cassandra-connector_2.11:2.0.2" --conf spark.jars.ivy="c:/tmp/ivy2" --verbose
> {code}
> This results in:
> {code}
> scala> import com.datastax.spark.connector._
> <console>:23: error: object datastax is not a member of package com
> import com.datastax.spark.connector._
> ^
> {code}
> NOTE: It is working as expected when running on Linux but not on Windows.
> Complete verbose output:
> {code}
> > spark-shell --packages "com.datastax.spark:spark-cassandra-connector_2.11:2.0.2" --conf spark.jars.ivy="c:/tmp/ivy2" --
> verbose
> Using properties file: null
> Parsed arguments:
> master local[*]
> deployMode null
> executorMemory null
> executorCores null
> totalExecutorCores null
> propertiesFile null
> driverMemory null
> driverCores null
> driverExtraClassPath null
> driverExtraLibraryPath null
> driverExtraJavaOptions null
> supervise false
> queue null
> numExecutors null
> files null
> pyFiles null
> archives null
> mainClass org.apache.spark.repl.Main
> primaryResource spark-shell
> name Spark shell
> childArgs []
> jars null
> packages com.datastax.spark:spark-cassandra-connector_2.11:2.0.2
> packagesExclusions null
> repositories null
> verbose true
> Spark properties used, including those specified through
> --conf and those from the properties file null:
> spark.jars.ivy -> c:/tmp/ivy2
> Ivy Default Cache set to: c:\tmp\ivy2\cache
> The jars for the packages stored in: c:\tmp\ivy2\jars
> :: loading settings :: url = jar:file:/C:/hadoop/spark-2.1.1-bin-hadoop2.7/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
> com.datastax.spark#spark-cassandra-connector_2.11 added as a dependency
> :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
> confs: [default]
> found com.datastax.spark#spark-cassandra-connector_2.11;2.0.2 in local-m2-cache
> found com.twitter#jsr166e;1.1.0 in local-m2-cache
> found commons-beanutils#commons-beanutils;1.9.3 in central
> found commons-collections#commons-collections;3.2.2 in local-m2-cache
> found org.joda#joda-convert;1.2 in local-m2-cache
> found joda-time#joda-time;2.3 in central
> found io.netty#netty-all;4.0.33.Final in local-m2-cache
> found org.scala-lang#scala-reflect;2.11.8 in local-m2-cache
> :: resolution report :: resolve 378ms :: artifacts dl 8ms
> :: modules in use:
> com.datastax.spark#spark-cassandra-connector_2.11;2.0.2 from local-m2-cache in [default]
> com.twitter#jsr166e;1.1.0 from local-m2-cache in [default]
> commons-beanutils#commons-beanutils;1.9.3 from central in [default]
> commons-collections#commons-collections;3.2.2 from local-m2-cache in [default]
> io.netty#netty-all;4.0.33.Final from local-m2-cache in [default]
> joda-time#joda-time;2.3 from central in [default]
> org.joda#joda-convert;1.2 from local-m2-cache in [default]
> org.scala-lang#scala-reflect;2.11.8 from local-m2-cache in [default]
> ---------------------------------------------------------------------
> | | modules || artifacts |
> | conf | number| search|dwnlded|evicted|| number|dwnlded|
> ---------------------------------------------------------------------
> | default | 8 | 0 | 0 | 0 || 8 | 0 |
> ---------------------------------------------------------------------
> :: retrieving :: org.apache.spark#spark-submit-parent
> confs: [default]
> 0 artifacts copied, 8 already retrieved (0kB/11ms)
> Main class:
> org.apache.spark.repl.Main
> Arguments:
> System properties:
> spark.jars.ivy -> c:/tmp/ivy2
> SPARK_SUBMIT -> true
> spark.app.name -> Spark shell
> spark.jars -> file:/c:/tmp/ivy2/jars/com.datastax.spark_spark-cassandra-connector_2.11-2.0.2.jar,file:/c:/tmp/ivy2/jars/com.twitter_jsr166e-1.1.0.jar,file:/c:/tmp/ivy2/jars/commons-beanutils_commons-beanutils-1.9.3.jar,file:/c:/tmp/ivy2/jars/org.joda_joda-convert-1.2.jar,file:/c:/tmp/ivy2/jars/joda-time_joda-time-2.3.jar,file:/c:/tmp/ivy2/jars/io.netty_netty-all-4.0.33.Final.jar,file:/c:/tmp/ivy2/jars/org.scala-lang_scala-reflect-2.11.8.jar,file:/c:/tmp/ivy2/jars/commons-collections_commons-collections-3.2.2.jar
> spark.submit.deployMode -> client
> spark.master -> local[*]
> Classpath elements:
> c:\tmp\ivy2\jars\com.datastax.spark_spark-cassandra-connector_2.11-2.0.2.jar
> c:\tmp\ivy2\jars\com.twitter_jsr166e-1.1.0.jar
> c:\tmp\ivy2\jars\commons-beanutils_commons-beanutils-1.9.3.jar
> c:\tmp\ivy2\jars\org.joda_joda-convert-1.2.jar
> c:\tmp\ivy2\jars\joda-time_joda-time-2.3.jar
> c:\tmp\ivy2\jars\io.netty_netty-all-4.0.33.Final.jar
> c:\tmp\ivy2\jars\org.scala-lang_scala-reflect-2.11.8.jar
> c:\tmp\ivy2\jars\commons-collections_commons-collections-3.2.2.jar
> Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
> Setting default log level to "WARN".
> To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
> 17/07/07 15:45:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 17/07/07 15:45:28 WARN General: Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/hadoop/spark-2.1.1-bin-hadoop2.7/bin/../jars/datanucleus-core-3.2.10.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/hadoop/spark-2.1.1-bin-hadoop2.7/jars/datanucleus-core-3.2.10.jar."
> 17/07/07 15:45:28 WARN General: Plugin (Bundle) "org.datanucleus.api.jdo" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/hadoop/spark-2.1.1-bin-hadoop2.7/bin/../jars/datanucleus-api-jdo-3.2.6.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/hadoop/spark-2.1.1-bin-hadoop2.7/jars/datanucleus-api-jdo-3.2.6.jar."
> 17/07/07 15:45:28 WARN General: Plugin (Bundle) "org.datanucleus.store.rdbms" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/hadoop/spark-2.1.1-bin-hadoop2.7/jars/datanucleus-rdbms-3.2.9.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/hadoop/spark-2.1.1-bin-hadoop2.7/bin/../jars/datanucleus-rdbms-3.2.9.jar."
> 17/07/07 15:45:31 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
> Spark context Web UI available at http://192.168.56.1:4040
> Spark context available as 'sc' (master = local[*], app id = local-1499435127578).
> Spark session available as 'spark'.
> Welcome to
> ____ __
> / __/__ ___ _____/ /__
> _\ \/ _ \/ _ `/ __/ '_/
> /___/ .__/\_,_/_/ /_/\_\ version 2.1.1
> /_/
> Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_121)
> Type in expressions to have them evaluated.
> Type :help for more information.
> scala>
> scala> import com.datastax.spark.connector._
> <console>:23: error: object datastax is not a member of package com
> import com.datastax.spark.connector._
> ^
> {code}
> The behaviour is different when we add the downloaded jar explicitly to the classpath with spark.driver.extraClassPath
> {code}
> spark-shell --conf spark.driver.extraClassPath="c:\tmp\ivy2\jars\com.datastax.spark_spark-cassandra-connector_2.11-2.0.2.jar" --verbose
> {code}
> Complete output:
> {code}
> spark-shell --conf spark.driver.extraClassPath="c:\tmp\ivy2\jars\com.data
> stax.spark_spark-cassandra-connector_2.11-2.0.2.jar" --verbose
> Using properties file: null
> Parsed arguments:
> master local[*]
> deployMode null
> executorMemory null
> executorCores null
> totalExecutorCores null
> propertiesFile null
> driverMemory null
> driverCores null
> driverExtraClassPath c:\tmp\ivy2\jars\com.datastax.spark_spark-cassandra-connector_2.11-2.0.2.jar
> driverExtraLibraryPath null
> driverExtraJavaOptions null
> supervise false
> queue null
> numExecutors null
> files null
> pyFiles null
> archives null
> mainClass org.apache.spark.repl.Main
> primaryResource spark-shell
> name Spark shell
> childArgs []
> jars null
> packages null
> packagesExclusions null
> repositories null
> verbose true
> Spark properties used, including those specified through
> --conf and those from the properties file null:
> spark.driver.extraClassPath -> c:\tmp\ivy2\jars\com.datastax.spark_spark-cassandra-connector_2.11-2.0.2.jar
> Main class:
> org.apache.spark.repl.Main
> Arguments:
> System properties:
> SPARK_SUBMIT -> true
> spark.app.name -> Spark shell
> spark.jars ->
> spark.submit.deployMode -> client
> spark.master -> local[*]
> spark.driver.extraClassPath -> c:\tmp\ivy2\jars\com.datastax.spark_spark-cassandra-connector_2.11-2.0.2.jar
> Classpath elements:
> Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
> Setting default log level to "WARN".
> To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
> 17/07/07 16:05:16 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 17/07/07 16:05:18 WARN General: Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/hadoop/spark-2.1.1-bin-hadoop2.7/bin/../jars/datanucleus-core-3.2.10.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/hadoop/spark-2.1.1-bin-hadoop2.7/jars/datanucleus-core-3.2.10.jar."
> 17/07/07 16:05:18 WARN General: Plugin (Bundle) "org.datanucleus.api.jdo" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/hadoop/spark-2.1.1-bin-hadoop2.7/bin/../jars/datanucleus-api-jdo-3.2.6.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/hadoop/spark-2.1.1-bin-hadoop2.7/jars/datanucleus-api-jdo-3.2.6.jar."
> 17/07/07 16:05:18 WARN General: Plugin (Bundle) "org.datanucleus.store.rdbms" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/hadoop/spark-2.1.1-bin-hadoop2.7/jars/datanucleus-rdbms-3.2.9.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/hadoop/spark-2.1.1-bin-hadoop2.7/bin/../jars/datanucleus-rdbms-3.2.9.jar."
> 17/07/07 16:05:21 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
> Spark context Web UI available at http://192.168.56.1:4040
> Spark context available as 'sc' (master = local[*], app id = local-1499436317287).
> Spark session available as 'spark'.
> Welcome to
> ____ __
> / __/__ ___ _____/ /__
> _\ \/ _ \/ _ `/ __/ '_/
> /___/ .__/\_,_/_/ /_/\_\ version 2.1.1
> /_/
> Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_121)
> Type in expressions to have them evaluated.
> Type :help for more information.
> scala> import com.datastax.spark.connector._
> import com.datastax.spark.connector._
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org