You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Goran Blankendal (JIRA)" <ji...@apache.org> on 2017/07/07 14:07:00 UTC
[jira] [Created] (SPARK-21339) spark-shell --packages option does
not add jars to classpath on windows
Goran Blankendal created SPARK-21339:
----------------------------------------
Summary: spark-shell --packages option does not add jars to classpath on windows
Key: SPARK-21339
URL: https://issues.apache.org/jira/browse/SPARK-21339
Project: Spark
Issue Type: Bug
Components: Spark Shell, Windows
Affects Versions: 2.1.1
Environment: Windows 10 Enterprise x64
Reporter: Goran Blankendal
I am unable to import symbols from dependencies specified with the packages option:
{code}
spark-shell --packages "com.datastax.spark:spark-cassandra-connector_2.11:2.0.2" --conf spark.jars.ivy="c:/tmp/ivy2" --verbose
{code}
This results in:
{code}
scala> import com.datastax.spark.connector._
<console>:23: error: object datastax is not a member of package com
import com.datastax.spark.connector._
^
{code}
NOTE: It is working as expected when running on Linux but not on Windows.
Complete verbose output:
{code}
> spark-shell --packages "com.datastax.spark:spark-cassandra-connector_2.11:2.0.2" --conf spark.jars.ivy="c:/tmp/ivy2" --
verbose
Using properties file: null
Parsed arguments:
master local[*]
deployMode null
executorMemory null
executorCores null
totalExecutorCores null
propertiesFile null
driverMemory null
driverCores null
driverExtraClassPath null
driverExtraLibraryPath null
driverExtraJavaOptions null
supervise false
queue null
numExecutors null
files null
pyFiles null
archives null
mainClass org.apache.spark.repl.Main
primaryResource spark-shell
name Spark shell
childArgs []
jars null
packages com.datastax.spark:spark-cassandra-connector_2.11:2.0.2
packagesExclusions null
repositories null
verbose true
Spark properties used, including those specified through
--conf and those from the properties file null:
spark.jars.ivy -> c:/tmp/ivy2
Ivy Default Cache set to: c:\tmp\ivy2\cache
The jars for the packages stored in: c:\tmp\ivy2\jars
:: loading settings :: url = jar:file:/C:/hadoop/spark-2.1.1-bin-hadoop2.7/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
com.datastax.spark#spark-cassandra-connector_2.11 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
confs: [default]
found com.datastax.spark#spark-cassandra-connector_2.11;2.0.2 in local-m2-cache
found com.twitter#jsr166e;1.1.0 in local-m2-cache
found commons-beanutils#commons-beanutils;1.9.3 in central
found commons-collections#commons-collections;3.2.2 in local-m2-cache
found org.joda#joda-convert;1.2 in local-m2-cache
found joda-time#joda-time;2.3 in central
found io.netty#netty-all;4.0.33.Final in local-m2-cache
found org.scala-lang#scala-reflect;2.11.8 in local-m2-cache
:: resolution report :: resolve 378ms :: artifacts dl 8ms
:: modules in use:
com.datastax.spark#spark-cassandra-connector_2.11;2.0.2 from local-m2-cache in [default]
com.twitter#jsr166e;1.1.0 from local-m2-cache in [default]
commons-beanutils#commons-beanutils;1.9.3 from central in [default]
commons-collections#commons-collections;3.2.2 from local-m2-cache in [default]
io.netty#netty-all;4.0.33.Final from local-m2-cache in [default]
joda-time#joda-time;2.3 from central in [default]
org.joda#joda-convert;1.2 from local-m2-cache in [default]
org.scala-lang#scala-reflect;2.11.8 from local-m2-cache in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 8 | 0 | 0 | 0 || 8 | 0 |
---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent
confs: [default]
0 artifacts copied, 8 already retrieved (0kB/11ms)
Main class:
org.apache.spark.repl.Main
Arguments:
System properties:
spark.jars.ivy -> c:/tmp/ivy2
SPARK_SUBMIT -> true
spark.app.name -> Spark shell
spark.jars -> file:/c:/tmp/ivy2/jars/com.datastax.spark_spark-cassandra-connector_2.11-2.0.2.jar,file:/c:/tmp/ivy2/jars/com.twitter_jsr166e-1.1.0.jar,file:/c:/tmp/ivy2/jars/commons-beanutils_commons-beanutils-1.9.3.jar,file:/c:/tmp/ivy2/jars/org.joda_joda-convert-1.2.jar,file:/c:/tmp/ivy2/jars/joda-time_joda-time-2.3.jar,file:/c:/tmp/ivy2/jars/io.netty_netty-all-4.0.33.Final.jar,file:/c:/tmp/ivy2/jars/org.scala-lang_scala-reflect-2.11.8.jar,file:/c:/tmp/ivy2/jars/commons-collections_commons-collections-3.2.2.jar
spark.submit.deployMode -> client
spark.master -> local[*]
Classpath elements:
c:\tmp\ivy2\jars\com.datastax.spark_spark-cassandra-connector_2.11-2.0.2.jar
c:\tmp\ivy2\jars\com.twitter_jsr166e-1.1.0.jar
c:\tmp\ivy2\jars\commons-beanutils_commons-beanutils-1.9.3.jar
c:\tmp\ivy2\jars\org.joda_joda-convert-1.2.jar
c:\tmp\ivy2\jars\joda-time_joda-time-2.3.jar
c:\tmp\ivy2\jars\io.netty_netty-all-4.0.33.Final.jar
c:\tmp\ivy2\jars\org.scala-lang_scala-reflect-2.11.8.jar
c:\tmp\ivy2\jars\commons-collections_commons-collections-3.2.2.jar
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
17/07/07 15:45:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/07/07 15:45:28 WARN General: Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/hadoop/spark-2.1.1-bin-hadoop2.7/bin/../jars/datanucleus-core-3.2.10.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/hadoop/spark-2.1.1-bin-hadoop2.7/jars/datanucleus-core-3.2.10.jar."
17/07/07 15:45:28 WARN General: Plugin (Bundle) "org.datanucleus.api.jdo" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/hadoop/spark-2.1.1-bin-hadoop2.7/bin/../jars/datanucleus-api-jdo-3.2.6.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/hadoop/spark-2.1.1-bin-hadoop2.7/jars/datanucleus-api-jdo-3.2.6.jar."
17/07/07 15:45:28 WARN General: Plugin (Bundle) "org.datanucleus.store.rdbms" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/hadoop/spark-2.1.1-bin-hadoop2.7/jars/datanucleus-rdbms-3.2.9.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/hadoop/spark-2.1.1-bin-hadoop2.7/bin/../jars/datanucleus-rdbms-3.2.9.jar."
17/07/07 15:45:31 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Spark context Web UI available at http://192.168.56.1:4040
Spark context available as 'sc' (master = local[*], app id = local-1499435127578).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.1.1
/_/
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_121)
Type in expressions to have them evaluated.
Type :help for more information.
scala>
scala> import com.datastax.spark.connector._
<console>:23: error: object datastax is not a member of package com
import com.datastax.spark.connector._
^
{code}
The behaviour is different when we add the downloaded jar explicitly to the classpath with spark.driver.extraClassPath
{code}
spark-shell --conf spark.driver.extraClassPath="c:\tmp\ivy2\jars\com.datastax.spark_spark-cassandra-connector_2.11-2.0.2.jar" --verbose
{code}
Complete output:
{code}
spark-shell --conf spark.driver.extraClassPath="c:\tmp\ivy2\jars\com.data
stax.spark_spark-cassandra-connector_2.11-2.0.2.jar" --verbose
Using properties file: null
Parsed arguments:
master local[*]
deployMode null
executorMemory null
executorCores null
totalExecutorCores null
propertiesFile null
driverMemory null
driverCores null
driverExtraClassPath c:\tmp\ivy2\jars\com.datastax.spark_spark-cassandra-connector_2.11-2.0.2.jar
driverExtraLibraryPath null
driverExtraJavaOptions null
supervise false
queue null
numExecutors null
files null
pyFiles null
archives null
mainClass org.apache.spark.repl.Main
primaryResource spark-shell
name Spark shell
childArgs []
jars null
packages null
packagesExclusions null
repositories null
verbose true
Spark properties used, including those specified through
--conf and those from the properties file null:
spark.driver.extraClassPath -> c:\tmp\ivy2\jars\com.datastax.spark_spark-cassandra-connector_2.11-2.0.2.jar
Main class:
org.apache.spark.repl.Main
Arguments:
System properties:
SPARK_SUBMIT -> true
spark.app.name -> Spark shell
spark.jars ->
spark.submit.deployMode -> client
spark.master -> local[*]
spark.driver.extraClassPath -> c:\tmp\ivy2\jars\com.datastax.spark_spark-cassandra-connector_2.11-2.0.2.jar
Classpath elements:
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
17/07/07 16:05:16 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/07/07 16:05:18 WARN General: Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/hadoop/spark-2.1.1-bin-hadoop2.7/bin/../jars/datanucleus-core-3.2.10.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/hadoop/spark-2.1.1-bin-hadoop2.7/jars/datanucleus-core-3.2.10.jar."
17/07/07 16:05:18 WARN General: Plugin (Bundle) "org.datanucleus.api.jdo" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/hadoop/spark-2.1.1-bin-hadoop2.7/bin/../jars/datanucleus-api-jdo-3.2.6.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/hadoop/spark-2.1.1-bin-hadoop2.7/jars/datanucleus-api-jdo-3.2.6.jar."
17/07/07 16:05:18 WARN General: Plugin (Bundle) "org.datanucleus.store.rdbms" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/hadoop/spark-2.1.1-bin-hadoop2.7/jars/datanucleus-rdbms-3.2.9.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/hadoop/spark-2.1.1-bin-hadoop2.7/bin/../jars/datanucleus-rdbms-3.2.9.jar."
17/07/07 16:05:21 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Spark context Web UI available at http://192.168.56.1:4040
Spark context available as 'sc' (master = local[*], app id = local-1499436317287).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.1.1
/_/
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_121)
Type in expressions to have them evaluated.
Type :help for more information.
scala> import com.datastax.spark.connector._
import com.datastax.spark.connector._
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org