You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jeff - Data Bean Australia <da...@gmail.com> on 2016/02/05 05:00:21 UTC
Please help with external package using --packages option in spark-shell
Hi,
Trying to use an external package with the following command line and it
doesn't work:
./spark-shell --packages harsha2010:magellan:1.0.3-s_2.10
Please find the package info here:
http://spark-packages.org/package/harsha2010/magellan
The error related to one of the dependencies was not found successfully:
[NOT FOUND ]
org.codehaus.jackson#jackson-core-asl;1.9.12!jackson-core-asl.jar (1ms)
And the weird part is that Spark or Ivy didn't actually try to look it up
online, but simply reported error when it didn't find the package in my
local m2 repository. Please see the complete error message below and help
me out.
MacBook-Pro-2:bin jeffhuang$ ./spark-shell --packages
harsha2010:magellan:1.0.3-s_2.10
Ivy Default Cache set to: /Users/jeffhuang/.ivy2/cache
The jars for the packages stored in: /Users/jeffhuang/.ivy2/jars
:: loading settings :: url =
jar:file:/Users/jeffhuang/learning/spark/spark-1.5.1-bin-hadoop2.6/lib/spark-assembly-1.5.1-hadoop2.6.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
harsha2010#magellan added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
confs: [default]
found harsha2010#magellan;1.0.3-s_2.10 in spark-packages
found commons-io#commons-io;2.4 in local-m2-cache
found com.esri.geometry#esri-geometry-api;1.2.1 in central
found org.json#json;20090211 in local-m2-cache
found org.codehaus.jackson#jackson-core-asl;1.9.12 in local-m2-cache
:: resolution report :: resolve 282ms :: artifacts dl 9ms
:: modules in use:
com.esri.geometry#esri-geometry-api;1.2.1 from central in [default]
commons-io#commons-io;2.4 from local-m2-cache in [default]
harsha2010#magellan;1.0.3-s_2.10 from spark-packages in [default]
org.codehaus.jackson#jackson-core-asl;1.9.12 from local-m2-cache in
[default]
org.json#json;20090211 from local-m2-cache in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 5 | 0 | 0 | 0 || 5 | 0 |
---------------------------------------------------------------------
:: problems summary ::
:::: WARNINGS
[NOT FOUND ]
org.codehaus.jackson#jackson-core-asl;1.9.12!jackson-core-asl.jar (1ms)
==== local-m2-cache: tried
file:/Users/jeffhuang/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.12/jackson-core-asl-1.9.12.jar
::::::::::::::::::::::::::::::::::::::::::::::
:: FAILED DOWNLOADS ::
:: ^ see resolution messages for details ^ ::
::::::::::::::::::::::::::::::::::::::::::::::
:: org.codehaus.jackson#jackson-core-asl;1.9.12!jackson-core-asl.jar
::::::::::::::::::::::::::::::::::::::::::::::
:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [download failed:
org.codehaus.jackson#jackson-core-asl;1.9.12!jackson-core-asl.jar]
at
org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1009)
at
org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:286)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:153)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
--
Data Bean - A Big Data Solution Provider in Australia.
Re: Please help with external package using --packages option in spark-shell
Posted by Jeff - Data Bean Australia <da...@gmail.com>.
Finally I figured out the problem and fixed it.
There was some inconsistency in my .ivy2 and .m2 repository. Spark resolves
the dependencies using meta data in ivy2/cache by not verifies its real
location. That was why Spark resolved jackson-core-asl in local-m2-cache.
But when Spark tried to load the JAR file, it couldn't find it and caused
the error.
To solve the problem, I deleted the corresponding directory for
jackson-core-asl under .ivy2.
On Fri, Feb 5, 2016 at 5:00 PM, Jeff - Data Bean Australia <
databean.au@gmail.com> wrote:
> Hi,
>
> Trying to use an external package with the following command line and it
> doesn't work:
>
> ./spark-shell --packages harsha2010:magellan:1.0.3-s_2.10
>
> Please find the package info here:
> http://spark-packages.org/package/harsha2010/magellan
>
> The error related to one of the dependencies was not found successfully:
>
> [NOT FOUND ]
> org.codehaus.jackson#jackson-core-asl;1.9.12!jackson-core-asl.jar (1ms)
>
> And the weird part is that Spark or Ivy didn't actually try to look it up
> online, but simply reported error when it didn't find the package in my
> local m2 repository. Please see the complete error message below and help
> me out.
>
> MacBook-Pro-2:bin jeffhuang$ ./spark-shell --packages
> harsha2010:magellan:1.0.3-s_2.10
> Ivy Default Cache set to: /Users/jeffhuang/.ivy2/cache
> The jars for the packages stored in: /Users/jeffhuang/.ivy2/jars
> :: loading settings :: url =
> jar:file:/Users/jeffhuang/learning/spark/spark-1.5.1-bin-hadoop2.6/lib/spark-assembly-1.5.1-hadoop2.6.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
> harsha2010#magellan added as a dependency
> :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
> confs: [default]
> found harsha2010#magellan;1.0.3-s_2.10 in spark-packages
> found commons-io#commons-io;2.4 in local-m2-cache
> found com.esri.geometry#esri-geometry-api;1.2.1 in central
> found org.json#json;20090211 in local-m2-cache
> found org.codehaus.jackson#jackson-core-asl;1.9.12 in local-m2-cache
> :: resolution report :: resolve 282ms :: artifacts dl 9ms
> :: modules in use:
> com.esri.geometry#esri-geometry-api;1.2.1 from central in [default]
> commons-io#commons-io;2.4 from local-m2-cache in [default]
> harsha2010#magellan;1.0.3-s_2.10 from spark-packages in [default]
> org.codehaus.jackson#jackson-core-asl;1.9.12 from local-m2-cache in
> [default]
> org.json#json;20090211 from local-m2-cache in [default]
> ---------------------------------------------------------------------
> | | modules || artifacts |
> | conf | number| search|dwnlded|evicted|| number|dwnlded|
> ---------------------------------------------------------------------
> | default | 5 | 0 | 0 | 0 || 5 | 0 |
> ---------------------------------------------------------------------
>
> :: problems summary ::
> :::: WARNINGS
> [NOT FOUND ]
> org.codehaus.jackson#jackson-core-asl;1.9.12!jackson-core-asl.jar (1ms)
>
> ==== local-m2-cache: tried
>
>
> file:/Users/jeffhuang/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.12/jackson-core-asl-1.9.12.jar
>
> ::::::::::::::::::::::::::::::::::::::::::::::
>
> :: FAILED DOWNLOADS ::
>
> :: ^ see resolution messages for details ^ ::
>
> ::::::::::::::::::::::::::::::::::::::::::::::
>
> :: org.codehaus.jackson#jackson-core-asl;1.9.12!jackson-core-asl.jar
>
> ::::::::::::::::::::::::::::::::::::::::::::::
>
>
>
> :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
> Exception in thread "main" java.lang.RuntimeException: [download failed:
> org.codehaus.jackson#jackson-core-asl;1.9.12!jackson-core-asl.jar]
> at
> org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1009)
> at
> org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:286)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:153)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> --
> Data Bean - A Big Data Solution Provider in Australia.
>
--
Data Bean - A Big Data Solution Provider in Australia.
Re: Please help with external package using --packages option in spark-shell
Posted by Jeff - Data Bean Australia <da...@gmail.com>.
Thanks Divya for helping me.
It does connect to the internet.
And I even tried to use a local artifactory repository and it didn't work
either.
$ ./spark-shell --packages harsha2010:magellan:1.0.3-s_2.10 --repositories
http://localhost:8081/artifactory/libs-release
Ivy Default Cache set to: /Users/jeffhuang/.ivy2/cache
The jars for the packages stored in: /Users/jeffhuang/.ivy2/jars
http://localhost:8081/artifactory/libs-release added as a remote repository
with the name: repo-1
:: loading settings :: url =
jar:file:/Users/jeffhuang/learning/spark/spark-1.5.1-bin-hadoop2.6/lib/spark-assembly-1.5.1-hadoop2.6.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
harsha2010#magellan added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
confs: [default]
found harsha2010#magellan;1.0.3-s_2.10 in spark-packages
found commons-io#commons-io;2.4 in local-m2-cache
found com.esri.geometry#esri-geometry-api;1.2.1 in central
found org.json#json;20090211 in local-m2-cache
found org.codehaus.jackson#jackson-core-asl;1.9.12 in local-m2-cache
:: resolution report :: resolve 282ms :: artifacts dl 10ms
:: modules in use:
com.esri.geometry#esri-geometry-api;1.2.1 from central in [default]
commons-io#commons-io;2.4 from local-m2-cache in [default]
harsha2010#magellan;1.0.3-s_2.10 from spark-packages in [default]
org.codehaus.jackson#jackson-core-asl;1.9.12 from local-m2-cache in
[default]
org.json#json;20090211 from local-m2-cache in [default]
On Fri, Feb 5, 2016 at 5:01 PM, Divya Gehlot <di...@gmail.com>
wrote:
> Is your machine connected to internet ?
>
>
> On 5 February 2016 at 12:00, Jeff - Data Bean Australia <
> databean.au@gmail.com> wrote:
>
>> Hi,
>>
>> Trying to use an external package with the following command line and it
>> doesn't work:
>>
>> ./spark-shell --packages harsha2010:magellan:1.0.3-s_2.10
>>
>> Please find the package info here:
>> http://spark-packages.org/package/harsha2010/magellan
>>
>> The error related to one of the dependencies was not found successfully:
>>
>> [NOT FOUND ]
>> org.codehaus.jackson#jackson-core-asl;1.9.12!jackson-core-asl.jar (1ms)
>>
>> And the weird part is that Spark or Ivy didn't actually try to look it up
>> online, but simply reported error when it didn't find the package in my
>> local m2 repository. Please see the complete error message below and help
>> me out.
>>
>> MacBook-Pro-2:bin jeffhuang$ ./spark-shell --packages
>> harsha2010:magellan:1.0.3-s_2.10
>> Ivy Default Cache set to: /Users/jeffhuang/.ivy2/cache
>> The jars for the packages stored in: /Users/jeffhuang/.ivy2/jars
>> :: loading settings :: url =
>> jar:file:/Users/jeffhuang/learning/spark/spark-1.5.1-bin-hadoop2.6/lib/spark-assembly-1.5.1-hadoop2.6.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
>> harsha2010#magellan added as a dependency
>> :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
>> confs: [default]
>> found harsha2010#magellan;1.0.3-s_2.10 in spark-packages
>> found commons-io#commons-io;2.4 in local-m2-cache
>> found com.esri.geometry#esri-geometry-api;1.2.1 in central
>> found org.json#json;20090211 in local-m2-cache
>> found org.codehaus.jackson#jackson-core-asl;1.9.12 in local-m2-cache
>> :: resolution report :: resolve 282ms :: artifacts dl 9ms
>> :: modules in use:
>> com.esri.geometry#esri-geometry-api;1.2.1 from central in [default]
>> commons-io#commons-io;2.4 from local-m2-cache in [default]
>> harsha2010#magellan;1.0.3-s_2.10 from spark-packages in [default]
>> org.codehaus.jackson#jackson-core-asl;1.9.12 from local-m2-cache in
>> [default]
>> org.json#json;20090211 from local-m2-cache in [default]
>> ---------------------------------------------------------------------
>> | | modules || artifacts |
>> | conf | number| search|dwnlded|evicted|| number|dwnlded|
>> ---------------------------------------------------------------------
>> | default | 5 | 0 | 0 | 0 || 5 | 0 |
>> ---------------------------------------------------------------------
>>
>> :: problems summary ::
>> :::: WARNINGS
>> [NOT FOUND ]
>> org.codehaus.jackson#jackson-core-asl;1.9.12!jackson-core-asl.jar (1ms)
>>
>> ==== local-m2-cache: tried
>>
>>
>> file:/Users/jeffhuang/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.12/jackson-core-asl-1.9.12.jar
>>
>> ::::::::::::::::::::::::::::::::::::::::::::::
>>
>> :: FAILED DOWNLOADS ::
>>
>> :: ^ see resolution messages for details ^ ::
>>
>> ::::::::::::::::::::::::::::::::::::::::::::::
>>
>> :: org.codehaus.jackson#jackson-core-asl;1.9.12!jackson-core-asl.jar
>>
>> ::::::::::::::::::::::::::::::::::::::::::::::
>>
>>
>>
>> :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
>> Exception in thread "main" java.lang.RuntimeException: [download failed:
>> org.codehaus.jackson#jackson-core-asl;1.9.12!jackson-core-asl.jar]
>> at
>> org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1009)
>> at
>> org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:286)
>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:153)
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>
>> --
>> Data Bean - A Big Data Solution Provider in Australia.
>>
>
>
--
Data Bean - A Big Data Solution Provider in Australia.