You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Venkat Krishnamurthy <ve...@yarcdata.com> on 2013/09/13 07:09:51 UTC
N00b alert - spark-ec2 script failure
All
Apologies if this post is misdirected. I'm trying to get started with Spark, and using the AMPLab hands-on as a guide – The set up went as far as creating and spinning up the AMI's, but part way through is this dependency resolution error from sbt:
Seems simple enough, but the failed dependency is spark-core… which I assume is An Important Thing.
I'm hoping this is embarrassingly obvious to find and fix (or ignore) - any pointers?
-----
[warn] module not found: org.apache.spark#spark-core_2.9.3;0.8.0-SNAPSHOT
[warn] ==== local: tried
[warn] /root/.ivy2/local/org.apache.spark/spark-core_2.9.3/0.8.0-SNAPSHOT/ivys/ivy.xml
[warn] ==== Typesafe Repository: tried
[warn] http://repo.typesafe.com/typesafe/releases/org/apache/spark/spark-core_2.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
[warn] ==== JBoss Repository: tried
[warn] http://repository.jboss.org/nexus/content/repositories/releases/org/apache/spark/spark-core_2.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
[warn] ==== Spray Repository: tried
[warn] http://repo.spray.cc/org/apache/spark/spark-core_2.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
[warn] ==== Cloudera Repository: tried
[warn] https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-core_2.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
[warn] ==== Local Maven: tried
[warn] file:/root/.m2/repository/org/apache/spark/spark-core_2.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
[warn] ==== public: tried
[warn] http://repo1.maven.org/maven2/org/apache/spark/spark-core_2.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
[info] Resolving org.apache.spark#spark-repl_2.9.3;0.8.0-SNAPSHOT ...
[warn] module not found: org.apache.spark#spark-repl_2.9.3;0.8.0-SNAPSHOT
[warn] ==== local: tried
[warn] /root/.ivy2/local/org.apache.spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/ivys/ivy.xml
[warn] ==== Typesafe Repository: tried
[warn] http://repo.typesafe.com/typesafe/releases/org/apache/spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
[warn] ==== JBoss Repository: tried
[warn] http://repository.jboss.org/nexus/content/repositories/releases/org/apache/spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
[warn] ==== Spray Repository: tried
[warn] http://repo.spray.cc/org/apache/spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
[warn] ==== Cloudera Repository: tried
[warn] https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
[warn] ==== Local Maven: tried
[warn] file:/root/.m2/repository/org/apache/spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
[warn] ==== public: tried
[warn] http://repo1.maven.org/maven2/org/apache/spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
[info] Resolving org.scala-tools.testing#test-interface;0.5 ...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: org.apache.spark#spark-core_2.9.3;0.8.0-SNAPSHOT: not found
[warn] :: org.apache.spark#spark-repl_2.9.3;0.8.0-SNAPSHOT: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
sbt.ResolveException: unresolved dependency: org.apache.spark#spark-core_2.9.3;0.8.0-SNAPSHOT: not found
Re: N00b alert - spark-ec2 script failure
Posted by Shivaram Venkataraman <sh...@gmail.com>.
Right - you will only encounter this for the day 2 exercises (MLI,
BlinkDB). I pushed out a fix for this [1] that should be effective if
you launch another cluster.
Thanks
Shivaram
[1] https://github.com/mesos/spark-ec2/commit/4465aab125c4ee486da00136b7f107162c30f9f8
On Fri, Sep 13, 2013 at 11:17 AM, Venkat Krishnamurthy
<ve...@yarcdata.com> wrote:
> Just an update - I actually managed to work through the Day 1 tutorial
> with zero hiccups so far (up to the twitter exercise) so this didn't
> really didn't prevent me from working.
>
> Will update if I face any issues on day 2 exercises.
>
> I'm still picking my jaw up off the floor. Looking forward to the meetup.
>
> Venkat
>
> On 9/13/13 11:11 AM, "Shivaram Venkataraman"
> <sh...@gmail.com> wrote:
>
>>Actually there is a problem with the AMP Camp 3 scripts now that the
>>package names have changed. We pull from master of MLI, BlinkDB that
>>depend on org.apache.spark but the package published on the AMI is
>>with old package names. I will push out script changes that will sync
>>MLI and BlinkDB to the version that we used for the Camp and ping this
>>thread soon.
>>
>>Thanks
>>Shivaram
>>
>>On Fri, Sep 13, 2013 at 8:05 AM, Jey Kottalam <je...@cs.berkeley.edu> wrote:
>>> Hi Venkat,
>>>
>>> The 0.8.0 branch has not yet been released (and furthermore, the
>>> 0.8.0-SNAPSHOT version refers to a prerelease build). You can install
>>> Spark 0.8.0-SNAPSHOT to your local repository by checking out the
>>> latest source code from http://github.com/mesos/spark and running:
>>>
>>> sbt/sbt publish-local
>>>
>>> If you are using Hadoop, specify the Hadoop version required as follows:
>>>
>>> SPARK_HADOOP_VERSION=1.0.4 sbt/sbt publish-local
>>>
>>> Hope that helps,
>>> -Jey
>>>
>>> On Thu, Sep 12, 2013 at 10:09 PM, Venkat Krishnamurthy
>>> <ve...@yarcdata.com> wrote:
>>>> All
>>>>
>>>> Apologies if this post is misdirected. I'm trying to get started with
>>>>Spark,
>>>> and using the AMPLab hands-on as a guide The set up went as far as
>>>> creating and spinning up the AMI's, but part way through is this
>>>>dependency
>>>> resolution error from sbt:
>>>>
>>>> Seems simple enough, but the failed dependency is spark-coreŠ which I
>>>>assume
>>>> is An Important Thing.
>>>>
>>>> I'm hoping this is embarrassingly obvious to find and fix (or ignore)
>>>>- any
>>>> pointers?
>>>>
>>>> -----
>>>>
>>>> [warn] module not found:
>>>>org.apache.spark#spark-core_2.9.3;0.8.0-SNAPSHOT
>>>> [warn] ==== local: tried
>>>> [warn]
>>>>
>>>>/root/.ivy2/local/org.apache.spark/spark-core_2.9.3/0.8.0-SNAPSHOT/ivys/
>>>>ivy.xml
>>>> [warn] ==== Typesafe Repository: tried
>>>> [warn]
>>>>
>>>>http://repo.typesafe.com/typesafe/releases/org/apache/spark/spark-core_2
>>>>.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
>>>> [warn] ==== JBoss Repository: tried
>>>> [warn]
>>>>
>>>>http://repository.jboss.org/nexus/content/repositories/releases/org/apac
>>>>he/spark/spark-core_2.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT
>>>>.pom
>>>> [warn] ==== Spray Repository: tried
>>>> [warn]
>>>>
>>>>http://repo.spray.cc/org/apache/spark/spark-core_2.9.3/0.8.0-SNAPSHOT/sp
>>>>ark-core_2.9.3-0.8.0-SNAPSHOT.pom
>>>> [warn] ==== Cloudera Repository: tried
>>>> [warn]
>>>>
>>>>https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/sp
>>>>ark/spark-core_2.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
>>>> [warn] ==== Local Maven: tried
>>>> [warn]
>>>>
>>>>file:/root/.m2/repository/org/apache/spark/spark-core_2.9.3/0.8.0-SNAPSH
>>>>OT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
>>>> [warn] ==== public: tried
>>>> [warn]
>>>>
>>>>http://repo1.maven.org/maven2/org/apache/spark/spark-core_2.9.3/0.8.0-SN
>>>>APSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
>>>> [info] Resolving org.apache.spark#spark-repl_2.9.3;0.8.0-SNAPSHOT ...
>>>> [warn] module not found:
>>>>org.apache.spark#spark-repl_2.9.3;0.8.0-SNAPSHOT
>>>> [warn] ==== local: tried
>>>> [warn]
>>>>
>>>>/root/.ivy2/local/org.apache.spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/ivys/
>>>>ivy.xml
>>>> [warn] ==== Typesafe Repository: tried
>>>> [warn]
>>>>
>>>>http://repo.typesafe.com/typesafe/releases/org/apache/spark/spark-repl_2
>>>>.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
>>>> [warn] ==== JBoss Repository: tried
>>>> [warn]
>>>>
>>>>http://repository.jboss.org/nexus/content/repositories/releases/org/apac
>>>>he/spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT
>>>>.pom
>>>> [warn] ==== Spray Repository: tried
>>>> [warn]
>>>>
>>>>http://repo.spray.cc/org/apache/spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/sp
>>>>ark-repl_2.9.3-0.8.0-SNAPSHOT.pom
>>>> [warn] ==== Cloudera Repository: tried
>>>> [warn]
>>>>
>>>>https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/sp
>>>>ark/spark-repl_2.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
>>>> [warn] ==== Local Maven: tried
>>>> [warn]
>>>>
>>>>file:/root/.m2/repository/org/apache/spark/spark-repl_2.9.3/0.8.0-SNAPSH
>>>>OT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
>>>> [warn] ==== public: tried
>>>> [warn]
>>>>
>>>>http://repo1.maven.org/maven2/org/apache/spark/spark-repl_2.9.3/0.8.0-SN
>>>>APSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
>>>> [info] Resolving org.scala-tools.testing#test-interface;0.5 ...
>>>> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
>>>> [warn] :: UNRESOLVED DEPENDENCIES ::
>>>> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
>>>> [warn] :: org.apache.spark#spark-core_2.9.3;0.8.0-SNAPSHOT: not found
>>>> [warn] :: org.apache.spark#spark-repl_2.9.3;0.8.0-SNAPSHOT: not found
>>>> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
>>>> sbt.ResolveException: unresolved dependency:
>>>> org.apache.spark#spark-core_2.9.3;0.8.0-SNAPSHOT: not found
>>>>
>
Re: N00b alert - spark-ec2 script failure
Posted by Venkat Krishnamurthy <ve...@yarcdata.com>.
Just an update - I actually managed to work through the Day 1 tutorial
with zero hiccups so far (up to the twitter exercise) so this didn't
really didn't prevent me from working.
Will update if I face any issues on day 2 exercises.
I'm still picking my jaw up off the floor. Looking forward to the meetup.
Venkat
On 9/13/13 11:11 AM, "Shivaram Venkataraman"
<sh...@gmail.com> wrote:
>Actually there is a problem with the AMP Camp 3 scripts now that the
>package names have changed. We pull from master of MLI, BlinkDB that
>depend on org.apache.spark but the package published on the AMI is
>with old package names. I will push out script changes that will sync
>MLI and BlinkDB to the version that we used for the Camp and ping this
>thread soon.
>
>Thanks
>Shivaram
>
>On Fri, Sep 13, 2013 at 8:05 AM, Jey Kottalam <je...@cs.berkeley.edu> wrote:
>> Hi Venkat,
>>
>> The 0.8.0 branch has not yet been released (and furthermore, the
>> 0.8.0-SNAPSHOT version refers to a prerelease build). You can install
>> Spark 0.8.0-SNAPSHOT to your local repository by checking out the
>> latest source code from http://github.com/mesos/spark and running:
>>
>> sbt/sbt publish-local
>>
>> If you are using Hadoop, specify the Hadoop version required as follows:
>>
>> SPARK_HADOOP_VERSION=1.0.4 sbt/sbt publish-local
>>
>> Hope that helps,
>> -Jey
>>
>> On Thu, Sep 12, 2013 at 10:09 PM, Venkat Krishnamurthy
>> <ve...@yarcdata.com> wrote:
>>> All
>>>
>>> Apologies if this post is misdirected. I'm trying to get started with
>>>Spark,
>>> and using the AMPLab hands-on as a guide The set up went as far as
>>> creating and spinning up the AMI's, but part way through is this
>>>dependency
>>> resolution error from sbt:
>>>
>>> Seems simple enough, but the failed dependency is spark-coreŠ which I
>>>assume
>>> is An Important Thing.
>>>
>>> I'm hoping this is embarrassingly obvious to find and fix (or ignore)
>>>- any
>>> pointers?
>>>
>>> -----
>>>
>>> [warn] module not found:
>>>org.apache.spark#spark-core_2.9.3;0.8.0-SNAPSHOT
>>> [warn] ==== local: tried
>>> [warn]
>>>
>>>/root/.ivy2/local/org.apache.spark/spark-core_2.9.3/0.8.0-SNAPSHOT/ivys/
>>>ivy.xml
>>> [warn] ==== Typesafe Repository: tried
>>> [warn]
>>>
>>>http://repo.typesafe.com/typesafe/releases/org/apache/spark/spark-core_2
>>>.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
>>> [warn] ==== JBoss Repository: tried
>>> [warn]
>>>
>>>http://repository.jboss.org/nexus/content/repositories/releases/org/apac
>>>he/spark/spark-core_2.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT
>>>.pom
>>> [warn] ==== Spray Repository: tried
>>> [warn]
>>>
>>>http://repo.spray.cc/org/apache/spark/spark-core_2.9.3/0.8.0-SNAPSHOT/sp
>>>ark-core_2.9.3-0.8.0-SNAPSHOT.pom
>>> [warn] ==== Cloudera Repository: tried
>>> [warn]
>>>
>>>https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/sp
>>>ark/spark-core_2.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
>>> [warn] ==== Local Maven: tried
>>> [warn]
>>>
>>>file:/root/.m2/repository/org/apache/spark/spark-core_2.9.3/0.8.0-SNAPSH
>>>OT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
>>> [warn] ==== public: tried
>>> [warn]
>>>
>>>http://repo1.maven.org/maven2/org/apache/spark/spark-core_2.9.3/0.8.0-SN
>>>APSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
>>> [info] Resolving org.apache.spark#spark-repl_2.9.3;0.8.0-SNAPSHOT ...
>>> [warn] module not found:
>>>org.apache.spark#spark-repl_2.9.3;0.8.0-SNAPSHOT
>>> [warn] ==== local: tried
>>> [warn]
>>>
>>>/root/.ivy2/local/org.apache.spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/ivys/
>>>ivy.xml
>>> [warn] ==== Typesafe Repository: tried
>>> [warn]
>>>
>>>http://repo.typesafe.com/typesafe/releases/org/apache/spark/spark-repl_2
>>>.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
>>> [warn] ==== JBoss Repository: tried
>>> [warn]
>>>
>>>http://repository.jboss.org/nexus/content/repositories/releases/org/apac
>>>he/spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT
>>>.pom
>>> [warn] ==== Spray Repository: tried
>>> [warn]
>>>
>>>http://repo.spray.cc/org/apache/spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/sp
>>>ark-repl_2.9.3-0.8.0-SNAPSHOT.pom
>>> [warn] ==== Cloudera Repository: tried
>>> [warn]
>>>
>>>https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/sp
>>>ark/spark-repl_2.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
>>> [warn] ==== Local Maven: tried
>>> [warn]
>>>
>>>file:/root/.m2/repository/org/apache/spark/spark-repl_2.9.3/0.8.0-SNAPSH
>>>OT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
>>> [warn] ==== public: tried
>>> [warn]
>>>
>>>http://repo1.maven.org/maven2/org/apache/spark/spark-repl_2.9.3/0.8.0-SN
>>>APSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
>>> [info] Resolving org.scala-tools.testing#test-interface;0.5 ...
>>> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
>>> [warn] :: UNRESOLVED DEPENDENCIES ::
>>> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
>>> [warn] :: org.apache.spark#spark-core_2.9.3;0.8.0-SNAPSHOT: not found
>>> [warn] :: org.apache.spark#spark-repl_2.9.3;0.8.0-SNAPSHOT: not found
>>> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
>>> sbt.ResolveException: unresolved dependency:
>>> org.apache.spark#spark-core_2.9.3;0.8.0-SNAPSHOT: not found
>>>
Re: N00b alert - spark-ec2 script failure
Posted by Shivaram Venkataraman <sh...@gmail.com>.
Actually there is a problem with the AMP Camp 3 scripts now that the
package names have changed. We pull from master of MLI, BlinkDB that
depend on org.apache.spark but the package published on the AMI is
with old package names. I will push out script changes that will sync
MLI and BlinkDB to the version that we used for the Camp and ping this
thread soon.
Thanks
Shivaram
On Fri, Sep 13, 2013 at 8:05 AM, Jey Kottalam <je...@cs.berkeley.edu> wrote:
> Hi Venkat,
>
> The 0.8.0 branch has not yet been released (and furthermore, the
> 0.8.0-SNAPSHOT version refers to a prerelease build). You can install
> Spark 0.8.0-SNAPSHOT to your local repository by checking out the
> latest source code from http://github.com/mesos/spark and running:
>
> sbt/sbt publish-local
>
> If you are using Hadoop, specify the Hadoop version required as follows:
>
> SPARK_HADOOP_VERSION=1.0.4 sbt/sbt publish-local
>
> Hope that helps,
> -Jey
>
> On Thu, Sep 12, 2013 at 10:09 PM, Venkat Krishnamurthy
> <ve...@yarcdata.com> wrote:
>> All
>>
>> Apologies if this post is misdirected. I'm trying to get started with Spark,
>> and using the AMPLab hands-on as a guide – The set up went as far as
>> creating and spinning up the AMI's, but part way through is this dependency
>> resolution error from sbt:
>>
>> Seems simple enough, but the failed dependency is spark-core… which I assume
>> is An Important Thing.
>>
>> I'm hoping this is embarrassingly obvious to find and fix (or ignore) - any
>> pointers?
>>
>> -----
>>
>> [warn] module not found: org.apache.spark#spark-core_2.9.3;0.8.0-SNAPSHOT
>> [warn] ==== local: tried
>> [warn]
>> /root/.ivy2/local/org.apache.spark/spark-core_2.9.3/0.8.0-SNAPSHOT/ivys/ivy.xml
>> [warn] ==== Typesafe Repository: tried
>> [warn]
>> http://repo.typesafe.com/typesafe/releases/org/apache/spark/spark-core_2.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
>> [warn] ==== JBoss Repository: tried
>> [warn]
>> http://repository.jboss.org/nexus/content/repositories/releases/org/apache/spark/spark-core_2.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
>> [warn] ==== Spray Repository: tried
>> [warn]
>> http://repo.spray.cc/org/apache/spark/spark-core_2.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
>> [warn] ==== Cloudera Repository: tried
>> [warn]
>> https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-core_2.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
>> [warn] ==== Local Maven: tried
>> [warn]
>> file:/root/.m2/repository/org/apache/spark/spark-core_2.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
>> [warn] ==== public: tried
>> [warn]
>> http://repo1.maven.org/maven2/org/apache/spark/spark-core_2.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
>> [info] Resolving org.apache.spark#spark-repl_2.9.3;0.8.0-SNAPSHOT ...
>> [warn] module not found: org.apache.spark#spark-repl_2.9.3;0.8.0-SNAPSHOT
>> [warn] ==== local: tried
>> [warn]
>> /root/.ivy2/local/org.apache.spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/ivys/ivy.xml
>> [warn] ==== Typesafe Repository: tried
>> [warn]
>> http://repo.typesafe.com/typesafe/releases/org/apache/spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
>> [warn] ==== JBoss Repository: tried
>> [warn]
>> http://repository.jboss.org/nexus/content/repositories/releases/org/apache/spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
>> [warn] ==== Spray Repository: tried
>> [warn]
>> http://repo.spray.cc/org/apache/spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
>> [warn] ==== Cloudera Repository: tried
>> [warn]
>> https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
>> [warn] ==== Local Maven: tried
>> [warn]
>> file:/root/.m2/repository/org/apache/spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
>> [warn] ==== public: tried
>> [warn]
>> http://repo1.maven.org/maven2/org/apache/spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
>> [info] Resolving org.scala-tools.testing#test-interface;0.5 ...
>> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
>> [warn] :: UNRESOLVED DEPENDENCIES ::
>> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
>> [warn] :: org.apache.spark#spark-core_2.9.3;0.8.0-SNAPSHOT: not found
>> [warn] :: org.apache.spark#spark-repl_2.9.3;0.8.0-SNAPSHOT: not found
>> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
>> sbt.ResolveException: unresolved dependency:
>> org.apache.spark#spark-core_2.9.3;0.8.0-SNAPSHOT: not found
>>
Re: N00b alert - spark-ec2 script failure
Posted by Jey Kottalam <je...@cs.berkeley.edu>.
Hi Venkat,
The 0.8.0 branch has not yet been released (and furthermore, the
0.8.0-SNAPSHOT version refers to a prerelease build). You can install
Spark 0.8.0-SNAPSHOT to your local repository by checking out the
latest source code from http://github.com/mesos/spark and running:
sbt/sbt publish-local
If you are using Hadoop, specify the Hadoop version required as follows:
SPARK_HADOOP_VERSION=1.0.4 sbt/sbt publish-local
Hope that helps,
-Jey
On Thu, Sep 12, 2013 at 10:09 PM, Venkat Krishnamurthy
<ve...@yarcdata.com> wrote:
> All
>
> Apologies if this post is misdirected. I'm trying to get started with Spark,
> and using the AMPLab hands-on as a guide – The set up went as far as
> creating and spinning up the AMI's, but part way through is this dependency
> resolution error from sbt:
>
> Seems simple enough, but the failed dependency is spark-core… which I assume
> is An Important Thing.
>
> I'm hoping this is embarrassingly obvious to find and fix (or ignore) - any
> pointers?
>
> -----
>
> [warn] module not found: org.apache.spark#spark-core_2.9.3;0.8.0-SNAPSHOT
> [warn] ==== local: tried
> [warn]
> /root/.ivy2/local/org.apache.spark/spark-core_2.9.3/0.8.0-SNAPSHOT/ivys/ivy.xml
> [warn] ==== Typesafe Repository: tried
> [warn]
> http://repo.typesafe.com/typesafe/releases/org/apache/spark/spark-core_2.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
> [warn] ==== JBoss Repository: tried
> [warn]
> http://repository.jboss.org/nexus/content/repositories/releases/org/apache/spark/spark-core_2.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
> [warn] ==== Spray Repository: tried
> [warn]
> http://repo.spray.cc/org/apache/spark/spark-core_2.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
> [warn] ==== Cloudera Repository: tried
> [warn]
> https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-core_2.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
> [warn] ==== Local Maven: tried
> [warn]
> file:/root/.m2/repository/org/apache/spark/spark-core_2.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
> [warn] ==== public: tried
> [warn]
> http://repo1.maven.org/maven2/org/apache/spark/spark-core_2.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
> [info] Resolving org.apache.spark#spark-repl_2.9.3;0.8.0-SNAPSHOT ...
> [warn] module not found: org.apache.spark#spark-repl_2.9.3;0.8.0-SNAPSHOT
> [warn] ==== local: tried
> [warn]
> /root/.ivy2/local/org.apache.spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/ivys/ivy.xml
> [warn] ==== Typesafe Repository: tried
> [warn]
> http://repo.typesafe.com/typesafe/releases/org/apache/spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
> [warn] ==== JBoss Repository: tried
> [warn]
> http://repository.jboss.org/nexus/content/repositories/releases/org/apache/spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
> [warn] ==== Spray Repository: tried
> [warn]
> http://repo.spray.cc/org/apache/spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
> [warn] ==== Cloudera Repository: tried
> [warn]
> https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
> [warn] ==== Local Maven: tried
> [warn]
> file:/root/.m2/repository/org/apache/spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
> [warn] ==== public: tried
> [warn]
> http://repo1.maven.org/maven2/org/apache/spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
> [info] Resolving org.scala-tools.testing#test-interface;0.5 ...
> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
> [warn] :: UNRESOLVED DEPENDENCIES ::
> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
> [warn] :: org.apache.spark#spark-core_2.9.3;0.8.0-SNAPSHOT: not found
> [warn] :: org.apache.spark#spark-repl_2.9.3;0.8.0-SNAPSHOT: not found
> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
> sbt.ResolveException: unresolved dependency:
> org.apache.spark#spark-core_2.9.3;0.8.0-SNAPSHOT: not found
>
Re: N00b alert - spark-ec2 script failure
Posted by Reynold Xin <rx...@cs.berkeley.edu>.
You will need to do
sbt/sbt publish-local
in Spark for this to work. The reason is that the spark packages haven't
been published to maven yet. They will be when they are released.
--
Reynold Xin, AMPLab, UC Berkeley
http://rxin.org
On Fri, Sep 13, 2013 at 1:09 PM, Venkat Krishnamurthy
<ve...@yarcdata.com>wrote:
> All
>
> Apologies if this post is misdirected. I'm trying to get started with
> Spark, and using the AMPLab hands-on as a guide – The set up went as far as
> creating and spinning up the AMI's, but part way through is this dependency
> resolution error from sbt:
>
> Seems simple enough, but the failed dependency is *spark-core… *which I
> assume is An Important Thing.
>
> I'm hoping this is embarrassingly obvious to find and fix (or ignore) -
> any pointers?
>
> -----
>
> [warn] module not found: org.apache.spark#spark-core_2.9.3;0.8.0-SNAPSHOT
> [warn] ==== local: tried
> [warn]
> /root/.ivy2/local/org.apache.spark/spark-core_2.9.3/0.8.0-SNAPSHOT/ivys/ivy.xml
> [warn] ==== Typesafe Repository: tried
> [warn]
> http://repo.typesafe.com/typesafe/releases/org/apache/spark/spark-core_2.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
> [warn] ==== JBoss Repository: tried
> [warn]
> http://repository.jboss.org/nexus/content/repositories/releases/org/apache/spark/spark-core_2.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
> [warn] ==== Spray Repository: tried
> [warn]
> http://repo.spray.cc/org/apache/spark/spark-core_2.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
> [warn] ==== Cloudera Repository: tried
> [warn]
> https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-core_2.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
> [warn] ==== Local Maven: tried
> [warn]
> file:/root/.m2/repository/org/apache/spark/spark-core_2.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
> [warn] ==== public: tried
> [warn]
> http://repo1.maven.org/maven2/org/apache/spark/spark-core_2.9.3/0.8.0-SNAPSHOT/spark-core_2.9.3-0.8.0-SNAPSHOT.pom
> [info] Resolving org.apache.spark#spark-repl_2.9.3;0.8.0-SNAPSHOT ...
> [warn] module not found: org.apache.spark#spark-repl_2.9.3;0.8.0-SNAPSHOT
> [warn] ==== local: tried
> [warn]
> /root/.ivy2/local/org.apache.spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/ivys/ivy.xml
> [warn] ==== Typesafe Repository: tried
> [warn]
> http://repo.typesafe.com/typesafe/releases/org/apache/spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
> [warn] ==== JBoss Repository: tried
> [warn]
> http://repository.jboss.org/nexus/content/repositories/releases/org/apache/spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
> [warn] ==== Spray Repository: tried
> [warn]
> http://repo.spray.cc/org/apache/spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
> [warn] ==== Cloudera Repository: tried
> [warn]
> https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
> [warn] ==== Local Maven: tried
> [warn]
> file:/root/.m2/repository/org/apache/spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
> [warn] ==== public: tried
> [warn]
> http://repo1.maven.org/maven2/org/apache/spark/spark-repl_2.9.3/0.8.0-SNAPSHOT/spark-repl_2.9.3-0.8.0-SNAPSHOT.pom
> [info] Resolving org.scala-tools.testing#test-interface;0.5 ...
> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
> [warn] :: UNRESOLVED DEPENDENCIES ::
> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
> [warn] :: org.apache.spark#spark-core_2.9.3;0.8.0-SNAPSHOT: not found
> [warn] :: org.apache.spark#spark-repl_2.9.3;0.8.0-SNAPSHOT: not found
> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
> sbt.ResolveException: unresolved dependency:
> org.apache.spark#spark-core_2.9.3;0.8.0-SNAPSHOT: not found
>
>