You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jianshi Huang <ji...@gmail.com> on 2014/11/18 07:49:08 UTC
Is it safe to use Scala 2.11 for Spark build?
Any notable issues for using Scala 2.11? Is it stable now?
Or can I use Scala 2.11 in my spark application and use Spark dist build
with 2.10 ?
I'm looking forward to migrate to 2.11 for some quasiquote features.
Couldn't make it run in 2.10...
Cheers,
--
Jianshi Huang
LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/
Re: Is it safe to use Scala 2.11 for Spark build?
Posted by Jianshi Huang <ji...@gmail.com>.
Ok, I'll wait until -Pscala-2.11 is more stable and used by more people.
Thanks for the help!
Jianshi
On Tue, Nov 18, 2014 at 3:49 PM, Ye Xianjin <ad...@gmail.com> wrote:
> Hi Prashant Sharma,
>
> It's not even ok to build with scala-2.11 profile on my machine.
>
> Just check out the master(c6e0c2ab1c29c184a9302d23ad75e4ccd8060242)
> run sbt/sbt -Pscala-2.11 clean assembly:
>
> ...... skip the normal part
> info] Resolving org.scalamacros#quasiquotes_2.11;2.0.1 ...
> [warn] module not found: org.scalamacros#quasiquotes_2.11;2.0.1
> [warn] ==== local: tried
> [warn]
> /Users/yexianjin/.ivy2/local/org.scalamacros/quasiquotes_2.11/2.0.1/ivys/ivy.xml
> [warn] ==== public: tried
> [warn]
> https://repo1.maven.org/maven2/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
> [warn] ==== central: tried
> [warn]
> https://repo1.maven.org/maven2/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
> [warn] ==== apache-repo: tried
> [warn]
> https://repository.apache.org/content/repositories/releases/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
> [warn] ==== jboss-repo: tried
> [warn]
> https://repository.jboss.org/nexus/content/repositories/releases/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
> [warn] ==== mqtt-repo: tried
> [warn]
> https://repo.eclipse.org/content/repositories/paho-releases/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
> [warn] ==== cloudera-repo: tried
> [warn]
> https://repository.cloudera.com/artifactory/cloudera-repos/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
> [warn] ==== mapr-repo: tried
> [warn]
> http://repository.mapr.com/maven/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
> [warn] ==== spring-releases: tried
> [warn]
> https://repo.spring.io/libs-release/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
> [warn] ==== spark-staging: tried
> [warn]
> https://oss.sonatype.org/content/repositories/orgspark-project-1085/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
> [warn] ==== spark-staging-hive13: tried
> [warn]
> https://oss.sonatype.org/content/repositories/orgspark-project-1089/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
> [warn] ==== apache.snapshots: tried
> [warn]
> http://repository.apache.org/snapshots/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
> [warn] ==== Maven2 Local: tried
> [warn]
> file:/Users/yexianjin/.m2/repository/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
> [info] Resolving jline#jline;2.12 ...
> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
> [warn] :: UNRESOLVED DEPENDENCIES ::
> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
> [warn] :: org.scalamacros#quasiquotes_2.11;2.0.1: not found
> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
> [info] Resolving org.scala-lang#scala-library;2.11.2 ...
> [warn]
> [warn] Note: Unresolved dependencies path:
> [warn] org.scalamacros:quasiquotes_2.11:2.0.1
> ((com.typesafe.sbt.pom.MavenHelper) MavenHelper.scala#L76)
> [warn] +- org.apache.spark:spark-catalyst_2.11:1.2.0-SNAPSHOT
> [info] Resolving jline#jline;2.12 ...
> [info] Done updating.
> [info] Updating {file:/Users/yexianjin/spark/}streaming-twitter...
> [info] Updating {file:/Users/yexianjin/spark/}streaming-zeromq...
> [info] Updating {file:/Users/yexianjin/spark/}streaming-flume...
> [info] Updating {file:/Users/yexianjin/spark/}streaming-mqtt...
> [info] Resolving jline#jline;2.12 ...
> [info] Done updating.
> [info] Resolving com.esotericsoftware.minlog#minlog;1.2 ...
> [info] Updating {file:/Users/yexianjin/spark/}streaming-kafka...
> [info] Resolving jline#jline;2.12 ...
> [info] Done updating.
> [info] Resolving jline#jline;2.12 ...
> [info] Done updating.
> [info] Resolving jline#jline;2.12 ...
> [info] Done updating.
> [info] Resolving org.apache.kafka#kafka_2.11;0.8.0 ...
> [warn] module not found: org.apache.kafka#kafka_2.11;0.8.0
> [warn] ==== local: tried
> [warn]
> /Users/yexianjin/.ivy2/local/org.apache.kafka/kafka_2.11/0.8.0/ivys/ivy.xml
> [warn] ==== public: tried
> [warn]
> https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
> [warn] ==== central: tried
> [warn]
> https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
> [warn] ==== apache-repo: tried
> [warn]
> https://repository.apache.org/content/repositories/releases/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
> [warn] ==== jboss-repo: tried
> [warn]
> https://repository.jboss.org/nexus/content/repositories/releases/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
> [warn] ==== mqtt-repo: tried
> [warn]
> https://repo.eclipse.org/content/repositories/paho-releases/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
> [warn] ==== cloudera-repo: tried
> [warn]
> https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
> [warn] ==== mapr-repo: tried
> [warn]
> http://repository.mapr.com/maven/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
> [warn] ==== spring-releases: tried
> [warn]
> https://repo.spring.io/libs-release/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
> [warn] ==== spark-staging: tried
> [warn]
> https://oss.sonatype.org/content/repositories/orgspark-project-1085/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
> [warn] ==== spark-staging-hive13: tried
> [warn]
> https://oss.sonatype.org/content/repositories/orgspark-project-1089/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
> [warn] ==== apache.snapshots: tried
> [warn]
> http://repository.apache.org/snapshots/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
> [warn] ==== Maven2 Local: tried
> [warn]
> file:/Users/yexianjin/.m2/repository/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
> [info] Resolving jline#jline;2.12 ...
> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
> [warn] :: UNRESOLVED DEPENDENCIES ::
> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
> [warn] :: org.apache.kafka#kafka_2.11;0.8.0: not found
> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
> [warn]
> [warn] Note: Unresolved dependencies path:
> [warn] org.apache.kafka:kafka_2.11:0.8.0
> ((com.typesafe.sbt.pom.MavenHelper) MavenHelper.scala#L76)
> [warn] +- org.apache.spark:spark-streaming-kafka_2.11:1.2.0-SNAPSHOT
> sbt.ResolveException: unresolved dependency:
> org.apache.kafka#kafka_2.11;0.8.0: not found
> at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:243)
> at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:158)
> at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:156)
> at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:147)
> at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:147)
> at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:124)
> at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:56)
> at sbt.IvySbt$$anon$3.call(Ivy.scala:64)
> at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:93)
> at
> xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:78)
> at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:97)
> at xsbt.boot.Using$.withResource(Using.scala:10)
> at xsbt.boot.Using$.apply(Using.scala:9)
> at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:58)
> at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:48)
> at xsbt.boot.Locks$.apply0(Locks.scala:31)
> at xsbt.boot.Locks$.apply(Locks.scala:28)
> at sbt.IvySbt.withDefaultLogger(Ivy.scala:64)
> at sbt.IvySbt.withIvy(Ivy.scala:119)
> at sbt.IvySbt.withIvy(Ivy.scala:116)
> at sbt.IvySbt$Module.withModule(Ivy.scala:147)
> at sbt.IvyActions$.updateEither(IvyActions.scala:156)
> at
> sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1282)
> at
> sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1279)
> at
> sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$84.apply(Defaults.scala:1309)
> at
> sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$84.apply(Defaults.scala:1307)
> at sbt.Tracked$$anonfun$lastOutput$1.apply(Tracked.scala:35)
> at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1312)
> at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1306)
> at sbt.Tracked$$anonfun$inputChanged$1.apply(Tracked.scala:45)
> at sbt.Classpaths$.cachedUpdate(Defaults.scala:1324)
> at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1264)
> at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1242)
> at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
> at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
> at sbt.std.Transform$$anon$4.work(System.scala:63)
> at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
> at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
> at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
> at sbt.Execute.work(Execute.scala:235)
> at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
> at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
> at
> sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
> at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> sbt.ResolveException: unresolved dependency:
> org.scalamacros#quasiquotes_2.11;2.0.1: not found
> at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:243)
> at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:158)
> at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:156)
> at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:147)
> at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:147)
> at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:124)
> at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:56)
> at sbt.IvySbt$$anon$3.call(Ivy.scala:64)
> at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:93)
> at
> xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:78)
> at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:97)
> at xsbt.boot.Using$.withResource(Using.scala:10)
> at xsbt.boot.Using$.apply(Using.scala:9)
> at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:58)
> at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:48)
> commit c6e0c2ab1c29c184a9302d23ad75e4ccd8060242
> at xsbt.boot.Locks$.apply0(Locks.scala:31)
> at xsbt.boot.Locks$.apply(Locks.scala:28)
> at sbt.IvySbt.withDefaultLogger(Ivy.scala:64)
> at sbt.IvySbt.withIvy(Ivy.scala:119)
> at sbt.IvySbt.withIvy(Ivy.scala:116)
> at sbt.IvySbt$Module.withModule(Ivy.scala:147)
> at sbt.IvyActions$.updateEither(IvyActions.scala:156)
> at
> sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1282)
> at
> sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1279)
> at
> sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$84.apply(Defaults.scala:1309)
> at
> sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$84.apply(Defaults.scala:1307)
> at sbt.Tracked$$anonfun$lastOutput$1.apply(Tracked.scala:35)
> at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1312)
> at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1306)
> at sbt.Tracked$$anonfun$inputChanged$1.apply(Tracked.scala:45)
> at sbt.Classpaths$.cachedUpdate(Defaults.scala:1324)
> at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1264)
> at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1242)
> at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
> at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
> at sbt.std.Transform$$anon$4.work(System.scala:63)
> at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
> at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
> at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
> at sbt.Execute.work(Execute.scala:235)
> at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
> at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
> at
> sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
> at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> [error] (streaming-kafka/*:update) sbt.ResolveException: unresolved
> dependency: org.apache.kafka#kafka_2.11;0.8.0: not found
> [error] (catalyst/*:update) sbt.ResolveException: unresolved dependency:
> org.scalamacros#quasiquotes_2.11;2.0.1: not found
>
> --
> Ye Xianjin
> Sent with Sparrow <http://www.sparrowmailapp.com/?sig>
>
> On Tuesday, November 18, 2014 at 3:27 PM, Prashant Sharma wrote:
>
> It is safe in the sense we would help you with the fix if you run into
> issues. I have used it, but since I worked on the patch the opinion can be
> biased. I am using scala 2.11 for day to day development. You should
> checkout the build instructions here :
> https://github.com/ScrapCodes/spark-1/blob/patch-3/docs/building-spark.md
>
> Prashant Sharma
>
>
>
> On Tue, Nov 18, 2014 at 12:19 PM, Jianshi Huang <ji...@gmail.com>
> wrote:
>
> Any notable issues for using Scala 2.11? Is it stable now?
>
> Or can I use Scala 2.11 in my spark application and use Spark dist build
> with 2.10 ?
>
> I'm looking forward to migrate to 2.11 for some quasiquote features.
> Couldn't make it run in 2.10...
>
> Cheers,
> --
> Jianshi Huang
>
> LinkedIn: jianshi
> Twitter: @jshuang
> Github & Blog: http://huangjs.github.com/
>
>
>
>
--
Jianshi Huang
LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/
Re: Is it safe to use Scala 2.11 for Spark build?
Posted by Ye Xianjin <ad...@gmail.com>.
Hi Prashant Sharma,
It's not even ok to build with scala-2.11 profile on my machine.
Just check out the master(c6e0c2ab1c29c184a9302d23ad75e4ccd8060242)
run sbt/sbt -Pscala-2.11 clean assembly:
...... skip the normal part
info] Resolving org.scalamacros#quasiquotes_2.11;2.0.1 ...
[warn] module not found: org.scalamacros#quasiquotes_2.11;2.0.1
[warn] ==== local: tried
[warn] /Users/yexianjin/.ivy2/local/org.scalamacros/quasiquotes_2.11/2.0.1/ivys/ivy.xml
[warn] ==== public: tried
[warn] https://repo1.maven.org/maven2/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
[warn] ==== central: tried
[warn] https://repo1.maven.org/maven2/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
[warn] ==== apache-repo: tried
[warn] https://repository.apache.org/content/repositories/releases/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
[warn] ==== jboss-repo: tried
[warn] https://repository.jboss.org/nexus/content/repositories/releases/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
[warn] ==== mqtt-repo: tried
[warn] https://repo.eclipse.org/content/repositories/paho-releases/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
[warn] ==== cloudera-repo: tried
[warn] https://repository.cloudera.com/artifactory/cloudera-repos/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
[warn] ==== mapr-repo: tried
[warn] http://repository.mapr.com/maven/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
[warn] ==== spring-releases: tried
[warn] https://repo.spring.io/libs-release/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
[warn] ==== spark-staging: tried
[warn] https://oss.sonatype.org/content/repositories/orgspark-project-1085/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
[warn] ==== spark-staging-hive13: tried
[warn] https://oss.sonatype.org/content/repositories/orgspark-project-1089/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
[warn] ==== apache.snapshots: tried
[warn] http://repository.apache.org/snapshots/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
[warn] ==== Maven2 Local: tried
[warn] file:/Users/yexianjin/.m2/repository/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
[info] Resolving jline#jline;2.12 ...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: org.scalamacros#quasiquotes_2.11;2.0.1: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[info] Resolving org.scala-lang#scala-library;2.11.2 ...
[warn]
[warn] Note: Unresolved dependencies path:
[warn] org.scalamacros:quasiquotes_2.11:2.0.1 ((com.typesafe.sbt.pom.MavenHelper) MavenHelper.scala#L76)
[warn] +- org.apache.spark:spark-catalyst_2.11:1.2.0-SNAPSHOT
[info] Resolving jline#jline;2.12 ...
[info] Done updating.
[info] Updating {file:/Users/yexianjin/spark/}streaming-twitter...
[info] Updating {file:/Users/yexianjin/spark/}streaming-zeromq...
[info] Updating {file:/Users/yexianjin/spark/}streaming-flume...
[info] Updating {file:/Users/yexianjin/spark/}streaming-mqtt...
[info] Resolving jline#jline;2.12 ...
[info] Done updating.
[info] Resolving com.esotericsoftware.minlog#minlog;1.2 ...
[info] Updating {file:/Users/yexianjin/spark/}streaming-kafka...
[info] Resolving jline#jline;2.12 ...
[info] Done updating.
[info] Resolving jline#jline;2.12 ...
[info] Done updating.
[info] Resolving jline#jline;2.12 ...
[info] Done updating.
[info] Resolving org.apache.kafka#kafka_2.11;0.8.0 ...
[warn] module not found: org.apache.kafka#kafka_2.11;0.8.0
[warn] ==== local: tried
[warn] /Users/yexianjin/.ivy2/local/org.apache.kafka/kafka_2.11/0.8.0/ivys/ivy.xml
[warn] ==== public: tried
[warn] https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
[warn] ==== central: tried
[warn] https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
[warn] ==== apache-repo: tried
[warn] https://repository.apache.org/content/repositories/releases/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
[warn] ==== jboss-repo: tried
[warn] https://repository.jboss.org/nexus/content/repositories/releases/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
[warn] ==== mqtt-repo: tried
[warn] https://repo.eclipse.org/content/repositories/paho-releases/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
[warn] ==== cloudera-repo: tried
[warn] https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
[warn] ==== mapr-repo: tried
[warn] http://repository.mapr.com/maven/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
[warn] ==== spring-releases: tried
[warn] https://repo.spring.io/libs-release/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
[warn] ==== spark-staging: tried
[warn] https://oss.sonatype.org/content/repositories/orgspark-project-1085/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
[warn] ==== spark-staging-hive13: tried
[warn] https://oss.sonatype.org/content/repositories/orgspark-project-1089/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
[warn] ==== apache.snapshots: tried
[warn] http://repository.apache.org/snapshots/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
[warn] ==== Maven2 Local: tried
[warn] file:/Users/yexianjin/.m2/repository/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
[info] Resolving jline#jline;2.12 ...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: org.apache.kafka#kafka_2.11;0.8.0: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] Note: Unresolved dependencies path:
[warn] org.apache.kafka:kafka_2.11:0.8.0 ((com.typesafe.sbt.pom.MavenHelper) MavenHelper.scala#L76)
[warn] +- org.apache.spark:spark-streaming-kafka_2.11:1.2.0-SNAPSHOT
sbt.ResolveException: unresolved dependency: org.apache.kafka#kafka_2.11;0.8.0: not found
at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:243)
at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:158)
at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:156)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:147)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:147)
at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:124)
at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:56)
at sbt.IvySbt$$anon$3.call(Ivy.scala:64)
at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:93)
at xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:78)
at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:97)
at xsbt.boot.Using$.withResource(Using.scala:10)
at xsbt.boot.Using$.apply(Using.scala:9)
at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:58)
at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:48)
at xsbt.boot.Locks$.apply0(Locks.scala:31)
at xsbt.boot.Locks$.apply(Locks.scala:28)
at sbt.IvySbt.withDefaultLogger(Ivy.scala:64)
at sbt.IvySbt.withIvy(Ivy.scala:119)
at sbt.IvySbt.withIvy(Ivy.scala:116)
at sbt.IvySbt$Module.withModule(Ivy.scala:147)
at sbt.IvyActions$.updateEither(IvyActions.scala:156)
at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1282)
at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1279)
at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$84.apply(Defaults.scala:1309)
at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$84.apply(Defaults.scala:1307)
at sbt.Tracked$$anonfun$lastOutput$1.apply(Tracked.scala:35)
at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1312)
at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1306)
at sbt.Tracked$$anonfun$inputChanged$1.apply(Tracked.scala:45)
at sbt.Classpaths$.cachedUpdate(Defaults.scala:1324)
at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1264)
at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1242)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
at sbt.std.Transform$$anon$4.work(System.scala:63)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
at sbt.Execute.work(Execute.scala:235)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
sbt.ResolveException: unresolved dependency: org.scalamacros#quasiquotes_2.11;2.0.1: not found
at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:243)
at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:158)
at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:156)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:147)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:147)
at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:124)
at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:56)
at sbt.IvySbt$$anon$3.call(Ivy.scala:64)
at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:93)
at xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:78)
at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:97)
at xsbt.boot.Using$.withResource(Using.scala:10)
at xsbt.boot.Using$.apply(Using.scala:9)
at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:58)
at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:48)
commit c6e0c2ab1c29c184a9302d23ad75e4ccd8060242
at xsbt.boot.Locks$.apply0(Locks.scala:31)
at xsbt.boot.Locks$.apply(Locks.scala:28)
at sbt.IvySbt.withDefaultLogger(Ivy.scala:64)
at sbt.IvySbt.withIvy(Ivy.scala:119)
at sbt.IvySbt.withIvy(Ivy.scala:116)
at sbt.IvySbt$Module.withModule(Ivy.scala:147)
at sbt.IvyActions$.updateEither(IvyActions.scala:156)
at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1282)
at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1279)
at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$84.apply(Defaults.scala:1309)
at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$84.apply(Defaults.scala:1307)
at sbt.Tracked$$anonfun$lastOutput$1.apply(Tracked.scala:35)
at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1312)
at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1306)
at sbt.Tracked$$anonfun$inputChanged$1.apply(Tracked.scala:45)
at sbt.Classpaths$.cachedUpdate(Defaults.scala:1324)
at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1264)
at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1242)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
at sbt.std.Transform$$anon$4.work(System.scala:63)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
at sbt.Execute.work(Execute.scala:235)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[error] (streaming-kafka/*:update) sbt.ResolveException: unresolved dependency: org.apache.kafka#kafka_2.11;0.8.0: not found
[error] (catalyst/*:update) sbt.ResolveException: unresolved dependency: org.scalamacros#quasiquotes_2.11;2.0.1: not found
--
Ye Xianjin
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Tuesday, November 18, 2014 at 3:27 PM, Prashant Sharma wrote:
> It is safe in the sense we would help you with the fix if you run into issues. I have used it, but since I worked on the patch the opinion can be biased. I am using scala 2.11 for day to day development. You should checkout the build instructions here : https://github.com/ScrapCodes/spark-1/blob/patch-3/docs/building-spark.md
>
> Prashant Sharma
>
>
>
> On Tue, Nov 18, 2014 at 12:19 PM, Jianshi Huang <jianshi.huang@gmail.com (mailto:jianshi.huang@gmail.com)> wrote:
> > Any notable issues for using Scala 2.11? Is it stable now?
> >
> > Or can I use Scala 2.11 in my spark application and use Spark dist build with 2.10 ?
> >
> > I'm looking forward to migrate to 2.11 for some quasiquote features. Couldn't make it run in 2.10...
> >
> > Cheers,--
> > Jianshi Huang
> >
> > LinkedIn: jianshi
> > Twitter: @jshuang
> > Github & Blog: http://huangjs.github.com/
>
Re: Is it safe to use Scala 2.11 for Spark build?
Posted by Prashant Sharma <sc...@gmail.com>.
Looks like sbt/sbt -Pscala-2.11 is broken by a recent patch for improving
maven build.
Prashant Sharma
On Tue, Nov 18, 2014 at 12:57 PM, Prashant Sharma <sc...@gmail.com>
wrote:
> It is safe in the sense we would help you with the fix if you run into
> issues. I have used it, but since I worked on the patch the opinion can be
> biased. I am using scala 2.11 for day to day development. You should
> checkout the build instructions here :
> https://github.com/ScrapCodes/spark-1/blob/patch-3/docs/building-spark.md
>
> Prashant Sharma
>
>
>
> On Tue, Nov 18, 2014 at 12:19 PM, Jianshi Huang <ji...@gmail.com>
> wrote:
>
>> Any notable issues for using Scala 2.11? Is it stable now?
>>
>> Or can I use Scala 2.11 in my spark application and use Spark dist build
>> with 2.10 ?
>>
>> I'm looking forward to migrate to 2.11 for some quasiquote features.
>> Couldn't make it run in 2.10...
>>
>> Cheers,
>> --
>> Jianshi Huang
>>
>> LinkedIn: jianshi
>> Twitter: @jshuang
>> Github & Blog: http://huangjs.github.com/
>>
>
>
Re: Is it safe to use Scala 2.11 for Spark build?
Posted by Prashant Sharma <sc...@gmail.com>.
It is safe in the sense we would help you with the fix if you run into
issues. I have used it, but since I worked on the patch the opinion can be
biased. I am using scala 2.11 for day to day development. You should
checkout the build instructions here :
https://github.com/ScrapCodes/spark-1/blob/patch-3/docs/building-spark.md
Prashant Sharma
On Tue, Nov 18, 2014 at 12:19 PM, Jianshi Huang <ji...@gmail.com>
wrote:
> Any notable issues for using Scala 2.11? Is it stable now?
>
> Or can I use Scala 2.11 in my spark application and use Spark dist build
> with 2.10 ?
>
> I'm looking forward to migrate to 2.11 for some quasiquote features.
> Couldn't make it run in 2.10...
>
> Cheers,
> --
> Jianshi Huang
>
> LinkedIn: jianshi
> Twitter: @jshuang
> Github & Blog: http://huangjs.github.com/
>