You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Mingyu Kim <mk...@palantir.com> on 2013/10/08 03:18:21 UTC

Spark dependency library causing problems with conflicting versions at import

Hi all,

I'm trying to use spark in our existing code base. However, a lot of spark
dependencies are not updated to the latest versions and they conflict with
our versions of the libraries. Most notably scala-2.9.2 and scala-2.10.1.
Have people run into these problems before? How did you work around this?

A few specific more questions I have are,
* I remember you have a plan to upgrade scala to the latest version of
scala. Is there any ETA?
* I see other open source projects private-namespacing the dependencies to
work around this problem. For example, Elasticsearch prepends dependent
libraries with "org.elasticsearch.". Would it be possible for Spark to take
this path in the future?
* We intend to just create a remote connection from our code base, so it's
not like we really need the full Spark implementation in our code base. Is
it possible to separate out "client" project that only allows remote
connections via SparkContext such that the "client" jar only contains a few
dependencies?
Mingyu

Re: Spark dependency library causing problems with conflicting versions at import

Posted by Matei Zaharia <ma...@gmail.com>.

Hi Mingyu,

The latest version of Spark works with Scala 2.9.3, which is the latest Scala-2.9 version. There's also a branch called branch-2.10 on GitHub that uses 2.10.3. What specific libraries are you having trouble with?
> I see other open source projects private-namespacing the dependencies to work around this problem. For example, Elasticsearch prepends dependent libraries with "org.elasticsearch….". Would it be possible for Spark to take this path in the future?
This can be tough to do because some libraries use reflection or require only one instance per JVM (e.g. Log4J). You may be able to package Spark in this way by modifying the Maven file though.
> We intend to just create a remote connection from our code base, so it's not like we really need the full Spark implementation in our code base. Is it possible to separate out "client" project that only allows remote connections via SparkContext such that the "client" jar only contains a few dependencies?
Yeah good question, this is hard today but might be possible later. I'd recommend writing a little RPC service on top and accessing that.

Matei