You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Oliver Schrenk <ol...@gmail.com> on 2015/08/31 18:33:41 UTC
Managing httpcomponent dependency in Spark/Solr
Hi,
We are running a distibuted indexing service for Solr (4.7) on a Spark (1.2) cluster. Now we wanted to upgrade to Solr 5.3 and are running into problems with dependencies.
Solr 5 brings in org.apache.httpcomponents httpclient 4.4.1 (1) and the prebuilt binary for Spark 1.2.2 for CDH 4 brings in httpclient 4.2.5 which crashes indexing Solr via SolrJ.
Is there a way of not bringing this dependency in? Is there a way of having a different classloader for my client application? I saw that there is `spark.driver.userClassPathFirst`, is it something that would help?
Cheers,
Olive
(1) https://github.com/apache/lucene-solr/blob/trunk/lucene/ivy-versions.properties#L163
(2) cd $HOME/Downloads
wget http://apache.xl-mirror.nl/spark/spark-1.2.2/spark-1.2.2-bin-cdh4.tgz
tar xzvf spark-1.2.2-bin-cdh4.tgz
unzip -p spark-1.2.2-bin-cdh4/lib/spark-assembly-1.2.2-hadoop2.0.0-mr1-cdh4.2.0.jar org/apache/http/version.properties | grep release
info.release = 4.2.5
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Managing httpcomponent dependency in Spark/Solr
Posted by Igor Berman <ig...@gmail.com>.
not sure if it will help, but have you checked
https://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html
On 31 August 2015 at 19:33, Oliver Schrenk <ol...@gmail.com> wrote:
> Hi,
>
> We are running a distibuted indexing service for Solr (4.7) on a Spark
> (1.2) cluster. Now we wanted to upgrade to Solr 5.3 and are running into
> problems with dependencies.
>
> Solr 5 brings in org.apache.httpcomponents httpclient 4.4.1 (1) and the
> prebuilt binary for Spark 1.2.2 for CDH 4 brings in httpclient 4.2.5 which
> crashes indexing Solr via SolrJ.
>
>
>
> Is there a way of not bringing this dependency in? Is there a way of
> having a different classloader for my client application? I saw that there
> is `spark.driver.userClassPathFirst`, is it something that would help?
>
>
>
> Cheers,
> Olive
>
>
>
> (1)
> https://github.com/apache/lucene-solr/blob/trunk/lucene/ivy-versions.properties#L163
> (2) cd $HOME/Downloads
> wget http://apache.xl-mirror.nl/spark/spark-1.2.2/spark-1.2.2-bin-cdh4.tgz
> tar xzvf spark-1.2.2-bin-cdh4.tgz
> unzip -p
> spark-1.2.2-bin-cdh4/lib/spark-assembly-1.2.2-hadoop2.0.0-mr1-cdh4.2.0.jar
> org/apache/http/version.properties | grep release
> info.release = 4.2.5
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Managing httpcomponent dependency in Spark/Solr
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
You can also set the *executor.userClassPathFirst,* There are couple of
classpath configurations available to override defaults, you can find them
from here
http://spark.apache.org/docs/latest/configuration.html#runtime-environment
Thanks
Best Regards
On Mon, Aug 31, 2015 at 10:03 PM, Oliver Schrenk <ol...@gmail.com>
wrote:
> Hi,
>
> We are running a distibuted indexing service for Solr (4.7) on a Spark
> (1.2) cluster. Now we wanted to upgrade to Solr 5.3 and are running into
> problems with dependencies.
>
> Solr 5 brings in org.apache.httpcomponents httpclient 4.4.1 (1) and the
> prebuilt binary for Spark 1.2.2 for CDH 4 brings in httpclient 4.2.5 which
> crashes indexing Solr via SolrJ.
>
>
>
> Is there a way of not bringing this dependency in? Is there a way of
> having a different classloader for my client application? I saw that there
> is `spark.driver.userClassPathFirst`, is it something that would help?
>
>
>
> Cheers,
> Olive
>
>
>
> (1)
> https://github.com/apache/lucene-solr/blob/trunk/lucene/ivy-versions.properties#L163
> (2) cd $HOME/Downloads
> wget http://apache.xl-mirror.nl/spark/spark-1.2.2/spark-1.2.2-bin-cdh4.tgz
> tar xzvf spark-1.2.2-bin-cdh4.tgz
> unzip -p
> spark-1.2.2-bin-cdh4/lib/spark-assembly-1.2.2-hadoop2.0.0-mr1-cdh4.2.0.jar
> org/apache/http/version.properties | grep release
> info.release = 4.2.5
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>