You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by William-Smith <wi...@gmail.com> on 2015/01/23 05:42:48 UTC
Re: spark 1.1.0 (w/ hadoop 2.4) vs aws java sdk 1.7.2
I have had the same issue while using HttpClient from AWS EMR Spark Streaming
to post to a nodejs server.
I have found ... using
Classloder.getResource('org/apache/http/client/HttpClient") .... that the
class
Is being loaded front the spark-assembly-1.1.0-hadoop2.4.0.jar.
That in itself is not the issue because the version is 4.2.5 .... the same
version I am using on my local machine with success .... using Hadoop cdh 5.
The issue is that HttpClient relies on Httpcore .... and there is an old
commons-httpcore-1.3.jar as well as httpcore-4.5.2 in the spark-assembly
jar.
It looks like the old one is getting loaded first.
So the fix might be to build the Spark jar myself without the httpcore-1.3
and replace it on bootstrap.
I will keep you posted on the outcome.
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/spark-1-1-0-w-hadoop-2-4-vs-aws-java-sdk-1-7-2-tp8481p10250.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: spark 1.1.0 (w/ hadoop 2.4) vs aws java sdk 1.7.2
Posted by William-Smith <wi...@gmail.com>.
Hey Sean,
I did find that after my post and tried it
....spark.files.userClassPathFirst = true
But it is deemed in the docs as instrumental .... and did not work.
Monitored that additional config via the logs and it did not complain ... no
change though.
So...
Due to my timeline for a Demo....
Used HttpUrl instead for my POST and abandoned using Apache HTTP for now.
My app works but I am sure that this will raise it's head again with other
transitive dependencies.
Surely will repost any findings after Demo and beyond on this subject .. to
benefit the group.
Thx.
Will
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/spark-1-1-0-w-hadoop-2-4-vs-aws-java-sdk-1-7-2-tp8481p10264.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: spark 1.1.0 (w/ hadoop 2.4) vs aws java sdk 1.7.2
Posted by Sean Owen <so...@cloudera.com>.
Did you use spark.files.userClassPathFirst = true? it's exactly for
this kind of problem.
On Fri, Jan 23, 2015 at 4:42 AM, William-Smith
<wi...@gmail.com> wrote:
> I have had the same issue while using HttpClient from AWS EMR Spark Streaming
> to post to a nodejs server.
>
> I have found ... using
> Classloder.getResource('org/apache/http/client/HttpClient") .... that the
> class
> Is being loaded front the spark-assembly-1.1.0-hadoop2.4.0.jar.
>
> That in itself is not the issue because the version is 4.2.5 .... the same
> version I am using on my local machine with success .... using Hadoop cdh 5.
>
>
>
> The issue is that HttpClient relies on Httpcore .... and there is an old
> commons-httpcore-1.3.jar as well as httpcore-4.5.2 in the spark-assembly
> jar.
>
> It looks like the old one is getting loaded first.
>
> So the fix might be to build the Spark jar myself without the httpcore-1.3
> and replace it on bootstrap.
> I will keep you posted on the outcome.
>
>
>
>
>
> --
> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/spark-1-1-0-w-hadoop-2-4-vs-aws-java-sdk-1-7-2-tp8481p10250.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org