You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by Cody Koeninger <co...@koeninger.org> on 2014/11/10 23:05:30 UTC

Http client dependency conflict when using AWS

I'm wondering why

https://issues.apache.org/jira/browse/SPARK-3638

only updated the version of http client for the kinesis-asl profile and
left the base dependencies unchanged.

Spark built without that profile still has the same

java.lang.NoSuchMethodError:
org.apache.http.impl.conn.DefaultClientConnectionOperator.<init>(Lorg/apache/http/conn/scheme/SchemeRegistry;Lorg/apache/http/conn/DnsResolver;)V


when using aws components in general, not just kinesis (e.g.
AmazonCloudWatchAsyncClient)


Re-Reading the "Dependency Hell in Spark applications" thread from
september didn't shed any light on the subject.  As noted in that thread,
userClassPathFirst doesn't help.


Is there a reason not to depend on an updated version of httpclient for all
spark builds, as opposed to just kinesis-asl?