spark-submit script and spark.files.userClassPathFirst

[_soumya_ <so...@gmail.com>]

Hi,
 I'm trying to get rid of an error (NoSuchMethodError) while using Amazon's
s3 client on Spark. I'm using the Spark Submit script to run my code.
Reading about my options and other threads, it seemed the most logical way
would be make sure my jar is loaded first. Spark submit on debug shows the
same: 

spark.files.userClassPathFirst -> true

However, I can't seem to get rid of the error at runtime. The issue I
believe is happening because Spark has the older httpclient library in it's
classpath. I'm using:

        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.3</version>
        </dependency>

Any clues what might be happening?

Stack trace below:
============

Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.http.impl.conn.DefaultClientConnectionOperator.<init>(Lorg/apache/http/conn/scheme/SchemeRegistry;Lorg/apache/http/conn/DnsResolver;)V
	at
org.apache.http.impl.conn.PoolingClientConnectionManager.createConnectionOperator(PoolingClientConnectionManager.java:138)
	at
org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:112)
	at
org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:101)
	at
org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:87)
	at
org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:95)
	at
com.amazonaws.http.ConnectionManagerFactory.createPoolingClientConnManager(ConnectionManagerFactory.java:26)
	at
com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:97)
	at com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:158)
	at
com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:119)
	at
com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:103)
	at com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:357)
	at com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:339)
	at com.evocalize.rickshaw.commons.s3.S3Util.getS3Client(S3Util.java:35)
	at
com.evocalize.rickshaw.commons.s3.S3Util.putFileFromLocalFS(S3Util.java:40)
	at
com.evocalize.rickshaw.spark.actions.PackageFilesFunction.call(PackageFilesFunction.java:48)
	at
com.evocalize.rickshaw.spark.applications.GenerateSEOContent.main(GenerateSEOContent.java:76)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:292)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-submit-script-and-spark-files-userClassPathFirst-tp8604.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.