You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jaka Jančar <ja...@kubje.org> on 2014/03/19 16:30:56 UTC

Transitive dependency incompatibility

Hi,

I'm getting the following error:

java.lang.NoSuchMethodError: org.apache.http.impl.conn.DefaultClientConnectionOperator.<init>(Lorg/apache/http/conn/scheme/SchemeRegistry;Lorg/apache/http/conn/DnsResolver;)V
at org.apache.http.impl.conn.PoolingClientConnectionManager.createConnectionOperator(PoolingClientConnectionManager.java:140)
at org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:114)
at org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:99)
at org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:85)
at org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:93)
at com.amazonaws.http.ConnectionManagerFactory.createPoolingClientConnManager(ConnectionManagerFactory.java:26)
at com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:96)
at com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:155)
at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:119)
at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:103)
at com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:334)
at com.celtra.analyzer.TrackingLogRDD.createClient(TrackingLogRDD.scala:131)
at com.celtra.analyzer.TrackingLogRDD.compute(TrackingLogRDD.scala:117)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:237)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:226)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:107)
at org.apache.spark.scheduler.Task.run(Task.scala:53)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:215)
at org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:50)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:182)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

My app uses the AWS SDK, which requires org.apache.httpcomponents:httpclient:4.2.

I believe the error is caused by the fact that an older version of the package is already present on the classpath:

Spark -> Akka -> sjson -> org.apache.httpcomponents:httpclient:4.1
Spark -> jets3t -> commons-httpclient:commons-httpclient:3.1


What are my options if I need to use a newer version of the library in my app?

Thanks,
Jaka

Re: Transitive dependency incompatibility

Posted by Jaka Jančar <ja...@kubje.org>.
Just found this issue and wanted to link it here, in case somebody finds this thread later:  

https://spark-project.atlassian.net/browse/SPARK-939



On Thursday, March 20, 2014 at 11:14 AM, Matei Zaharia wrote:

> Hi Jaka,
>  
> I’d recommend rebuilding Spark with a new version of the HTTPClient dependency. In the future we want to add a “put the user’s classpath first” option to let users overwrite dependencies.
>  
> Matei
>  
> On Mar 20, 2014, at 8:42 AM, Jaka Jančar <jaka@kubje.org (mailto:jaka@kubje.org)> wrote:
> > Could the executor use isolated classloaders, in order not to pullute the environment with it's own stuff?
> >  
> >  
> >  
> >  
> >  
> > On Wednesday, March 19, 2014 at 8:30 AM, Jaka Jančar wrote:
> >  
> > > Hi,
> > >  
> > > I'm getting the following error:
> > >  
> > > java.lang.NoSuchMethodError: org.apache.http.impl.conn.DefaultClientConnectionOperator.<init>(Lorg/apache/http/conn/scheme/SchemeRegistry;Lorg/apache/http/conn/DnsResolver;)V
> > > at org.apache.http.impl.conn.PoolingClientConnectionManager.createConnectionOperator(PoolingClientConnectionManager.java:140)
> > > at org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:114)
> > > at org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:99)
> > > at org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:85)
> > > at org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:93)
> > > at com.amazonaws.http.ConnectionManagerFactory.createPoolingClientConnManager(ConnectionManagerFactory.java:26)
> > > at com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:96)
> > > at com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:155)
> > > at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:119)
> > > at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:103)
> > > at com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:334)
> > > at com.celtra.analyzer.TrackingLogRDD.createClient(TrackingLogRDD.scala:131)
> > > at com.celtra.analyzer.TrackingLogRDD.compute(TrackingLogRDD.scala:117)
> > > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:237)
> > > at org.apache.spark.rdd.RDD.iterator(RDD.scala:226)
> > > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:107)
> > > at org.apache.spark.scheduler.Task.run(Task.scala:53)
> > > at org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:215)
> > > at org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:50)
> > > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:182)
> > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > at java.lang.Thread.run(Thread.java:744)
> > >  
> > > My app uses the AWS SDK, which requires org.apache.httpcomponents:httpclient:4.2.
> > >  
> > > I believe the error is caused by the fact that an older version of the package is already present on the classpath:
> > >  
> > > Spark -> Akka -> sjson -> org.apache.httpcomponents:httpclient:4.1
> > > Spark -> jets3t -> commons-httpclient:commons-httpclient:3.1
> > >  
> > >  
> > > What are my options if I need to use a newer version of the library in my app?
> > >  
> > > Thanks,
> > > Jaka
> > >  
> > >  
> > >  
> > >  
> > >  
> >  
>

Re: Transitive dependency incompatibility

Posted by Matei Zaharia <ma...@gmail.com>.
Hi Jaka,

I’d recommend rebuilding Spark with a new version of the HTTPClient dependency. In the future we want to add a “put the user’s classpath first” option to let users overwrite dependencies.

Matei

On Mar 20, 2014, at 8:42 AM, Jaka Jančar <ja...@kubje.org> wrote:

> Could the executor use isolated classloaders, in order not to pullute the environment with it's own stuff?
> 
> 
> 
> 
> On Wednesday, March 19, 2014 at 8:30 AM, Jaka Jančar wrote:
> 
>> Hi,
>> 
>> I'm getting the following error:
>> 
>> java.lang.NoSuchMethodError: org.apache.http.impl.conn.DefaultClientConnectionOperator.<init>(Lorg/apache/http/conn/scheme/SchemeRegistry;Lorg/apache/http/conn/DnsResolver;)V
>> at org.apache.http.impl.conn.PoolingClientConnectionManager.createConnectionOperator(PoolingClientConnectionManager.java:140)
>> at org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:114)
>> at org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:99)
>> at org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:85)
>> at org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:93)
>> at com.amazonaws.http.ConnectionManagerFactory.createPoolingClientConnManager(ConnectionManagerFactory.java:26)
>> at com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:96)
>> at com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:155)
>> at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:119)
>> at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:103)
>> at com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:334)
>> at com.celtra.analyzer.TrackingLogRDD.createClient(TrackingLogRDD.scala:131)
>> at com.celtra.analyzer.TrackingLogRDD.compute(TrackingLogRDD.scala:117)
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:237)
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:226)
>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:107)
>> at org.apache.spark.scheduler.Task.run(Task.scala:53)
>> at org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:215)
>> at org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:50)
>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:182)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>> 
>> My app uses the AWS SDK, which requires org.apache.httpcomponents:httpclient:4.2.
>> 
>> I believe the error is caused by the fact that an older version of the package is already present on the classpath:
>> 
>> Spark -> Akka -> sjson -> org.apache.httpcomponents:httpclient:4.1
>> Spark -> jets3t -> commons-httpclient:commons-httpclient:3.1
>> 
>> 
>> What are my options if I need to use a newer version of the library in my app?
>> 
>> Thanks,
>> Jaka
>> 
>> 
>> 
>> 
>

Re: Transitive dependency incompatibility

Posted by Jaka Jančar <ja...@kubje.org>.
Could the executor use isolated classloaders, in order not to pullute the environment with it's own stuff?





On Wednesday, March 19, 2014 at 8:30 AM, Jaka Jančar wrote:

> Hi,
>  
> I'm getting the following error:
>  
> java.lang.NoSuchMethodError: org.apache.http.impl.conn.DefaultClientConnectionOperator.<init>(Lorg/apache/http/conn/scheme/SchemeRegistry;Lorg/apache/http/conn/DnsResolver;)V
> at org.apache.http.impl.conn.PoolingClientConnectionManager.createConnectionOperator(PoolingClientConnectionManager.java:140)
> at org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:114)
> at org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:99)
> at org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:85)
> at org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:93)
> at com.amazonaws.http.ConnectionManagerFactory.createPoolingClientConnManager(ConnectionManagerFactory.java:26)
> at com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:96)
> at com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:155)
> at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:119)
> at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:103)
> at com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:334)
> at com.celtra.analyzer.TrackingLogRDD.createClient(TrackingLogRDD.scala:131)
> at com.celtra.analyzer.TrackingLogRDD.compute(TrackingLogRDD.scala:117)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:237)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:226)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:107)
> at org.apache.spark.scheduler.Task.run(Task.scala:53)
> at org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:215)
> at org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:50)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:182)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
>  
> My app uses the AWS SDK, which requires org.apache.httpcomponents:httpclient:4.2.
>  
> I believe the error is caused by the fact that an older version of the package is already present on the classpath:
>  
> Spark -> Akka -> sjson -> org.apache.httpcomponents:httpclient:4.1
> Spark -> jets3t -> commons-httpclient:commons-httpclient:3.1
>  
>  
> What are my options if I need to use a newer version of the library in my app?
>  
> Thanks,
> Jaka
>  
>  
>  
>  
>