You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "r. r." <ro...@abv.bg> on 2017/10/19 11:34:40 UTC
java.lang.NoSuchMethodError and dependencies problem
Hello
I have a job that runs an Apache Tika pipeline and it fails with "Caused by: java.lang.NoSuchMethodError: org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String;"
Flink includes commons-compress 1.4.1, while Tika needs 1.14.
I also have Apache Avro in the project with commons-compress at 1.8.1, so I force 1.14 with
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-compress</artifactId>
<version>1.14</version>
</dependency>
</dependencies>
</dependencyManagement>
this seems to work as mvn dependency:tree -Ddetail=true only shows 1.14 and after purge, the local maven repo also only contains 1.14
yet, after i deploy the job and it reads an Avro package from kafka and passes it to Tika, it fails with the error above, which leads me to think it somehow uses commons-compress at a version prior to 1.14, because method 'detect' is not present in older versions
I excluded/included it from the fat-jar
<exclude>org.apache.commons:commons-compress</exclude>
still the same problem
thanks for any hints!
Re: java.lang.NoSuchMethodError and dependencies problem
Posted by Piotr Nowojski <pi...@data-artisans.com>.
That’s good to hear :)
Unfortunately at this moment dependencies can pollute class path in both ways (Flink’s can mess with user’s application and also the other way around).
Cheers, Piotrek
> On 20 Oct 2017, at 15:11, r. r. <ro...@abv.bg> wrote:
>
> By Ali Baba's beard and his forty bandits, Piotrek, this worked!
> My understanding was that I have to prevent Flink from loading the older compress.jar and force the newer one.
> One I shade-relocated org.apache.commons.compress for my project the problem went away
>
> Many thanks!
>
>
>
>
>
>
>> -------- Оригинално писмо --------
>
>> От: Piotr Nowojski piotr@data-artisans.com
>
>> Относно: Re: java.lang.NoSuchMethodError and dependencies problem
>
>> До: "r. r." <ro...@abv.bg>
>
>> Изпратено на: 20.10.2017 14:46
>
>
>
>
>> But you said
>
>>
>
>>> this seems to work as mvn dependency:tree -Ddetail=true only shows 1.14
>
>>
>
>> To avoid this error that you describe I think that you have to ensure, that no 1.14 commons-compress comes from your application, because it can conflict with 1.4.1 used by flink cluster.
>
>>
>
>> By shading I meant that you could shade/relocate 1.14 usages in your application, so that they don’t collide with Flink’s 1.4.1.
>
>>
>
>> Piotrek
>
>>
>
>>> On 19 Oct 2017, at 19:58, r. r. <ro...@abv.bg> wrote:
>
>>>
>
>>> Thanks, Piotr
>
>>> but my app code is self-contained in a fat-jar with maven-shade, so why would the class path affect this?
>
>>>
>
>>> by shade commons-compress do you mean :
>
>>> <!-- <exclude>org.apache.commons:commons-compress</exclude> -->
>
>>> it doesn't have effect either
>
>>>
>
>>> as a last resort i may try to rebuild Flink to use 1.14, but don't want to go there yet =/
>
>>>
>
>>>
>
>>> Best regards
>
>>>
>
>>>
>
>>>
>
>>>
>
>>>
>
>>>
>
>>>> -------- Оригинално писмо --------
>
>>>
>
>>>> От: Piotr Nowojski piotr@data-artisans.com
>
>>>
>
>>>> Относно: Re: java.lang.NoSuchMethodError and dependencies problem
>
>>>
>
>>>> До: "r. r." <ro...@abv.bg>
>
>>>
>
>>>> Изпратено на: 19.10.2017 20:04
>
>>>
>
>>>
>
>>>
>
>>>
>
>>>> I’m not 100% sure, so treat my answer with a grain of salt.
>
>>>
>
>>>>
>
>>>
>
>>>> I think when you start the cluster this way, dependencies (some? all?) are being loaded to the class path before loading user’s application. At that point, it doesn’t matter whether you have excluded commons-compress 1.4.1 in yours application pom.xml. I’m not sure if this is solvable in some way, or not.
>
>>>
>
>>>>
>
>>>
>
>>>> Maybe as a walk around, you could shade commons-compress usages in your pom.xml?
>
>>>
>
>>>>
>
>>>
>
>>>> Piotr Nowojski
>
>>>
>
>>>>
>
>>>
>
>>>>> On 19 Oct 2017, at 17:36, r. r. <ro...@abv.bg> wrote:
>
>>>
>
>>>>>
>
>>>
>
>>>>> flink is started with bin/start-local.sh
>
>>>
>
>>>>>
>
>>>
>
>>>>> there is no classpath variable in the environment; flink/lib/flink-dist_2.11-1.3.2.jar contains commons-compress, still it should be overridden by the dependencyManagement directive
>
>>>
>
>>>>>
>
>>>
>
>>>>> here is the stacktrace:
>
>>>
>
>>>>>
>
>>>
>
>>>>> The program finished with the following exception:
>
>>>
>
>>>>>
>
>>>
>
>>>>> org.apache.flink.client.program.ProgramInvocationException: The program execution failed: Job execution failed.
>
>>>
>
>>>>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:478)
>
>>>
>
>>>>> at org.apache.flink.client.program.StandaloneClusterClient.submitJob(StandaloneClusterClient.java:105)
>
>>>
>
>>>>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:442)
>
>>>
>
>>>>> at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:73)
>
>>>
>
>>>>> at com.foot.semantic.flink.PipelineJob.main(PipelineJob.java:73)
>
>>>
>
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>>>
>
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
>>>
>
>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
>>>
>
>>>>> at java.lang.reflect.Method.invoke(Method.java:497)
>
>>>
>
>>>>> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
>
>>>
>
>>>>> at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419)
>
>>>
>
>>>>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:381)
>
>>>
>
>>>>> at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:838)
>
>>>
>
>>>>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:259)
>
>>>
>
>>>>> at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1086)
>
>>>
>
>>>>> at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1133)
>
>>>
>
>>>>> at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1130)
>
>>>
>
>>>>> at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
>
>>>
>
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>
>>>
>
>>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>
>>>
>
>>>>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>
>>>
>
>>>>> at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
>
>>>
>
>>>>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1130)
>
>>>
>
>>>>> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
>
>>>
>
>>>>> at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply$mcV$sp(JobManager.scala:933)
>
>>>
>
>>>>> at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:876)
>
>>>
>
>>>>> at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:876)
>
>>>
>
>>>>> at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
>
>>>
>
>>>>> at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
>
>>>
>
>>>>> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
>
>>>
>
>>>>> at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
>
>>>
>
>>>>> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>
>>>
>
>>>>> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>
>>>
>
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>
>>>
>
>>>>> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
>>>
>
>>>>> Caused by: java.lang.NoSuchMethodError: org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String;
>
>>>
>
>>>>> at org.apache.tika.parser.pkg.ZipContainerDetector.detectArchiveFormat(ZipContainerDetector.java:114)
>
>>>
>
>>>>> at org.apache.tika.parser.pkg.ZipContainerDetector.detect(ZipContainerDetector.java:85)
>
>>>
>
>>>>> at org.apache.tika.detect.CompositeDetector.detect(CompositeDetector.java:77)
>
>>>
>
>>>>> at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:115)
>
>>>
>
>>>>> at com.foot.semantic.gafe.flink.services.TikaService.parse(TikaService.java:83)
>
>>>
>
>>>>> at com.foot.semantic.gafe.flink.services.TikaService.parse(TikaService.java:59)
>
>>>
>
>>>>> at com.foot.semantic.flink.uima.TikaReader.process(TikaReader.java:137)
>
>>>
>
>>>>> at com.foot.semantic.flink.uima.TikaReader.getNext(TikaReader.java:82)
>
>>>
>
>>>>> at org.apache.uima.fit.pipeline.SimplePipeline.runPipeline(SimplePipeline.java:175)
>
>>>
>
>>>>> at com.foot.semantic.flink.PipelineJob$1.map(PipelineJob.java:63)
>
>>>
>
>>>>> at com.foot.semantic.flink.PipelineJob$1.map(PipelineJob.java:37)
>
>>>
>
>>>>> at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41)
>
>>>
>
>>>>> at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:528)
>
>>>
>
>>>>> at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:503)
>
>>>
>
>>>>> at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:483)
>
>>>
>
>>>>> at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:891)
>
>>>
>
>>>>> at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:869)
>
>>>
>
>>>>> at org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:103)
>
>>>
>
>>>>> at org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collectWithTimestamp(StreamSourceContexts.java:110)
>
>>>
>
>>>>> at org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordWithTimestamp(AbstractFetcher.java:269)
>
>>>
>
>>>>> at org.apache.flink.streaming.connectors.kafka.internal.Kafka010Fetcher.emitRecord(Kafka010Fetcher.java:86)
>
>>>
>
>>>>> at org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:152)
>
>>>
>
>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:483)
>
>>>
>
>>>>> at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
>
>>>
>
>>>>> at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55)
>
>>>
>
>>>>> at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:95)
>
>>>
>
>>>>> at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:263)
>
>>>
>
>>>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
>
>>>
>
>>>>> at java.lang.Thread.run(Thread.java:745)
>
>>>
>
>>>>>
>
>>>
>
>>>>>
>
>>>
>
>>>>>
>
>>>
>
>>>>>
>
>>>
>
>>>>>
>
>>>
>
>>>>>
>
>>>
>
>>>>>
>
>>>
>
>>>>>
>
>>>
>
>>>>>
>
>>>
>
>>>>>> -------- Оригинално писмо --------
>
>>>
>
>>>>>
>
>>>
>
>>>>>> От: Piotr Nowojski piotr@data-artisans.com
>
>>>
>
>>>>>
>
>>>
>
>>>>>> Относно: Re: java.lang.NoSuchMethodError and dependencies problem
>
>>>
>
>>>>>
>
>>>
>
>>>>>> До: "r. r." <ro...@abv.bg>
>
>>>
>
>>>>>
>
>>>
>
>>>>>> Изпратено на: 19.10.2017 18:00
>
>>>
>
>>>>>
>
>>>
>
>>>>>
>
>>>
>
>>>>>
>
>>>
>
>>>>>
>
>>>
>
>>>>>> Hi,
>
>>>
>
>>>>>
>
>>>
>
>>>>>>
>
>>>
>
>>>>>
>
>>>
>
>>>>>> What is the full stack trace of the error?
>
>>>
>
>>>>>
>
>>>
>
>>>>>> Are you sure that there is no commons-compresss somewhere in the classpath (like in the lib directory)? How are you running your Flink cluster?
>
>>>
>
>>>>>
>
>>>
>
>>>>>>
>
>>>
>
>>>>>
>
>>>
>
>>>>>> Piotrek
>
>>>
>
>>>>>
>
>>>
>
>>>>>>
>
>>>
>
>>>>>
>
>>>
>
>>>>>>> On 19 Oct 2017, at 13:34, r. r. <ro...@abv.bg> wrote:
>
>>>
>
>>>>>
>
>>>
>
>>>>>>>
>
>>>
>
>>>>>
>
>>>
>
>>>>>>> Hello
>
>>>
>
>>>>>
>
>>>
>
>>>>>>> I have a job that runs an Apache Tika pipeline and it fails with "Caused by: java.lang.NoSuchMethodError: org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String;"
>
>>>
>
>>>>>
>
>>>
>
>>>>>>>
>
>>>
>
>>>>>
>
>>>
>
>>>>>>> Flink includes commons-compress 1.4.1, while Tika needs 1.14.
>
>>>
>
>>>>>
>
>>>
>
>>>>>>> I also have Apache Avro in the project with commons-compress at 1.8.1, so I force 1.14 with
>
>>>
>
>>>>>
>
>>>
>
>>>>>>>
>
>>>
>
>>>>>
>
>>>
>
>>>>>>> <dependencyManagement>
>
>>>
>
>>>>>
>
>>>
>
>>>>>>> <dependencies>
>
>>>
>
>>>>>
>
>>>
>
>>>>>>> <dependency>
>
>>>
>
>>>>>
>
>>>
>
>>>>>>> <groupId>org.apache.commons</groupId>
>
>>>
>
>>>>>
>
>>>
>
>>>>>>> <artifactId>commons-compress</artifactId>
>
>>>
>
>>>>>
>
>>>
>
>>>>>>> <version>1.14</version>
>
>>>
>
>>>>>
>
>>>
>
>>>>>>> </dependency>
>
>>>
>
>>>>>
>
>>>
>
>>>>>>> </dependencies>
>
>>>
>
>>>>>
>
>>>
>
>>>>>>> </dependencyManagement>
>
>>>
>
>>>>>
>
>>>
>
>>>>>>>
>
>>>
>
>>>>>
>
>>>
>
>>>>>>> this seems to work as mvn dependency:tree -Ddetail=true only shows 1.14 and after purge, the local maven repo also only contains 1.14
>
>>>
>
>>>>>
>
>>>
>
>>>>>>>
>
>>>
>
>>>>>
>
>>>
>
>>>>>>> yet, after i deploy the job and it reads an Avro package from kafka and passes it to Tika, it fails with the error above, which leads me to think it somehow uses commons-compress at a version prior to 1.14, because method 'detect' is not present in older versions
>
>>>
>
>>>>>
>
>>>
>
>>>>>>>
>
>>>
>
>>>>>
>
>>>
>
>>>>>>> I excluded/included it from the fat-jar
>
>>>
>
>>>>>
>
>>>
>
>>>>>>> <exclude>org.apache.commons:commons-compress</exclude>
>
>>>
>
>>>>>
>
>>>
>
>>>>>>> still the same problem
>
>>>
>
>>>>>
>
>>>
>
>>>>>>>
>
>>>
>
>>>>>
>
>>>
>
>>>>>>> thanks for any hints!
>
>>>
>
>>>>>
>
>>>
>
>>>>>>>
>
>>>
>
>>>>>
>
>>>
>
>>>>>>>
Re: java.lang.NoSuchMethodError and dependencies problem
Posted by "r. r." <ro...@abv.bg>.
By Ali Baba's beard and his forty bandits, Piotrek, this worked!
My understanding was that I have to prevent Flink from loading the older compress.jar and force the newer one.
One I shade-relocated org.apache.commons.compress for my project the problem went away
Many thanks!
>-------- Оригинално писмо --------
>От: Piotr Nowojski piotr@data-artisans.com
>Относно: Re: java.lang.NoSuchMethodError and dependencies problem
>До: "r. r." <ro...@abv.bg>
>Изпратено на: 20.10.2017 14:46
> But you said
>
> > this seems to work as mvn dependency:tree -Ddetail=true only shows 1.14
>
> To avoid this error that you describe I think that you have to ensure, that no 1.14 commons-compress comes from your application, because it can conflict with 1.4.1 used by flink cluster.
>
> By shading I meant that you could shade/relocate 1.14 usages in your application, so that they don’t collide with Flink’s 1.4.1.
>
> Piotrek
>
> > On 19 Oct 2017, at 19:58, r. r. <ro...@abv.bg> wrote:
> >
> > Thanks, Piotr
> > but my app code is self-contained in a fat-jar with maven-shade, so why would the class path affect this?
> >
> > by shade commons-compress do you mean :
> > <!-- <exclude>org.apache.commons:commons-compress</exclude> -->
> > it doesn't have effect either
> >
> > as a last resort i may try to rebuild Flink to use 1.14, but don't want to go there yet =/
> >
> >
> > Best regards
> >
> >
> >
> >
> >
> >
> >> -------- Оригинално писмо --------
> >
> >> От: Piotr Nowojski piotr@data-artisans.com
> >
> >> Относно: Re: java.lang.NoSuchMethodError and dependencies problem
> >
> >> До: "r. r." <ro...@abv.bg>
> >
> >> Изпратено на: 19.10.2017 20:04
> >
> >
> >
> >
> >> I’m not 100% sure, so treat my answer with a grain of salt.
> >
> >>
> >
> >> I think when you start the cluster this way, dependencies (some? all?) are being loaded to the class path before loading user’s application. At that point, it doesn’t matter whether you have excluded commons-compress 1.4.1 in yours application pom.xml. I’m not sure if this is solvable in some way, or not.
> >
> >>
> >
> >> Maybe as a walk around, you could shade commons-compress usages in your pom.xml?
> >
> >>
> >
> >> Piotr Nowojski
> >
> >>
> >
> >>> On 19 Oct 2017, at 17:36, r. r. <ro...@abv.bg> wrote:
> >
> >>>
> >
> >>> flink is started with bin/start-local.sh
> >
> >>>
> >
> >>> there is no classpath variable in the environment; flink/lib/flink-dist_2.11-1.3.2.jar contains commons-compress, still it should be overridden by the dependencyManagement directive
> >
> >>>
> >
> >>> here is the stacktrace:
> >
> >>>
> >
> >>> The program finished with the following exception:
> >
> >>>
> >
> >>> org.apache.flink.client.program.ProgramInvocationException: The program execution failed: Job execution failed.
> >
> >>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:478)
> >
> >>> at org.apache.flink.client.program.StandaloneClusterClient.submitJob(StandaloneClusterClient.java:105)
> >
> >>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:442)
> >
> >>> at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:73)
> >
> >>> at com.foot.semantic.flink.PipelineJob.main(PipelineJob.java:73)
> >
> >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >
> >>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >
> >>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >
> >>> at java.lang.reflect.Method.invoke(Method.java:497)
> >
> >>> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
> >
> >>> at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419)
> >
> >>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:381)
> >
> >>> at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:838)
> >
> >>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:259)
> >
> >>> at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1086)
> >
> >>> at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1133)
> >
> >>> at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1130)
> >
> >>> at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
> >
> >>> at java.security.AccessController.doPrivileged(Native Method)
> >
> >>> at javax.security.auth.Subject.doAs(Subject.java:422)
> >
> >>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> >
> >>> at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
> >
> >>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1130)
> >
> >>> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> >
> >>> at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply$mcV$sp(JobManager.scala:933)
> >
> >>> at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:876)
> >
> >>> at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:876)
> >
> >>> at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
> >
> >>> at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
> >
> >>> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
> >
> >>> at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
> >
> >>> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> >
> >>> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> >
> >>> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> >
> >>> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >
> >>> Caused by: java.lang.NoSuchMethodError: org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String;
> >
> >>> at org.apache.tika.parser.pkg.ZipContainerDetector.detectArchiveFormat(ZipContainerDetector.java:114)
> >
> >>> at org.apache.tika.parser.pkg.ZipContainerDetector.detect(ZipContainerDetector.java:85)
> >
> >>> at org.apache.tika.detect.CompositeDetector.detect(CompositeDetector.java:77)
> >
> >>> at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:115)
> >
> >>> at com.foot.semantic.gafe.flink.services.TikaService.parse(TikaService.java:83)
> >
> >>> at com.foot.semantic.gafe.flink.services.TikaService.parse(TikaService.java:59)
> >
> >>> at com.foot.semantic.flink.uima.TikaReader.process(TikaReader.java:137)
> >
> >>> at com.foot.semantic.flink.uima.TikaReader.getNext(TikaReader.java:82)
> >
> >>> at org.apache.uima.fit.pipeline.SimplePipeline.runPipeline(SimplePipeline.java:175)
> >
> >>> at com.foot.semantic.flink.PipelineJob$1.map(PipelineJob.java:63)
> >
> >>> at com.foot.semantic.flink.PipelineJob$1.map(PipelineJob.java:37)
> >
> >>> at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41)
> >
> >>> at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:528)
> >
> >>> at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:503)
> >
> >>> at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:483)
> >
> >>> at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:891)
> >
> >>> at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:869)
> >
> >>> at org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:103)
> >
> >>> at org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collectWithTimestamp(StreamSourceContexts.java:110)
> >
> >>> at org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordWithTimestamp(AbstractFetcher.java:269)
> >
> >>> at org.apache.flink.streaming.connectors.kafka.internal.Kafka010Fetcher.emitRecord(Kafka010Fetcher.java:86)
> >
> >>> at org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:152)
> >
> >>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:483)
> >
> >>> at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
> >
> >>> at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55)
> >
> >>> at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:95)
> >
> >>> at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:263)
> >
> >>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
> >
> >>> at java.lang.Thread.run(Thread.java:745)
> >
> >>>
> >
> >>>
> >
> >>>
> >
> >>>
> >
> >>>
> >
> >>>
> >
> >>>
> >
> >>>
> >
> >>>
> >
> >>>> -------- Оригинално писмо --------
> >
> >>>
> >
> >>>> От: Piotr Nowojski piotr@data-artisans.com
> >
> >>>
> >
> >>>> Относно: Re: java.lang.NoSuchMethodError and dependencies problem
> >
> >>>
> >
> >>>> До: "r. r." <ro...@abv.bg>
> >
> >>>
> >
> >>>> Изпратено на: 19.10.2017 18:00
> >
> >>>
> >
> >>>
> >
> >>>
> >
> >>>
> >
> >>>> Hi,
> >
> >>>
> >
> >>>>
> >
> >>>
> >
> >>>> What is the full stack trace of the error?
> >
> >>>
> >
> >>>> Are you sure that there is no commons-compresss somewhere in the classpath (like in the lib directory)? How are you running your Flink cluster?
> >
> >>>
> >
> >>>>
> >
> >>>
> >
> >>>> Piotrek
> >
> >>>
> >
> >>>>
> >
> >>>
> >
> >>>>> On 19 Oct 2017, at 13:34, r. r. <ro...@abv.bg> wrote:
> >
> >>>
> >
> >>>>>
> >
> >>>
> >
> >>>>> Hello
> >
> >>>
> >
> >>>>> I have a job that runs an Apache Tika pipeline and it fails with "Caused by: java.lang.NoSuchMethodError: org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String;"
> >
> >>>
> >
> >>>>>
> >
> >>>
> >
> >>>>> Flink includes commons-compress 1.4.1, while Tika needs 1.14.
> >
> >>>
> >
> >>>>> I also have Apache Avro in the project with commons-compress at 1.8.1, so I force 1.14 with
> >
> >>>
> >
> >>>>>
> >
> >>>
> >
> >>>>> <dependencyManagement>
> >
> >>>
> >
> >>>>> <dependencies>
> >
> >>>
> >
> >>>>> <dependency>
> >
> >>>
> >
> >>>>> <groupId>org.apache.commons</groupId>
> >
> >>>
> >
> >>>>> <artifactId>commons-compress</artifactId>
> >
> >>>
> >
> >>>>> <version>1.14</version>
> >
> >>>
> >
> >>>>> </dependency>
> >
> >>>
> >
> >>>>> </dependencies>
> >
> >>>
> >
> >>>>> </dependencyManagement>
> >
> >>>
> >
> >>>>>
> >
> >>>
> >
> >>>>> this seems to work as mvn dependency:tree -Ddetail=true only shows 1.14 and after purge, the local maven repo also only contains 1.14
> >
> >>>
> >
> >>>>>
> >
> >>>
> >
> >>>>> yet, after i deploy the job and it reads an Avro package from kafka and passes it to Tika, it fails with the error above, which leads me to think it somehow uses commons-compress at a version prior to 1.14, because method 'detect' is not present in older versions
> >
> >>>
> >
> >>>>>
> >
> >>>
> >
> >>>>> I excluded/included it from the fat-jar
> >
> >>>
> >
> >>>>> <exclude>org.apache.commons:commons-compress</exclude>
> >
> >>>
> >
> >>>>> still the same problem
> >
> >>>
> >
> >>>>>
> >
> >>>
> >
> >>>>> thanks for any hints!
> >
> >>>
> >
> >>>>>
> >
> >>>
> >
> >>>>>
Re: java.lang.NoSuchMethodError and dependencies problem
Posted by Piotr Nowojski <pi...@data-artisans.com>.
But you said
> this seems to work as mvn dependency:tree -Ddetail=true only shows 1.14
To avoid this error that you describe I think that you have to ensure, that no 1.14 commons-compress comes from your application, because it can conflict with 1.4.1 used by flink cluster.
By shading I meant that you could shade/relocate 1.14 usages in your application, so that they don’t collide with Flink’s 1.4.1.
Piotrek
> On 19 Oct 2017, at 19:58, r. r. <ro...@abv.bg> wrote:
>
> Thanks, Piotr
> but my app code is self-contained in a fat-jar with maven-shade, so why would the class path affect this?
>
> by shade commons-compress do you mean :
> <!-- <exclude>org.apache.commons:commons-compress</exclude> -->
> it doesn't have effect either
>
> as a last resort i may try to rebuild Flink to use 1.14, but don't want to go there yet =/
>
>
> Best regards
>
>
>
>
>
>
>> -------- Оригинално писмо --------
>
>> От: Piotr Nowojski piotr@data-artisans.com
>
>> Относно: Re: java.lang.NoSuchMethodError and dependencies problem
>
>> До: "r. r." <ro...@abv.bg>
>
>> Изпратено на: 19.10.2017 20:04
>
>
>
>
>> I’m not 100% sure, so treat my answer with a grain of salt.
>
>>
>
>> I think when you start the cluster this way, dependencies (some? all?) are being loaded to the class path before loading user’s application. At that point, it doesn’t matter whether you have excluded commons-compress 1.4.1 in yours application pom.xml. I’m not sure if this is solvable in some way, or not.
>
>>
>
>> Maybe as a walk around, you could shade commons-compress usages in your pom.xml?
>
>>
>
>> Piotr Nowojski
>
>>
>
>>> On 19 Oct 2017, at 17:36, r. r. <ro...@abv.bg> wrote:
>
>>>
>
>>> flink is started with bin/start-local.sh
>
>>>
>
>>> there is no classpath variable in the environment; flink/lib/flink-dist_2.11-1.3.2.jar contains commons-compress, still it should be overridden by the dependencyManagement directive
>
>>>
>
>>> here is the stacktrace:
>
>>>
>
>>> The program finished with the following exception:
>
>>>
>
>>> org.apache.flink.client.program.ProgramInvocationException: The program execution failed: Job execution failed.
>
>>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:478)
>
>>> at org.apache.flink.client.program.StandaloneClusterClient.submitJob(StandaloneClusterClient.java:105)
>
>>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:442)
>
>>> at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:73)
>
>>> at com.foot.semantic.flink.PipelineJob.main(PipelineJob.java:73)
>
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
>>> at java.lang.reflect.Method.invoke(Method.java:497)
>
>>> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
>
>>> at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419)
>
>>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:381)
>
>>> at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:838)
>
>>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:259)
>
>>> at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1086)
>
>>> at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1133)
>
>>> at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1130)
>
>>> at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
>
>>> at java.security.AccessController.doPrivileged(Native Method)
>
>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>
>>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>
>>> at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
>
>>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1130)
>
>>> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
>
>>> at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply$mcV$sp(JobManager.scala:933)
>
>>> at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:876)
>
>>> at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:876)
>
>>> at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
>
>>> at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
>
>>> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
>
>>> at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
>
>>> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>
>>> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>
>>> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>
>>> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
>>> Caused by: java.lang.NoSuchMethodError: org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String;
>
>>> at org.apache.tika.parser.pkg.ZipContainerDetector.detectArchiveFormat(ZipContainerDetector.java:114)
>
>>> at org.apache.tika.parser.pkg.ZipContainerDetector.detect(ZipContainerDetector.java:85)
>
>>> at org.apache.tika.detect.CompositeDetector.detect(CompositeDetector.java:77)
>
>>> at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:115)
>
>>> at com.foot.semantic.gafe.flink.services.TikaService.parse(TikaService.java:83)
>
>>> at com.foot.semantic.gafe.flink.services.TikaService.parse(TikaService.java:59)
>
>>> at com.foot.semantic.flink.uima.TikaReader.process(TikaReader.java:137)
>
>>> at com.foot.semantic.flink.uima.TikaReader.getNext(TikaReader.java:82)
>
>>> at org.apache.uima.fit.pipeline.SimplePipeline.runPipeline(SimplePipeline.java:175)
>
>>> at com.foot.semantic.flink.PipelineJob$1.map(PipelineJob.java:63)
>
>>> at com.foot.semantic.flink.PipelineJob$1.map(PipelineJob.java:37)
>
>>> at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41)
>
>>> at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:528)
>
>>> at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:503)
>
>>> at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:483)
>
>>> at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:891)
>
>>> at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:869)
>
>>> at org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:103)
>
>>> at org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collectWithTimestamp(StreamSourceContexts.java:110)
>
>>> at org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordWithTimestamp(AbstractFetcher.java:269)
>
>>> at org.apache.flink.streaming.connectors.kafka.internal.Kafka010Fetcher.emitRecord(Kafka010Fetcher.java:86)
>
>>> at org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:152)
>
>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:483)
>
>>> at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
>
>>> at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55)
>
>>> at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:95)
>
>>> at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:263)
>
>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
>
>>> at java.lang.Thread.run(Thread.java:745)
>
>>>
>
>>>
>
>>>
>
>>>
>
>>>
>
>>>
>
>>>
>
>>>
>
>>>
>
>>>> -------- Оригинално писмо --------
>
>>>
>
>>>> От: Piotr Nowojski piotr@data-artisans.com
>
>>>
>
>>>> Относно: Re: java.lang.NoSuchMethodError and dependencies problem
>
>>>
>
>>>> До: "r. r." <ro...@abv.bg>
>
>>>
>
>>>> Изпратено на: 19.10.2017 18:00
>
>>>
>
>>>
>
>>>
>
>>>
>
>>>> Hi,
>
>>>
>
>>>>
>
>>>
>
>>>> What is the full stack trace of the error?
>
>>>
>
>>>> Are you sure that there is no commons-compresss somewhere in the classpath (like in the lib directory)? How are you running your Flink cluster?
>
>>>
>
>>>>
>
>>>
>
>>>> Piotrek
>
>>>
>
>>>>
>
>>>
>
>>>>> On 19 Oct 2017, at 13:34, r. r. <ro...@abv.bg> wrote:
>
>>>
>
>>>>>
>
>>>
>
>>>>> Hello
>
>>>
>
>>>>> I have a job that runs an Apache Tika pipeline and it fails with "Caused by: java.lang.NoSuchMethodError: org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String;"
>
>>>
>
>>>>>
>
>>>
>
>>>>> Flink includes commons-compress 1.4.1, while Tika needs 1.14.
>
>>>
>
>>>>> I also have Apache Avro in the project with commons-compress at 1.8.1, so I force 1.14 with
>
>>>
>
>>>>>
>
>>>
>
>>>>> <dependencyManagement>
>
>>>
>
>>>>> <dependencies>
>
>>>
>
>>>>> <dependency>
>
>>>
>
>>>>> <groupId>org.apache.commons</groupId>
>
>>>
>
>>>>> <artifactId>commons-compress</artifactId>
>
>>>
>
>>>>> <version>1.14</version>
>
>>>
>
>>>>> </dependency>
>
>>>
>
>>>>> </dependencies>
>
>>>
>
>>>>> </dependencyManagement>
>
>>>
>
>>>>>
>
>>>
>
>>>>> this seems to work as mvn dependency:tree -Ddetail=true only shows 1.14 and after purge, the local maven repo also only contains 1.14
>
>>>
>
>>>>>
>
>>>
>
>>>>> yet, after i deploy the job and it reads an Avro package from kafka and passes it to Tika, it fails with the error above, which leads me to think it somehow uses commons-compress at a version prior to 1.14, because method 'detect' is not present in older versions
>
>>>
>
>>>>>
>
>>>
>
>>>>> I excluded/included it from the fat-jar
>
>>>
>
>>>>> <exclude>org.apache.commons:commons-compress</exclude>
>
>>>
>
>>>>> still the same problem
>
>>>
>
>>>>>
>
>>>
>
>>>>> thanks for any hints!
>
>>>
>
>>>>>
>
>>>
>
>>>>>
Re: java.lang.NoSuchMethodError and dependencies problem
Posted by "r. r." <ro...@abv.bg>.
Thanks, Piotr
but my app code is self-contained in a fat-jar with maven-shade, so why would the class path affect this?
by shade commons-compress do you mean :
<!-- <exclude>org.apache.commons:commons-compress</exclude> -->
it doesn't have effect either
as a last resort i may try to rebuild Flink to use 1.14, but don't want to go there yet =/
Best regards
>-------- Оригинално писмо --------
>От: Piotr Nowojski piotr@data-artisans.com
>Относно: Re: java.lang.NoSuchMethodError and dependencies problem
>До: "r. r." <ro...@abv.bg>
>Изпратено на: 19.10.2017 20:04
> I’m not 100% sure, so treat my answer with a grain of salt.
>
> I think when you start the cluster this way, dependencies (some? all?) are being loaded to the class path before loading user’s application. At that point, it doesn’t matter whether you have excluded commons-compress 1.4.1 in yours application pom.xml. I’m not sure if this is solvable in some way, or not.
>
> Maybe as a walk around, you could shade commons-compress usages in your pom.xml?
>
> Piotr Nowojski
>
> > On 19 Oct 2017, at 17:36, r. r. <ro...@abv.bg> wrote:
> >
> > flink is started with bin/start-local.sh
> >
> > there is no classpath variable in the environment; flink/lib/flink-dist_2.11-1.3.2.jar contains commons-compress, still it should be overridden by the dependencyManagement directive
> >
> > here is the stacktrace:
> >
> > The program finished with the following exception:
> >
> > org.apache.flink.client.program.ProgramInvocationException: The program execution failed: Job execution failed.
> > at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:478)
> > at org.apache.flink.client.program.StandaloneClusterClient.submitJob(StandaloneClusterClient.java:105)
> > at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:442)
> > at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:73)
> > at com.foot.semantic.flink.PipelineJob.main(PipelineJob.java:73)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:497)
> > at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
> > at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419)
> > at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:381)
> > at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:838)
> > at org.apache.flink.client.CliFrontend.run(CliFrontend.java:259)
> > at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1086)
> > at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1133)
> > at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1130)
> > at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:422)
> > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> > at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
> > at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1130)
> > Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> > at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply$mcV$sp(JobManager.scala:933)
> > at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:876)
> > at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:876)
> > at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
> > at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
> > at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
> > at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
> > at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> > at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> > at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> > Caused by: java.lang.NoSuchMethodError: org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String;
> > at org.apache.tika.parser.pkg.ZipContainerDetector.detectArchiveFormat(ZipContainerDetector.java:114)
> > at org.apache.tika.parser.pkg.ZipContainerDetector.detect(ZipContainerDetector.java:85)
> > at org.apache.tika.detect.CompositeDetector.detect(CompositeDetector.java:77)
> > at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:115)
> > at com.foot.semantic.gafe.flink.services.TikaService.parse(TikaService.java:83)
> > at com.foot.semantic.gafe.flink.services.TikaService.parse(TikaService.java:59)
> > at com.foot.semantic.flink.uima.TikaReader.process(TikaReader.java:137)
> > at com.foot.semantic.flink.uima.TikaReader.getNext(TikaReader.java:82)
> > at org.apache.uima.fit.pipeline.SimplePipeline.runPipeline(SimplePipeline.java:175)
> > at com.foot.semantic.flink.PipelineJob$1.map(PipelineJob.java:63)
> > at com.foot.semantic.flink.PipelineJob$1.map(PipelineJob.java:37)
> > at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41)
> > at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:528)
> > at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:503)
> > at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:483)
> > at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:891)
> > at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:869)
> > at org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:103)
> > at org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collectWithTimestamp(StreamSourceContexts.java:110)
> > at org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordWithTimestamp(AbstractFetcher.java:269)
> > at org.apache.flink.streaming.connectors.kafka.internal.Kafka010Fetcher.emitRecord(Kafka010Fetcher.java:86)
> > at org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:152)
> > at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:483)
> > at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
> > at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55)
> > at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:95)
> > at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:263)
> > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
> > at java.lang.Thread.run(Thread.java:745)
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >> -------- Оригинално писмо --------
> >
> >> От: Piotr Nowojski piotr@data-artisans.com
> >
> >> Относно: Re: java.lang.NoSuchMethodError and dependencies problem
> >
> >> До: "r. r." <ro...@abv.bg>
> >
> >> Изпратено на: 19.10.2017 18:00
> >
> >
> >
> >
> >> Hi,
> >
> >>
> >
> >> What is the full stack trace of the error?
> >
> >> Are you sure that there is no commons-compresss somewhere in the classpath (like in the lib directory)? How are you running your Flink cluster?
> >
> >>
> >
> >> Piotrek
> >
> >>
> >
> >>> On 19 Oct 2017, at 13:34, r. r. <ro...@abv.bg> wrote:
> >
> >>>
> >
> >>> Hello
> >
> >>> I have a job that runs an Apache Tika pipeline and it fails with "Caused by: java.lang.NoSuchMethodError: org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String;"
> >
> >>>
> >
> >>> Flink includes commons-compress 1.4.1, while Tika needs 1.14.
> >
> >>> I also have Apache Avro in the project with commons-compress at 1.8.1, so I force 1.14 with
> >
> >>>
> >
> >>> <dependencyManagement>
> >
> >>> <dependencies>
> >
> >>> <dependency>
> >
> >>> <groupId>org.apache.commons</groupId>
> >
> >>> <artifactId>commons-compress</artifactId>
> >
> >>> <version>1.14</version>
> >
> >>> </dependency>
> >
> >>> </dependencies>
> >
> >>> </dependencyManagement>
> >
> >>>
> >
> >>> this seems to work as mvn dependency:tree -Ddetail=true only shows 1.14 and after purge, the local maven repo also only contains 1.14
> >
> >>>
> >
> >>> yet, after i deploy the job and it reads an Avro package from kafka and passes it to Tika, it fails with the error above, which leads me to think it somehow uses commons-compress at a version prior to 1.14, because method 'detect' is not present in older versions
> >
> >>>
> >
> >>> I excluded/included it from the fat-jar
> >
> >>> <exclude>org.apache.commons:commons-compress</exclude>
> >
> >>> still the same problem
> >
> >>>
> >
> >>> thanks for any hints!
> >
> >>>
> >
> >>>
Re: java.lang.NoSuchMethodError and dependencies problem
Posted by Piotr Nowojski <pi...@data-artisans.com>.
I’m not 100% sure, so treat my answer with a grain of salt.
I think when you start the cluster this way, dependencies (some? all?) are being loaded to the class path before loading user’s application. At that point, it doesn’t matter whether you have excluded commons-compress 1.4.1 in yours application pom.xml. I’m not sure if this is solvable in some way, or not.
Maybe as a walk around, you could shade commons-compress usages in your pom.xml?
Piotr Nowojski
> On 19 Oct 2017, at 17:36, r. r. <ro...@abv.bg> wrote:
>
> flink is started with bin/start-local.sh
>
> there is no classpath variable in the environment; flink/lib/flink-dist_2.11-1.3.2.jar contains commons-compress, still it should be overridden by the dependencyManagement directive
>
> here is the stacktrace:
>
> The program finished with the following exception:
>
> org.apache.flink.client.program.ProgramInvocationException: The program execution failed: Job execution failed.
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:478)
> at org.apache.flink.client.program.StandaloneClusterClient.submitJob(StandaloneClusterClient.java:105)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:442)
> at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:73)
> at com.foot.semantic.flink.PipelineJob.main(PipelineJob.java:73)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
> at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:381)
> at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:838)
> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:259)
> at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1086)
> at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1133)
> at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1130)
> at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1130)
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply$mcV$sp(JobManager.scala:933)
> at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:876)
> at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:876)
> at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
> at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
> at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: java.lang.NoSuchMethodError: org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String;
> at org.apache.tika.parser.pkg.ZipContainerDetector.detectArchiveFormat(ZipContainerDetector.java:114)
> at org.apache.tika.parser.pkg.ZipContainerDetector.detect(ZipContainerDetector.java:85)
> at org.apache.tika.detect.CompositeDetector.detect(CompositeDetector.java:77)
> at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:115)
> at com.foot.semantic.gafe.flink.services.TikaService.parse(TikaService.java:83)
> at com.foot.semantic.gafe.flink.services.TikaService.parse(TikaService.java:59)
> at com.foot.semantic.flink.uima.TikaReader.process(TikaReader.java:137)
> at com.foot.semantic.flink.uima.TikaReader.getNext(TikaReader.java:82)
> at org.apache.uima.fit.pipeline.SimplePipeline.runPipeline(SimplePipeline.java:175)
> at com.foot.semantic.flink.PipelineJob$1.map(PipelineJob.java:63)
> at com.foot.semantic.flink.PipelineJob$1.map(PipelineJob.java:37)
> at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41)
> at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:528)
> at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:503)
> at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:483)
> at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:891)
> at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:869)
> at org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:103)
> at org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collectWithTimestamp(StreamSourceContexts.java:110)
> at org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordWithTimestamp(AbstractFetcher.java:269)
> at org.apache.flink.streaming.connectors.kafka.internal.Kafka010Fetcher.emitRecord(Kafka010Fetcher.java:86)
> at org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:152)
> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:483)
> at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
> at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55)
> at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:95)
> at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:263)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
> at java.lang.Thread.run(Thread.java:745)
>
>
>
>
>
>
>
>
>
>> -------- Оригинално писмо --------
>
>> От: Piotr Nowojski piotr@data-artisans.com
>
>> Относно: Re: java.lang.NoSuchMethodError and dependencies problem
>
>> До: "r. r." <ro...@abv.bg>
>
>> Изпратено на: 19.10.2017 18:00
>
>
>
>
>> Hi,
>
>>
>
>> What is the full stack trace of the error?
>
>> Are you sure that there is no commons-compresss somewhere in the classpath (like in the lib directory)? How are you running your Flink cluster?
>
>>
>
>> Piotrek
>
>>
>
>>> On 19 Oct 2017, at 13:34, r. r. <ro...@abv.bg> wrote:
>
>>>
>
>>> Hello
>
>>> I have a job that runs an Apache Tika pipeline and it fails with "Caused by: java.lang.NoSuchMethodError: org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String;"
>
>>>
>
>>> Flink includes commons-compress 1.4.1, while Tika needs 1.14.
>
>>> I also have Apache Avro in the project with commons-compress at 1.8.1, so I force 1.14 with
>
>>>
>
>>> <dependencyManagement>
>
>>> <dependencies>
>
>>> <dependency>
>
>>> <groupId>org.apache.commons</groupId>
>
>>> <artifactId>commons-compress</artifactId>
>
>>> <version>1.14</version>
>
>>> </dependency>
>
>>> </dependencies>
>
>>> </dependencyManagement>
>
>>>
>
>>> this seems to work as mvn dependency:tree -Ddetail=true only shows 1.14 and after purge, the local maven repo also only contains 1.14
>
>>>
>
>>> yet, after i deploy the job and it reads an Avro package from kafka and passes it to Tika, it fails with the error above, which leads me to think it somehow uses commons-compress at a version prior to 1.14, because method 'detect' is not present in older versions
>
>>>
>
>>> I excluded/included it from the fat-jar
>
>>> <exclude>org.apache.commons:commons-compress</exclude>
>
>>> still the same problem
>
>>>
>
>>> thanks for any hints!
>
>>>
>
>>>
Re: java.lang.NoSuchMethodError and dependencies problem
Posted by "r. r." <ro...@abv.bg>.
flink is started with bin/start-local.sh
there is no classpath variable in the environment; flink/lib/flink-dist_2.11-1.3.2.jar contains commons-compress, still it should be overridden by the dependencyManagement directive
here is the stacktrace:
The program finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: The program execution failed: Job execution failed.
at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:478)
at org.apache.flink.client.program.StandaloneClusterClient.submitJob(StandaloneClusterClient.java:105)
at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:442)
at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:73)
at com.foot.semantic.flink.PipelineJob.main(PipelineJob.java:73)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419)
at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:381)
at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:838)
at org.apache.flink.client.CliFrontend.run(CliFrontend.java:259)
at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1086)
at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1133)
at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1130)
at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1130)
Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply$mcV$sp(JobManager.scala:933)
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:876)
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:876)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.NoSuchMethodError: org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String;
at org.apache.tika.parser.pkg.ZipContainerDetector.detectArchiveFormat(ZipContainerDetector.java:114)
at org.apache.tika.parser.pkg.ZipContainerDetector.detect(ZipContainerDetector.java:85)
at org.apache.tika.detect.CompositeDetector.detect(CompositeDetector.java:77)
at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:115)
at com.foot.semantic.gafe.flink.services.TikaService.parse(TikaService.java:83)
at com.foot.semantic.gafe.flink.services.TikaService.parse(TikaService.java:59)
at com.foot.semantic.flink.uima.TikaReader.process(TikaReader.java:137)
at com.foot.semantic.flink.uima.TikaReader.getNext(TikaReader.java:82)
at org.apache.uima.fit.pipeline.SimplePipeline.runPipeline(SimplePipeline.java:175)
at com.foot.semantic.flink.PipelineJob$1.map(PipelineJob.java:63)
at com.foot.semantic.flink.PipelineJob$1.map(PipelineJob.java:37)
at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41)
at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:528)
at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:503)
at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:483)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:891)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:869)
at org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:103)
at org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collectWithTimestamp(StreamSourceContexts.java:110)
at org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordWithTimestamp(AbstractFetcher.java:269)
at org.apache.flink.streaming.connectors.kafka.internal.Kafka010Fetcher.emitRecord(Kafka010Fetcher.java:86)
at org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:152)
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:483)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55)
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:95)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:263)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:745)
>-------- Оригинално писмо --------
>От: Piotr Nowojski piotr@data-artisans.com
>Относно: Re: java.lang.NoSuchMethodError and dependencies problem
>До: "r. r." <ro...@abv.bg>
>Изпратено на: 19.10.2017 18:00
> Hi,
>
> What is the full stack trace of the error?
> Are you sure that there is no commons-compresss somewhere in the classpath (like in the lib directory)? How are you running your Flink cluster?
>
> Piotrek
>
> > On 19 Oct 2017, at 13:34, r. r. <ro...@abv.bg> wrote:
> >
> > Hello
> > I have a job that runs an Apache Tika pipeline and it fails with "Caused by: java.lang.NoSuchMethodError: org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String;"
> >
> > Flink includes commons-compress 1.4.1, while Tika needs 1.14.
> > I also have Apache Avro in the project with commons-compress at 1.8.1, so I force 1.14 with
> >
> > <dependencyManagement>
> > <dependencies>
> > <dependency>
> > <groupId>org.apache.commons</groupId>
> > <artifactId>commons-compress</artifactId>
> > <version>1.14</version>
> > </dependency>
> > </dependencies>
> > </dependencyManagement>
> >
> > this seems to work as mvn dependency:tree -Ddetail=true only shows 1.14 and after purge, the local maven repo also only contains 1.14
> >
> > yet, after i deploy the job and it reads an Avro package from kafka and passes it to Tika, it fails with the error above, which leads me to think it somehow uses commons-compress at a version prior to 1.14, because method 'detect' is not present in older versions
> >
> > I excluded/included it from the fat-jar
> > <exclude>org.apache.commons:commons-compress</exclude>
> > still the same problem
> >
> > thanks for any hints!
> >
> >
Re: java.lang.NoSuchMethodError and dependencies problem
Posted by Piotr Nowojski <pi...@data-artisans.com>.
Hi,
What is the full stack trace of the error?
Are you sure that there is no commons-compresss somewhere in the classpath (like in the lib directory)? How are you running your Flink cluster?
Piotrek
> On 19 Oct 2017, at 13:34, r. r. <ro...@abv.bg> wrote:
>
> Hello
> I have a job that runs an Apache Tika pipeline and it fails with "Caused by: java.lang.NoSuchMethodError: org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String;"
>
> Flink includes commons-compress 1.4.1, while Tika needs 1.14.
> I also have Apache Avro in the project with commons-compress at 1.8.1, so I force 1.14 with
>
> <dependencyManagement>
> <dependencies>
> <dependency>
> <groupId>org.apache.commons</groupId>
> <artifactId>commons-compress</artifactId>
> <version>1.14</version>
> </dependency>
> </dependencies>
> </dependencyManagement>
>
> this seems to work as mvn dependency:tree -Ddetail=true only shows 1.14 and after purge, the local maven repo also only contains 1.14
>
> yet, after i deploy the job and it reads an Avro package from kafka and passes it to Tika, it fails with the error above, which leads me to think it somehow uses commons-compress at a version prior to 1.14, because method 'detect' is not present in older versions
>
> I excluded/included it from the fat-jar
> <exclude>org.apache.commons:commons-compress</exclude>
> still the same problem
>
> thanks for any hints!
>
>