[jira] [Comment Edited] (FLINK-24346) Flink on yarn application mode,LinkageError

["Biao Geng (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-24346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17561986#comment-17561986 ] 

Biao Geng edited comment on FLINK-24346 at 7/4/22 6:47 AM:
-----------------------------------------------------------

Hi [~soberChina0@gamil.com] have you ever tried to run the program with {{flink run-application -t yarn-application  -Dyarn.per-job-cluster.include-user-jar=DISABLED  flink-demo-1.13.3.jar}} ?  (the option name `yarn.per-job-cluster.include-user-jar` is somehow misleading as it can work for application mode as well. This naming has been improved in flink1.15)
I test your program locally and above command can be a workaround for the class loading problem. 
Besides, I believe your question is truly valuable and we may need to spend more time finding a way to see what happened for the class loading process.


was (Author: bgeng777):
Hi [~soberChina0@gamil.com] have you ever tried to run the program with {{flink run-application -t yarn-application  -Dyarn.per-job-cluster.include-user-jar=DISABLED  flink-demo-1.13.3.jar}} ?  (the option name `yarn.per-job-cluster.include-user-jar` is somehow misleading as it can work for application mode as well. This naming has been improved in flink1.15)
I test your program locally and above command can be a workaround for the class loading problem.

> Flink on yarn application mode,LinkageError
> -------------------------------------------
>
>                 Key: FLINK-24346
>                 URL: https://issues.apache.org/jira/browse/FLINK-24346
>             Project: Flink
>          Issue Type: Bug
>          Components: API / DataStream
>    Affects Versions: 1.13.1
>         Environment: hadoop version 2.6.x
>            Reporter: 李伟高
>            Priority: Major
>
> Hello, I'm changing from per job mode to application mode to submit tasks to yarn.All jars that my task depends on are typed into my task jar.I submit the task as perjob and work normally, but change to application mode and report an error.
> {code:java}
> [0;39mjava.util.concurrent.CompletionException: org.apache.flink.client.deployment.application.ApplicationExecutionException: Could not execute application. at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[na:1.8.0_271] at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[na:1.8.0_271] at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:957) ~[na:1.8.0_271] at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:940) ~[na:1.8.0_271] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[na:1.8.0_271] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[na:1.8.0_271] at org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:257) ~[hb-bigdata-wash-location-1.0-SNAPSHOT.jar:na] at org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAsync$1(ApplicationDispatcherBootstrap.java:212) ~[hb-bigdata-wash-location-1.0-SNAPSHOT.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_271] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_271] at org.apache.flink.runtime.concurrent.akka.ActorSystemScheduledExecutorAdapter$ScheduledFutureTask.run(ActorSystemScheduledExecutorAdapter.java:159) ~[hb-bigdata-wash-location-1.0-SNAPSHOT.jar:na] at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) ~[hb-bigdata-wash-location-1.0-SNAPSHOT.jar:na] at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44) ~[hb-bigdata-wash-location-1.0-SNAPSHOT.jar:na] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[hb-bigdata-wash-location-1.0-SNAPSHOT.jar:na] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[hb-bigdata-wash-location-1.0-SNAPSHOT.jar:na] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[hb-bigdata-wash-location-1.0-SNAPSHOT.jar:na] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[hb-bigdata-wash-location-1.0-SNAPSHOT.jar:na] Caused by: org.apache.flink.client.deployment.application.ApplicationExecutionException: Could not execute application. ... 11 common frames omitted Caused by: java.lang.LinkageError: loader constraint violation: loader (instance of org/apache/flink/util/ChildFirstClassLoader) previously initiated loading for a different type with name "org/elasticsearch/client/RestClientBuilder" at java.lang.ClassLoader.defineClass1(Native Method) ~[na:1.8.0_271] at java.lang.ClassLoader.defineClass(ClassLoader.java:756) ~[na:1.8.0_271] at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) ~[na:1.8.0_271] at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) ~[na:1.8.0_271] at java.net.URLClassLoader.access$100(URLClassLoader.java:74) ~[na:1.8.0_271] at java.net.URLClassLoader$1.run(URLClassLoader.java:369) ~[na:1.8.0_271] at java.net.URLClassLoader$1.run(URLClassLoader.java:363) ~[na:1.8.0_271] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_271] at java.net.URLClassLoader.findClass(URLClassLoader.java:362) ~[na:1.8.0_271] at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71) ~[hb-bigdata-wash-location-1.0-SNAPSHOT.jar:na] at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48) ~[hb-bigdata-wash-location-1.0-SNAPSHOT.jar:na] at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[na:1.8.0_271] at java.lang.Class.getDeclaredMethods0(Native Method) ~[na:1.8.0_271] at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) ~[na:1.8.0_271] at java.lang.Class.getDeclaredMethod(Class.java:2128) ~[na:1.8.0_271] at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1644) ~[na:1.8.0_271] at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:79) ~[na:1.8.0_271] at java.io.ObjectStreamClass$3.run(ObjectStreamClass.java:520) ~[na:1.8.0_271] at java.io.ObjectStreamClass$3.run(ObjectStreamClass.java:494) ~[na:1.8.0_271] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_271]
> {code}
> I checked a lot of relevant information and told me to modify the
> {code:java}
> classloader.resolve-order
> {code}
>  parameter. I tried back and forth from `parent-first` to `child-first`.The error is still reported. I think this question should be asked on stackoverflow or other websites. I have tried many times,I don't know what's wrong.
> I test the example you gave. It works well. Shouldn't the jar that my project depends on be typed into my task jar? If so, why is per job or yen session OK,
> We look forward to your reply , Thanks!
>  
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)