You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Rasmus Thygesen (Jira)" <ji...@apache.org> on 2024/05/15 07:56:00 UTC
[jira] [Created] (FLINK-35358) Breaking change when loading artifacts
Rasmus Thygesen created FLINK-35358:
---------------------------------------
Summary: Breaking change when loading artifacts
Key: FLINK-35358
URL: https://issues.apache.org/jira/browse/FLINK-35358
Project: Flink
Issue Type: New Feature
Components: Client / Job Submission, flink-docker
Affects Versions: 1.19.0
Reporter: Rasmus Thygesen
We have been using the following code snippet in our Dockerfiles for running a Flink job in application mode
{code:java}
FROM flink:1.18.1-scala_2.12-java17
COPY --from=build /app/target/my-job*.jar /opt/flink/usrlib/artifacts/my-job.jar
USER flink {code}
Which has been working since at least around Flink 1.14, but the 1.19 update has broken our Dockerfiles. The fix is to put the jar file a step further out so the code snippet becomes
{code:java}
FROM flink:1.18.1-scala_2.12-java17
COPY --from=build /app/target/my-job*.jar /opt/flink/usrlib/my-job.jar
USER flink {code}
We have not spent too much time looking into what the cause is, but we get the stack trace
{code:java}
myjob-jobmanager-1 | org.apache.flink.util.FlinkException: Could not load the provided entrypoint class.myjob-jobmanager-1 | at org.apache.flink.client.program.DefaultPackagedProgramRetriever.getPackagedProgram(DefaultPackagedProgramRetriever.java:230) ~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.getPackagedProgram(StandaloneApplicationClusterEntryPoint.java:149) ~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.lambda$main$0(StandaloneApplicationClusterEntryPoint.java:90) ~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28) ~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:89) [flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | Caused by: org.apache.flink.client.program.ProgramInvocationException: The program's entry point class 'my.company.job.MyJob' was not found in the jar file.myjob-jobmanager-1 | at org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:481) ~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:153) ~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:65) ~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at org.apache.flink.client.program.PackagedProgram$Builder.build(PackagedProgram.java:691) ~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at org.apache.flink.client.program.DefaultPackagedProgramRetriever.getPackagedProgram(DefaultPackagedProgramRetriever.java:228) ~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | ... 4 moremyjob-jobmanager-1 | Caused by: java.lang.ClassNotFoundException: my.company.job.MyJobmyjob-jobmanager-1 | at java.net.URLClassLoader.findClass(Unknown Source) ~[?:?]myjob-jobmanager-1 | at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?]myjob-jobmanager-1 | at org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:67) ~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:74) ~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:51) ~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?]myjob-jobmanager-1 | at org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.loadClass(FlinkUserCodeClassLoaders.java:197) ~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at java.lang.Class.forName0(Native Method) ~[?:?]myjob-jobmanager-1 | at java.lang.Class.forName(Unknown Source) ~[?:?]myjob-jobmanager-1 | at org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:479) ~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:153) ~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:65) ~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at org.apache.flink.client.program.PackagedProgram$Builder.build(PackagedProgram.java:691) ~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at org.apache.flink.client.program.DefaultPackagedProgramRetriever.getPackagedProgram(DefaultPackagedProgramRetriever.java:228) ~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | ... 4 more{code}
I have changed some text in the stack trace to keep it anonymous so it is possible there is a typo but that is not the issue. As you can see, the stack trace leads to PackagedProgram and DefaultPackagedProgramRetriever to which the only commits after Flink 1.18 are [PackagedProgram commit|https://github.com/apache/flink/commit/d0ce5349fdf1a611518eba20a169c475ee0b46c5] and [DefaultPackagedProgramRetriever commit|https://github.com/apache/flink/commit/e63aa12252843d0098a56f3091b28d48aff5b5af] and we suspect the culprit is the latter, specifically [this line|https://github.com/apache/flink/commit/e63aa12252843d0098a56f3091b28d48aff5b5af#diff-11b5162d6745014c68e96303d26c71bdb88bac068c27834dbdbb7c9089ffbe9fL227] which we think has made the artifact check non-recursive. We assume it is intended to have your artifacts directly in /opt/flink/usrlib without the artifacts directory so we are planning on changing that for our Dockerfiles anyway, but it is still a breaking change so we wanted to make an issue on it first.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)