You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Dhiru <us...@yahoo.com> on 2021/10/05 09:34:47 UTC

k8s not able to submit job from jobmanager

My DockerFile 
FROM flink:1.13.2-scala_2.12-java11
RUN mkdir -p /opt/flink/plugins/flink-s3-fs-hadoopRUN ln -fs /opt/flink/opt/flink-s3-fs-hadoop-*.jar /opt/flink/plugins/flink-s3-fs-hadoop/.
RUN mkdir -p /opt/flink/plugins/flink-s3-fs-prestoRUN ln -fs /opt/flink/opt/flink-s3-fs-presto-*.jar /opt/flink/plugins/flink-s3-fs-presto/.
COPY WordCount.jar  /opt/flink/bin/

I am trying to run the job manually from my local laptop, able to run job successfullydocker run -it images sh 
./start-cluster.sh flink run WordCount.sh 
for Kubernetes Now using the same image am trying to create a cluster in Application mode (https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/resource-providers/native_kubernetes/)

 Trying to use the same image and run from Kubernetes jobmanager, getting this error 
k exec -it flink-jobmanager-kzgwk  sh -n ha 

$ flink run WordCount.jar Executing WordCount example with default input data set.Use --input to specify file input.Printing result to stdout. Use --output to specify output path.WARNING: An illegal reflective access operation has occurredWARNING: Illegal reflective access by org.apache.flink.api.java.ClosureCleaner (file:/opt/flink/lib/flink-dist_2.12-1.13.2.jar) to field java.lang.String.valueWARNING: Please consider reporting this to the maintainers of org.apache.flink.api.java.ClosureCleanerWARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operationsWARNING: All illegal access operations will be denied in a future release09:30:38.530 [main] ERROR org.apache.flink.client.cli.CliFrontend - Error while running the command.org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Failed to execute job 'Streaming WordCount'.        at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372) ~[flink-dist_2.12-1.13.2.jar:1.13.2]        at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) ~[flink-dist_2.12-1.13.2.jar:1.13.2]        at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) ~[flink-dist_2.12-1.13.2.jar:1.13.2]        at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812) ~[flink-dist_2.12-1.13.2.jar:1.13.2]        at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246) ~[flink-dist_2.12-1.13.2.jar:1.13.2]        at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054) ~[flink-dist_2.12-1.13.2.jar:1.13.2]        at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132) ~[flink-dist_2.12-1.13.2.jar:1.13.2]        at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28) [flink-dist_2.12-1.13.2.jar:1.13.2]        at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132) [flink-dist_2.12-1.13.2.jar:1.13.2]Caused by: org.apache.flink.util.FlinkException: Failed to execute job 'Streaming WordCount'.
anything which I am missing ?

Re: k8s not able to submit job from jobmanager

Posted by Dhiru <us...@yahoo.com>.
 I think , I got the answer ( application mode cluster doesn't support running job using cli )
    On Tuesday, October 5, 2021, 08:38:24 AM EDT, Israel Ekpo <is...@gmail.com> wrote:  
 
 Your Flink versions are different 
Your Docker container has version 1.13.2 but it seems your job is attempting to submit with 1.14 in application mode 
That is the first obvious observation 
On Tue, Oct 5, 2021 at 5:35 AM Dhiru <us...@yahoo.com> wrote:

My DockerFile 
FROM flink:1.13.2-scala_2.12-java11
RUN mkdir -p /opt/flink/plugins/flink-s3-fs-hadoopRUN ln -fs /opt/flink/opt/flink-s3-fs-hadoop-*.jar /opt/flink/plugins/flink-s3-fs-hadoop/.
RUN mkdir -p /opt/flink/plugins/flink-s3-fs-prestoRUN ln -fs /opt/flink/opt/flink-s3-fs-presto-*.jar /opt/flink/plugins/flink-s3-fs-presto/.
COPY WordCount.jar  /opt/flink/bin/

I am trying to run the job manually from my local laptop, able to run job successfullydocker run -it images sh 
./start-cluster.sh flink run WordCount.sh 
for Kubernetes Now using the same image am trying to create a cluster in Application mode (https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/resource-providers/native_kubernetes/)

 Trying to use the same image and run from Kubernetes jobmanager, getting this error 
k exec -it flink-jobmanager-kzgwk  sh -n ha 

$ flink run WordCount.jar Executing WordCount example with default input data set.Use --input to specify file input.Printing result to stdout. Use --output to specify output path.WARNING: An illegal reflective access operation has occurredWARNING: Illegal reflective access by org.apache.flink.api.java.ClosureCleaner (file:/opt/flink/lib/flink-dist_2.12-1.13.2.jar) to field java.lang.String.valueWARNING: Please consider reporting this to the maintainers of org.apache.flink.api.java.ClosureCleanerWARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operationsWARNING: All illegal access operations will be denied in a future release09:30:38.530 [main] ERROR org.apache.flink.client.cli.CliFrontend - Error while running the command.org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Failed to execute job 'Streaming WordCount'.        at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372) ~[flink-dist_2.12-1.13.2.jar:1.13.2]        at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) ~[flink-dist_2.12-1.13.2.jar:1.13.2]        at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) ~[flink-dist_2.12-1.13.2.jar:1.13.2]        at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812) ~[flink-dist_2.12-1.13.2.jar:1.13.2]        at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246) ~[flink-dist_2.12-1.13.2.jar:1.13.2]        at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054) ~[flink-dist_2.12-1.13.2.jar:1.13.2]        at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132) ~[flink-dist_2.12-1.13.2.jar:1.13.2]        at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28) [flink-dist_2.12-1.13.2.jar:1.13.2]        at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132) [flink-dist_2.12-1.13.2.jar:1.13.2]Caused by: org.apache.flink.util.FlinkException: Failed to execute job 'Streaming WordCount'.
anything which I am missing ?

  

Re: k8s not able to submit job from jobmanager

Posted by Israel Ekpo <is...@gmail.com>.
Your Flink versions are different

Your Docker container has version 1.13.2 but it seems your job is
attempting to submit with 1.14 in application mode

That is the first obvious observation

On Tue, Oct 5, 2021 at 5:35 AM Dhiru <us...@yahoo.com> wrote:

> *My DockerFile *
>
> FROM flink:1.13.2-scala_2.12-java11
>
> RUN mkdir -p /opt/flink/plugins/flink-s3-fs-hadoop
> RUN ln -fs /opt/flink/opt/flink-s3-fs-hadoop-*.jar
> /opt/flink/plugins/flink-s3-fs-hadoop/.
>
> RUN mkdir -p /opt/flink/plugins/flink-s3-fs-presto
> RUN ln -fs /opt/flink/opt/flink-s3-fs-presto-*.jar
> /opt/flink/plugins/flink-s3-fs-presto/.
>
> COPY WordCount.jar  /opt/flink/bin/
>
>
> I am trying to run the job manually from my local laptop, able to run job
> successfully
> docker run -it images sh
>
> ./start-cluster.sh
> flink run WordCount.sh
>
> *for Kubernetes *
> Now using the same image am trying to create a cluster in Application mode
> (
> https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/resource-providers/native_kubernetes/
> )
>
>
>  Trying to use the same image and run from Kubernetes jobmanager, getting
> this error
>
> k exec -it flink-jobmanager-kzgwk  sh -n ha
>
> $ flink run WordCount.jar
> Executing WordCount example with default input data set.
> Use --input to specify file input.
> Printing result to stdout. Use --output to specify output path.
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.flink.api.java.ClosureCleaner
> (file:/opt/flink/lib/flink-dist_2.12-1.13.2.jar) to field
> java.lang.String.value
> WARNING: Please consider reporting this to the maintainers of
> org.apache.flink.api.java.ClosureCleaner
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> *09:30:38.530 [main] ERROR org.apache.flink.client.cli.CliFrontend - Error
> while running the command.*
> org.apache.flink.client.program.ProgramInvocationException: The main
> method caused an error: Failed to execute job 'Streaming WordCount'.
>         at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
> ~[flink-dist_2.12-1.13.2.jar:1.13.2]
>         at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
> ~[flink-dist_2.12-1.13.2.jar:1.13.2]
>         at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
> ~[flink-dist_2.12-1.13.2.jar:1.13.2]
>         at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
> ~[flink-dist_2.12-1.13.2.jar:1.13.2]
>         at
> org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
> ~[flink-dist_2.12-1.13.2.jar:1.13.2]
>         at
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
> ~[flink-dist_2.12-1.13.2.jar:1.13.2]
>         at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
> ~[flink-dist_2.12-1.13.2.jar:1.13.2]
>         at
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
> [flink-dist_2.12-1.13.2.jar:1.13.2]
>         at
> org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
> [flink-dist_2.12-1.13.2.jar:1.13.2]
> Caused by: org.apache.flink.util.FlinkException: Failed to execute job
> 'Streaming WordCount'.
>
> anything which I am missing ?
>
>