You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Thad Truman <tt...@neovest.com> on 2018/10/18 16:21:43 UTC
Flink Job Cluster Deployment on K8s
Hello,
I am trying to experiment with the new Flink job cluster on Kubernetes that is available with the Flink 1.6.x release.
I am using the instructions here<https://github.com/apache/flink/blob/release-1.6/flink-container/docker/README.md> to create the docker image, which is working great. This image then gets pushed to our Artifactory.
I am able to create the job cluster service using this<https://github.com/apache/flink/blob/release-1.6/flink-container/kubernetes/job-cluster-service.yaml> helm chart.
However when I try to deploy the job cluster job using the helm chart below (based on this<https://github.com/apache/flink/blob/release-1.6/flink-container/kubernetes/job-cluster-job.yaml.template> one):
apiVersion: batch/v1
kind: Job
metadata:
name: flink-job-cluster
spec:
template:
metadata:
labels:
app: flink
component: job-cluster
spec:
imagePullSecrets:
- name: artifactory-docker-registry
restartPolicy: OnFailure
containers:
- name: flink-job-cluster
image: {ImageOnArtifactory}
args: ["job-cluster", "--job-classname", "job.jar", "-Djobmanager.rpc.address=flink-job-cluster",
"-Dparallelism.default=1", "-Dblob.server.port=6124", "-Dquery.server.ports=6125"]
ports:
- containerPort: 6123
name: rpc
- containerPort: 6124
name: blob
- containerPort: 6125
name: query
- containerPort: 8081
name: ui
I get this error on the pod:
%d [%thread] %-5level %logger - %msg%n org.apache.flink.util.FlinkException: Could not load the provied entrypoint class.
at org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.createPackagedProgram(StandaloneJobClusterEntryPoint.java:102) ~[flink-dist_2.11-1.6.1.jar:1.6.1]
at org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.retrieveJobGraph(StandaloneJobClusterEntryPoint.java:84) ~[flink-dist_2.11-1.6.1.jar:1.6.1]
at org.apache.flink.runtime.entrypoint.JobClusterEntrypoint.createDispatcher(JobClusterEntrypoint.java:107) ~[flink-dist_2.11-1.6.1.jar:1.6.1]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startClusterComponents(ClusterEntrypoint.java:353) ~[flink-dist_2.11-1.6.1.jar:1.6.1]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:232) ~[flink-dist_2.11-1.6.1.jar:1.6.1]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:190) ~[flink-dist_2.11-1.6.1.jar:1.6.1]
at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) ~[flink-dist_2.11-1.6.1.jar:1.6.1]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:189) [flink-dist_2.11-1.6.1.jar:1.6.1]
at org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.main(StandaloneJobClusterEntryPoint.java:170) [flink-dist_2.11-1.6.1.jar:1.6.1]
Caused by: java.lang.ClassNotFoundException: job.jar
at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_111-internal]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_111-internal]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) ~[?:1.8.0_111-internal]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_111-internal]
at org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.createPackagedProgram(StandaloneJobClusterEntryPoint.java:99) ~[flink-dist_2.11-1.6.1.jar:1.6.1]
... 8 more
When I launch and attach the docker image, job.jar exists in /opt/flink and there is a symbolic link in /opt/flink/lib.
Any ideas as to why job.jar can't be found?
Our flink version is 1.6.1. We are using the flink-1.6.1-bin-scala_2.11 distribution.
Any help would be much appreciated.
Thanks,
Thad Truman | Software Engineer | Neovest, Inc.
A:
T:
E:
1145 S 800 E, Ste 310 Orem, UT 84097
+1 801 900 2480
ttruman@neovest.com<ma...@neovest.com>
Support Desk: support@neovest.com<ma...@neovest.com> / +1 800 433 4276
[Alt logo for white backgrounds (Grey Flat)2]
This email is confidential and subject to important disclaimers and conditions including on offers for purchase or sale of securities accuracy and completeness of information viruses confidentiality legal privilege and legal entity disclaimers available at www.neovest.com/disclosures.html<http://www.neovest.com/disclosures.html>
Re: Flink Job Cluster Deployment on K8s
Posted by Joey Echeverria <je...@splunk.com>.
Try replacing the job.jar in the args in your helm chart with the classname for your job rather than the name of the jar file.
-Joey
On Oct 18, 2018, at 9:21 AM, Thad Truman <tt...@neovest.com>> wrote:
Hello,
I am trying to experiment with the new Flink job cluster on Kubernetes that is available with the Flink 1.6.x release.
I am using the instructions here<https://github.com/apache/flink/blob/release-1.6/flink-container/docker/README.md> to create the docker image, which is working great. This image then gets pushed to our Artifactory.
I am able to create the job cluster service using this<https://github.com/apache/flink/blob/release-1.6/flink-container/kubernetes/job-cluster-service.yaml> helm chart.
However when I try to deploy the job cluster job using the helm chart below (based on this<https://github.com/apache/flink/blob/release-1.6/flink-container/kubernetes/job-cluster-job.yaml.template> one):
apiVersion: batch/v1
kind: Job
metadata:
name: flink-job-cluster
spec:
template:
metadata:
labels:
app: flink
component: job-cluster
spec:
imagePullSecrets:
- name: artifactory-docker-registry
restartPolicy: OnFailure
containers:
- name: flink-job-cluster
image: {ImageOnArtifactory}
args: ["job-cluster", "--job-classname", "job.jar", "-Djobmanager.rpc.address=flink-job-cluster",
"-Dparallelism.default=1", "-Dblob.server.port=6124", "-Dquery.server.ports=6125"]
ports:
- containerPort: 6123
name: rpc
- containerPort: 6124
name: blob
- containerPort: 6125
name: query
- containerPort: 8081
name: ui
I get this error on the pod:
%d [%thread] %-5level %logger - %msg%n org.apache.flink.util.FlinkException: Could not load the provied entrypoint class.
at org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.createPackagedProgram(StandaloneJobClusterEntryPoint.java:102) ~[flink-dist_2.11-1.6.1.jar:1.6.1]
at org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.retrieveJobGraph(StandaloneJobClusterEntryPoint.java:84) ~[flink-dist_2.11-1.6.1.jar:1.6.1]
at org.apache.flink.runtime.entrypoint.JobClusterEntrypoint.createDispatcher(JobClusterEntrypoint.java:107) ~[flink-dist_2.11-1.6.1.jar:1.6.1]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startClusterComponents(ClusterEntrypoint.java:353) ~[flink-dist_2.11-1.6.1.jar:1.6.1]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:232) ~[flink-dist_2.11-1.6.1.jar:1.6.1]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:190) ~[flink-dist_2.11-1.6.1.jar:1.6.1]
at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) ~[flink-dist_2.11-1.6.1.jar:1.6.1]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:189) [flink-dist_2.11-1.6.1.jar:1.6.1]
at org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.main(StandaloneJobClusterEntryPoint.java:170) [flink-dist_2.11-1.6.1.jar:1.6.1]
Caused by: java.lang.ClassNotFoundException: job.jar
at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_111-internal]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_111-internal]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) ~[?:1.8.0_111-internal]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_111-internal]
at org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.createPackagedProgram(StandaloneJobClusterEntryPoint.java:99) ~[flink-dist_2.11-1.6.1.jar:1.6.1]
... 8 more
When I launch and attach the docker image, job.jar exists in /opt/flink and there is a symbolic link in /opt/flink/lib.
Any ideas as to why job.jar can’t be found?
Our flink version is 1.6.1. We are using the flink-1.6.1-bin-scala_2.11 distribution.
Any help would be much appreciated.
Thanks,
Thad Truman | Software Engineer | Neovest, Inc.
A:
T:
E:
1145 S 800 E, Ste 310 Orem, UT 84097
+1 801 900 2480
ttruman@neovest.com<ma...@neovest.com>
Support Desk: support@neovest.com<ma...@neovest.com> / +1 800 433 4276
<image001.png>
This email is confidential and subject to important disclaimers and conditions including on offers for purchase or sale of securities accuracy and completeness of information viruses confidentiality legal privilege and legal entity disclaimers available at www.neovest.com/disclosures.html<http://www.neovest.com/disclosures.html>