You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Dhiru <us...@yahoo.com> on 2021/09/24 17:45:31 UTC
how to run job and make jobmanager HA
spec: replicas: 1 selector: matchLabels: app: flink component: jobmanager template: metadata: labels: app: flink component: jobmanager spec: serviceAccountName: msc-s3-shared-content containers: - name: jobmanager image: test:latest ( flink:1.11.3-scala_2.12-java11 image has DeliveryStreams-0.0.1_3.1.0.jar jar copied to ./bin/flink) args: ["jobmanager"] command: ['./bin/flink', 'run', './bin/DeliveryStreams-0.0.1_3.1.0.jar', 'DeduplicationJob'] . ( I am planning to run job ... Please let me know if this is right way) ports: - containerPort: 6123 name: rpc - containerPort: 6124 name: blob-server - containerPort: 8081 name: webui env: - name: JOB_MANAGER_RPC_ADDRESS value: flink-jobmanager - name: KAFKA_BROKERS value: kafka:29092 livenessProbe: tcpSocket: port: 6123 initialDelaySeconds: 30 periodSeconds: 60 volumeMounts: - name: flink-config-volume mountPath: /opt/flink/conf securityContext: runAsUser: 9999 # refers to user _flink_ from official flink image, change if necessary volumes: - name: flink-config-volume configMap: name: flink-config items: - key: flink-conf.yaml path: flink-conf.yaml - key: log4j-console.properties path: log4j-console.properties"msc-jobmanager-deployment.yaml" 54L, 1640C
a) Do not want to run my job from UI , want to run my job from jobmanager image during boot time , when I am trying to run its throwing me error b) How to make sure my jobManager and taskmanager is HA configured ( so that if jobmanager goes off do not loose data)
Thanks Kumar
Re: how to run job and make jobmanager HA
Posted by houssem <me...@gmail.com>.
1)You can use the application cluster mode you can find how to configure in the official flink documentation
https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html#deploy-application-cluster
2)for HA you can use kubernetes HA:
https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/ha/kubernetes_ha.html
Best,
On 2021/09/24 22:58:48, Dhiru <us...@yahoo.com> wrote:
>
> please let me know if anyone can help me on this On Friday, September 24, 2021, 01:45:39 PM EDT, Dhiru <us...@yahoo.com> wrote:
>
> spec: replicas: 1 selector: matchLabels: app: flink component: jobmanager template: metadata: labels: app: flink component: jobmanager spec: serviceAccountName: msc-s3-shared-content containers: - name: jobmanager image: test:latest ( flink:1.11.3-scala_2.12-java11 image has DeliveryStreams-0.0.1_3.1.0.jar jar copied to ./bin/flink) args: ["jobmanager"] command: ['./bin/flink', 'run', './bin/DeliveryStreams-0.0.1_3.1.0.jar', 'DeduplicationJob'] . ( I am planning to run job ... Please let me know if this is right way) ports: - containerPort: 6123 name: rpc - containerPort: 6124 name: blob-server - containerPort: 8081 name: webui env: - name: JOB_MANAGER_RPC_ADDRESS value: flink-jobmanager - name: KAFKA_BROKERS
value: kafka:29092 livenessProbe: tcpSocket: port: 6123 initialDelaySeconds: 30 periodSeconds: 60 volumeMounts: - name: flink-config-volume mountPath: /opt/flink/conf securityContext: runAsUser: 9999 # refers to user _flink_ from official flink image, change if necessary volumes: - name: flink-config-volume configMap: name: flink-config items: - key: flink-conf.yaml path: flink-conf.yaml - key: log4j-console.properties path: log4j-console.properties"msc-jobmanager-deployment.yaml" 54L, 1640C
> a) Do not want to run my job from UI , want to run my job from jobmanager image during boot time , when I am trying to run its throwing me error b) How to make sure my jobManager and taskmanager is HA configured ( so that if jobmanager goes off do not loose data)
> Thanks Kumar
>
>
Re: how to run job and make jobmanager HA
Posted by Dhiru <us...@yahoo.com>.
please let me know if anyone can help me on this On Friday, September 24, 2021, 01:45:39 PM EDT, Dhiru <us...@yahoo.com> wrote:
spec: replicas: 1 selector: matchLabels: app: flink component: jobmanager template: metadata: labels: app: flink component: jobmanager spec: serviceAccountName: msc-s3-shared-content containers: - name: jobmanager image: test:latest ( flink:1.11.3-scala_2.12-java11 image has DeliveryStreams-0.0.1_3.1.0.jar jar copied to ./bin/flink) args: ["jobmanager"] command: ['./bin/flink', 'run', './bin/DeliveryStreams-0.0.1_3.1.0.jar', 'DeduplicationJob'] . ( I am planning to run job ... Please let me know if this is right way) ports: - containerPort: 6123 name: rpc - containerPort: 6124 name: blob-server - containerPort: 8081 name: webui env: - name: JOB_MANAGER_RPC_ADDRESS value: flink-jobmanager - name: KAFKA_BROKERS value: kafka:29092 livenessProbe: tcpSocket: port: 6123 initialDelaySeconds: 30 periodSeconds: 60 volumeMounts: - name: flink-config-volume mountPath: /opt/flink/conf securityContext: runAsUser: 9999 # refers to user _flink_ from official flink image, change if necessary volumes: - name: flink-config-volume configMap: name: flink-config items: - key: flink-conf.yaml path: flink-conf.yaml - key: log4j-console.properties path: log4j-console.properties"msc-jobmanager-deployment.yaml" 54L, 1640C
a) Do not want to run my job from UI , want to run my job from jobmanager image during boot time , when I am trying to run its throwing me error b) How to make sure my jobManager and taskmanager is HA configured ( so that if jobmanager goes off do not loose data)
Thanks Kumar