You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Julian Cardarelli <ju...@thentia.com> on 2021/09/02 17:54:22 UTC
Flink on Kubernetes
Hello -
We have implemented Flink on Kubernetes with Google Cloud Storage in high availability configuration as per the below configmap. Everything appears to be working normally, state is being saved to GCS.
However, every now and then - perhaps weekly or every other week, all of the submitted jobs are lost and the cluster appears completely reset. Perhaps GKE is doing maintenance or something of this nature, but the point being that the cluster does not resume from this activity in an operational state with all jobs placed into running status.
Is there something we are missing? Thanks!
-jc
apiVersion: v1
kind: ConfigMap
metadata:
name: flink-config
labels:
app: flink
data:
flink-conf.yaml: |+
jobmanager.rpc.address: flink-jobmanager
taskmanager.numberOfTaskSlots: 1
blob.server.port: 6124
jobmanager.rpc.port: 6123
taskmanager.rpc.port: 6122
jobmanager.heap.size: 1024m
taskmanager.memory.process.size: 1024m
kubernetes.cluster-id: cluster1
high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
high-availability.storageDir: gs://storage-uswest.yyy.com/kubernetes-flink
state.backend: filesystem
state.checkpoints.dir: gs://storage-uswest.yyy.com/kubernetes-checkpoint
state.savepoints.dir: gs://storage-uswest.yyy.com/kubernetes-savepoint
execution.checkpointing.interval: 3min
execution.checkpointing.externalized-checkpoint-retention: DELETE_ON_CANCELLATION
execution.checkpointing.max-concurrent-checkpoints: 1
execution.checkpointing.min-pause: 0
execution.checkpointing.mode: EXACTLY_ONCE
execution.checkpointing.timeout: 10min
execution.checkpointing.tolerable-failed-checkpoints: 0
execution.checkpointing.unaligned: false
restart-strategy: fixed-delay
restart-strategy.fixed-delay.attempts: 10
restart-strategy.fixed-delay.delay 10s
log4j.properties: |+
log4j.rootLogger=INFO, file
log4j.logger.akka=INFO
log4j.logger.org.apache.kafka=INFO
log4j.logger.org.apache.hadoop=INFO
log4j.logger.org.apache.zookeeper=INFO
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n
log4j.logger.org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline=ERROR, file
___
Julian Cardarelli
CEO
T (800) 961-1549
Ejulian@thentia.com
LinkedIn
DISCLAIMER
Neither Thentia Corporation, nor its directors, officers, shareholders, representatives, employees, non-arms length companies, subsidiaries, parent, affiliated brands and/or agencies are licensed to provide legal advice. This e-mail may contain among other things legal information. We disclaim any and all responsibility for the content of this e-mail. YOU MUST NOT rely on any of our communications as legal advice. Only a licensed legal professional may give you advice. Our communications are never provided as legal advice, because we are not licensed to provide legal advice nor do we possess the knowledge, skills or capacity to provide legal advice. We disclaim any and all responsibility related to any action you might take based upon our communications and emphasize the need for you to never rely on our communications as the basis of any claim or proceeding.
CONFIDENTIALITY
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual(s) named. If you are not the named addressee(s) you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.
Disclaimer
The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.
This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast, a leader in email security and cyber resilience. Mimecast integrates email defenses with brand protection, security awareness training, web security, compliance and other essential capabilities. Mimecast helps protect large and small organizations from malicious activity, human error and technology failure; and to lead the movement toward building a more resilient world. To find out more, visit our website.
RE: [External] Re: Flink on Kubernetes
Posted by Julian Cardarelli <ju...@thentia.com>.
OK – I will increase the value to something higher and see how it does in recovering. Thank you for your help!
___
Julian Cardarelli
CEO
T (800) 961-1549
Ejulian@thentia.com
LinkedIn
DISCLAIMER
Neither Thentia Corporation, nor its directors, officers, shareholders, representatives, employees, non-arms length companies, subsidiaries, parent, affiliated brands and/or agencies are licensed to provide legal advice. This e-mail may contain among other things legal information. We disclaim any and all responsibility for the content of this e-mail. YOU MUST NOT rely on any of our communications as legal advice. Only a licensed legal professional may give you advice. Our communications are never provided as legal advice, because we are not licensed to provide legal advice nor do we possess the knowledge, skills or capacity to provide legal advice. We disclaim any and all responsibility related to any action you might take based upon our communications and emphasize the need for you to never rely on our communications as the basis of any claim or proceeding.
CONFIDENTIALITY
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual(s) named. If you are not the named addressee(s) you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.
From: Guowei Ma <gu...@gmail.com>
Sent: Thursday, September 2, 2021 11:32 PM
To: Julian Cardarelli <ju...@thentia.com>
Cc: user <us...@flink.apache.org>
Subject: [External] Re: Flink on Kubernetes
Hi, Julian
I notice that your configuration includes "restart-strategy.fixed-delay.attempts: 10". It means that the job would fail after 10 times failure. So maybe it leads to the job not restarting again and you could increase this value.
But I am not sure if this is the root cause. So if this does not work I think you could share the log at that time and the flink version you use.
Best,
Guowei
On Fri, Sep 3, 2021 at 2:00 AM Julian Cardarelli <ju...@thentia.com>> wrote:
Hello –
We have implemented Flink on Kubernetes with Google Cloud Storage in high availability configuration as per the below configmap. Everything appears to be working normally, state is being saved to GCS.
However, every now and then – perhaps weekly or every other week, all of the submitted jobs are lost and the cluster appears completely reset. Perhaps GKE is doing maintenance or something of this nature, but the point being that the cluster does not resume from this activity in an operational state with all jobs placed into running status.
Is there something we are missing? Thanks!
-jc
apiVersion: v1
kind: ConfigMap
metadata:
name: flink-config
labels:
app: flink
data:
flink-conf.yaml: |+
jobmanager.rpc.address: flink-jobmanager
taskmanager.numberOfTaskSlots: 1
blob.server.port: 6124
jobmanager.rpc.port: 6123
taskmanager.rpc.port: 6122
jobmanager.heap.size: 1024m
taskmanager.memory.process.size: 1024m
kubernetes.cluster-id: cluster1
high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
high-availability.storageDir: gs://storage-uswest.yyy.com/kubernetes-flink<https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fstorage-uswest.yyy.com%2Fkubernetes-flink&data=04%7C01%7Cjulian%40thentia.com%7C27156a30f4d74f0083a608d96e8b7831%7Caaed208b28414c339a4df5008ba71d0d%7C0%7C0%7C637662367559754751%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=NBB%2Fik2NdJzpbjvPHxFhB6%2BndkgLJ8qa7tLqUX%2FMbZk%3D&reserved=0>
state.backend: filesystem
state.checkpoints.dir: gs://storage-uswest.yyy.com/kubernetes-checkpoint<https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fstorage-uswest.yyy.com%2Fkubernetes-checkpoint&data=04%7C01%7Cjulian%40thentia.com%7C27156a30f4d74f0083a608d96e8b7831%7Caaed208b28414c339a4df5008ba71d0d%7C0%7C0%7C637662367559754751%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=7rj%2BTRqnzaGYtKSbs60NWQfjM7BqjdGYSpbYzyr0xsM%3D&reserved=0>
state.savepoints.dir: gs://storage-uswest.yyy.com/kubernetes-savepoint<https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fstorage-uswest.yyy.com%2Fkubernetes-savepoint&data=04%7C01%7Cjulian%40thentia.com%7C27156a30f4d74f0083a608d96e8b7831%7Caaed208b28414c339a4df5008ba71d0d%7C0%7C0%7C637662367559764703%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=f7Xkp3Rckj%2BieleEaX2lfNBcK4MQy3SHQ6if2stKLgI%3D&reserved=0>
execution.checkpointing.interval: 3min
execution.checkpointing.externalized-checkpoint-retention: DELETE_ON_CANCELLATION
execution.checkpointing.max-concurrent-checkpoints: 1
execution.checkpointing.min-pause: 0
execution.checkpointing.mode: EXACTLY_ONCE
execution.checkpointing.timeout: 10min
execution.checkpointing.tolerable-failed-checkpoints: 0
execution.checkpointing.unaligned: false
restart-strategy: fixed-delay
restart-strategy.fixed-delay.attempts: 10
restart-strategy.fixed-delay.delay 10s
log4j.properties: |+
log4j.rootLogger=INFO, file
log4j.logger.akka=INFO
log4j.logger.org.apache.kafka=INFO
log4j.logger.org.apache.hadoop=INFO
log4j.logger.org.apache.zookeeper=INFO
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n
log4j.logger.org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline=ERROR, file
___
Julian
Cardarelli
CEO
T
(800) 961-1549<tel:(800)%20961-1549>
E
julian@thentia.com<ma...@thentia.com>
LinkedIn<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fin%2Fjulian-cardarelli%2F&data=04%7C01%7Cjulian%40thentia.com%7C27156a30f4d74f0083a608d96e8b7831%7Caaed208b28414c339a4df5008ba71d0d%7C0%7C0%7C637662367559764703%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=HeMQDKZRcTQp0I2oUIjWFYe85AgW00bJVf7sdXPIFWE%3D&reserved=0>
[Thentia Website]<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.thentia.com%2F%3Futm_source%3Dsignature%26utm_medium%3Dbanner%26utm_campaign%3Devergreen&data=04%7C01%7Cjulian%40thentia.com%7C27156a30f4d74f0083a608d96e8b7831%7Caaed208b28414c339a4df5008ba71d0d%7C0%7C0%7C637662367559764703%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=FuDVRVXAX853W7JcfiUWpoQO7%2FkIMSqEAOhvkOp7dG0%3D&reserved=0>
DISCLAIMER
Neither Thentia Corporation, nor its directors, officers, shareholders, representatives, employees, non-arms length companies, subsidiaries, parent, affiliated brands and/or agencies are licensed to provide legal advice. This e-mail may contain among other things legal information. We disclaim any and all responsibility for the content of this e-mail. YOU MUST NOT rely on any of our communications as legal advice. Only a licensed legal professional may give you advice. Our communications are never provided as legal advice, because we are not licensed to provide legal advice nor do we possess the knowledge, skills or capacity to provide legal advice. We disclaim any and all responsibility related to any action you might take based upon our communications and emphasize the need for you to never rely on our communications as the basis of any claim or proceeding.
CONFIDENTIALITY
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual(s) named. If you are not the named addressee(s) you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.
Disclaimer
The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.
This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast, a leader in email security and cyber resilience. Mimecast integrates email defenses with brand protection, security awareness training, web security, compliance and other essential capabilities. Mimecast helps protect large and small organizations from malicious activity, human error and technology failure; and to lead the movement toward building a more resilient world. To find out more, visit our website.
Disclaimer
The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.
This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast, a leader in email security and cyber resilience. Mimecast integrates email defenses with brand protection, security awareness training, web security, compliance and other essential capabilities. Mimecast helps protect large and small organizations from malicious activity, human error and technology failure; and to lead the movement toward building a more resilient world. To find out more, visit our website.
Re: Flink on Kubernetes
Posted by Guowei Ma <gu...@gmail.com>.
Hi, Julian
I notice that your configuration
includes "restart-strategy.fixed-delay.attempts: 10". It means that the job
would fail after 10 times failure. So maybe it leads to the job not
restarting again and you could increase this value.
But I am not sure if this is the root cause. So if this does not work I
think you could share the log at that time and the flink version you use.
Best,
Guowei
On Fri, Sep 3, 2021 at 2:00 AM Julian Cardarelli <ju...@thentia.com> wrote:
> Hello –
>
>
>
> We have implemented Flink on Kubernetes with Google Cloud Storage in high
> availability configuration as per the below configmap. Everything appears
> to be working normally, state is being saved to GCS.
>
>
>
> However, every now and then – perhaps weekly or every other week, all of
> the submitted jobs are lost and the cluster appears completely reset.
> Perhaps GKE is doing maintenance or something of this nature, but the point
> being that the cluster does not resume from this activity in an operational
> state with all jobs placed into running status.
>
>
>
> Is there something we are missing? Thanks!
>
> -jc
>
>
>
>
>
> apiVersion: v1
>
> kind: ConfigMap
>
> metadata:
>
> name: flink-config
>
> labels:
>
> app: flink
>
> data:
>
> flink-conf.yaml: |+
>
> jobmanager.rpc.address: flink-jobmanager
>
> taskmanager.numberOfTaskSlots: 1
>
> blob.server.port: 6124
>
> jobmanager.rpc.port: 6123
>
> taskmanager.rpc.port: 6122
>
> jobmanager.heap.size: 1024m
>
> taskmanager.memory.process.size: 1024m
>
> kubernetes.cluster-id: cluster1
>
> high-availability:
> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
>
> high-availability.storageDir: gs://
> storage-uswest.yyy.com/kubernetes-flink
>
> state.backend: filesystem
>
> state.checkpoints.dir: gs://
> storage-uswest.yyy.com/kubernetes-checkpoint
>
> state.savepoints.dir: gs://storage-uswest.yyy.com/kubernetes-savepoint
>
> execution.checkpointing.interval: 3min
>
> execution.checkpointing.externalized-checkpoint-retention:
> DELETE_ON_CANCELLATION
>
> execution.checkpointing.max-concurrent-checkpoints: 1
>
> execution.checkpointing.min-pause: 0
>
> execution.checkpointing.mode: EXACTLY_ONCE
>
> execution.checkpointing.timeout: 10min
>
> execution.checkpointing.tolerable-failed-checkpoints: 0
>
> execution.checkpointing.unaligned: false
>
> restart-strategy: fixed-delay
>
> restart-strategy.fixed-delay.attempts: 10
>
> restart-strategy.fixed-delay.delay 10s
>
>
>
> log4j.properties: |+
>
> log4j.rootLogger=INFO, file
>
> log4j.logger.akka=INFO
>
> log4j.logger.org.apache.kafka=INFO
>
> log4j.logger.org.apache.hadoop=INFO
>
> log4j.logger.org.apache.zookeeper=INFO
>
> log4j.appender.file=org.apache.log4j.FileAppender
>
> log4j.appender.file.file=${log.file}
>
> log4j.appender.file.layout=org.apache.log4j.PatternLayout
>
> log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd
> HH:mm:ss,SSS} %-5p %-60c %x - %m%n
>
>
> log4j.logger.org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,
> file
>
>
>
>
> ___
> Julian Cardarelli
> CEO
> T *(800) 961-1549* <(800)%20961-1549>
> E *julian@thentia.com* <ju...@thentia.com>
> *LinkedIn* <https://www.linkedin.com/in/julian-cardarelli/>
> [image: Thentia Website]
> <https://www.thentia.com/?utm_source=signature&utm_medium=banner&utm_campaign=evergreen>
> DISCLAIMER
>
> Neither Thentia Corporation, nor its directors, officers, shareholders,
> representatives, employees, non-arms length companies, subsidiaries,
> parent, affiliated brands and/or agencies are licensed to provide legal
> advice. This e-mail may contain among other things legal information. We
> disclaim any and all responsibility for the content of this e-mail. YOU
> MUST NOT rely on any of our communications as legal advice. Only a licensed
> legal professional may give you advice. Our communications are never
> provided as legal advice, because we are not licensed to provide legal
> advice nor do we possess the knowledge, skills or capacity to provide legal
> advice. We disclaim any and all responsibility related to any action you
> might take based upon our communications and emphasize the need for you to
> never rely on our communications as the basis of any claim or proceeding.
>
> CONFIDENTIALITY
>
> This email and any files transmitted with it are confidential and
> intended solely for the use of the individual or entity to whom they are
> addressed. If you have received this email in error please notify the
> system manager. This message contains confidential information and is
> intended only for the individual(s) named. If you are not the named
> addressee(s) you should not disseminate, distribute or copy this e-mail.
> Please notify the sender immediately by e-mail if you have received this
> e-mail by mistake and delete this e-mail from your system. If you are not
> the intended recipient you are notified that disclosing, copying,
> distributing or taking any action in reliance on the contents of this
> information is strictly prohibited.
>
>
> *Disclaimer*
>
> The information contained in this communication from the sender is
> confidential. It is intended solely for use by the recipient and others
> authorized to receive it. If you are not the recipient, you are hereby
> notified that any disclosure, copying, distribution or taking action in
> relation of the contents of this information is strictly prohibited and may
> be unlawful.
>
> This email has been scanned for viruses and malware, and may have been
> automatically archived by Mimecast, a leader in email security and cyber
> resilience. Mimecast integrates email defenses with brand protection,
> security awareness training, web security, compliance and other essential
> capabilities. Mimecast helps protect large and small organizations from
> malicious activity, human error and technology failure; and to lead the
> movement toward building a more resilient world. To find out more, visit
> our website.
>