You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Fuyao Li <fu...@oracle.com> on 2021/04/13 00:03:39 UTC
Conflict in the document - About native Kubernetes per job mode
Hello Community, Yang,
I noticed a conflict in the document for per-job mode support for Kubernetes.
In the doc here [1], it mentions
in a Flink Job Cluster, the available cluster manager (like YARN or Kubernetes) is used to spin up a cluster for each submitted job and this cluster is available to that job only.
It implies per job mode is supported in Kubernetes.
However, in the docs [2] and [3], it clearly points out per-job mode is not supported in Kubernetes.
This is a conflict statement and is kind of misleading. If needed, I can create an MR to delete the statement in [1] for Kubernetes.. It is a small fix.
I also noticed another thing in the video [4] at 25:08. Yang, you are executing a command with -e kubernetes-per-job flag. I tried and found such command is not supported in Flink distribution at all. I noticed the version you are using is 1.11-snapshot during the demo. Are you modifying the source code and generated an internal version of Flink….?
[1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html#flink-job-cluster
[2] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html#per-job-cluster-mode
[3] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html#per-job-cluster-mode
[4] https://www.youtube.com/watch?v=pdFPr_VOWTU&t=833s
Best,
Fuyao
Re: [External] : Re: Conflict in the document - About native
Kubernetes per job mode
Posted by Fuyao Li <fu...@oracle.com>.
Hello Yang,
Please take a look at the PR when you are free.
https://github.com/apache/flink/pull/15602<https://urldefense.com/v3/__https:/github.com/apache/flink/pull/15602__;!!GqivPVa7Brio!PvSWfGFchHWHUDIT-tZWHhlT14WLLgt0o8bsKbxezhdomLWIwR-rHuUJl1Y99Po$>
Should be a simple change. Thanks!
Best,
Fuyao
From: Fuyao Li <fu...@oracle.com>
Date: Tuesday, April 13, 2021 at 19:10
To: Yang Wang <da...@gmail.com>
Cc: user <us...@flink.apache.org>
Subject: Re: [External] : Re: Conflict in the document - About native Kubernetes per job mode
Hello Yang,
I also created a PR for this issue. Please take a look.
Refer to https://github.com/apache/flink/pull/15602<https://urldefense.com/v3/__https:/github.com/apache/flink/pull/15602__;!!GqivPVa7Brio!PvSWfGFchHWHUDIT-tZWHhlT14WLLgt0o8bsKbxezhdomLWIwR-rHuUJl1Y99Po$>
Thanks,
Fuyao
From: Fuyao Li <fu...@oracle.com>
Date: Tuesday, April 13, 2021 at 18:23
To: Yang Wang <da...@gmail.com>
Cc: user <us...@flink.apache.org>
Subject: Re: [External] : Re: Conflict in the document - About native Kubernetes per job mode
Hello Yang,
I tried to create a ticket https://issues.apache.org/jira/browse/FLINK-22264<https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/FLINK-22264__;!!GqivPVa7Brio!JAL6kfYxsKxUWjI6RfOyzKgsII1Qvg0NKUSVfiwMVonKnBNQXm7kTHJekM03uUU$>
I just registered as a user and I can’t find a place to assign the task to myself… Any idea on this jira issue?
Thanks.
Best,
Fuyao
From: Yang Wang <da...@gmail.com>
Date: Tuesday, April 13, 2021 at 03:01
To: Fuyao Li <fu...@oracle.com>
Cc: user <us...@flink.apache.org>, Yan Wang <y....@oracle.com>
Subject: Re: [External] : Re: Conflict in the document - About native Kubernetes per job mode
I think it makes sense to have such a simple fix.
Could you please create a ticket and attach a PR?
Best,
Yang
Fuyao Li <fu...@oracle.com>> 于2021年4月13日周二 下午2:24写道：
Hello Yang,
It is very kind of you to give such a detailed explanation! Thanks for clarification.
For the small document fix I mentioned, what do you think?
Best,
Fuyao
From: Yang Wang <da...@gmail.com>>
Date: Monday, April 12, 2021 at 23:03
To: Fuyao Li <fu...@oracle.com>>
Cc: user <us...@flink.apache.org>>, Yan Wang <y....@oracle.com>>
Subject: [External] : Re: Conflict in the document - About native Kubernetes per job mode
Hi Fuyao,
Currently, Flink only supports perjob mode for Yarn. The standalone job cluster has been replaced with standalone application mode
after FLIP-85[1]. Both standalone Flink on K8s and native K8s integration do not support per-job mode.
In your attached video, it is a PoC implementation for presentation. We have introduced the Kubernetes application mode in release 1.11
and agree to not have a per-job cluster support. The only reason is that it is not very convenient to ship the job graphs, user jars, artifacts in
Kubernetes environment.
[1]. https://cwiki.apache.org/confluence/display/FLINK/FLIP-85+Flink+Application+Mode<https://urldefense.com/v3/__https:/cwiki.apache.org/confluence/display/FLINK/FLIP-85*Flink*Application*Mode__;Kysr!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_Bg9TMlg$>
Best,
Yang
Fuyao Li <fu...@oracle.com>> 于2021年4月13日周二 上午8:10写道：
Hello Community, Yang,
I noticed a conflict in the document for per-job mode support for Kubernetes.
In the doc here [1], it mentions
in a Flink Job Cluster, the available cluster manager (like YARN or Kubernetes) is used to spin up a cluster for each submitted job and this cluster is available to that job only.
It implies per job mode is supported in Kubernetes.
However, in the docs [2] and [3], it clearly points out per-job mode is not supported in Kubernetes.
This is a conflict statement and is kind of misleading. If needed, I can create an MR to delete the statement in [1] for Kubernetes.. It is a small fix.
I also noticed another thing in the video [4] at 25:08. Yang, you are executing a command with -e kubernetes-per-job flag. I tried and found such command is not supported in Flink distribution at all. I noticed the version you are using is 1.11-snapshot during the demo. Are you modifying the source code and generated an internal version of Flink….?
[1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html#flink-job-cluster<https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html*flink-job-cluster__;Iw!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_YCOfjX4$>
[2] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html#per-job-cluster-mode<https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html*per-job-cluster-mode__;Iw!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_-CMjlT8$>
[3] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html#per-job-cluster-mode<https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html*per-job-cluster-mode__;Iw!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_yXjZ_Mk$>
[4] https://www.youtube.com/watch?v=pdFPr_VOWTU&t=833s<https://urldefense.com/v3/__https:/www.youtube.com/watch?v=pdFPr_VOWTU&t=833s__;!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_J8hhHqE$>
Best,
Fuyao
Re: [External] : Re: Conflict in the document - About native
Kubernetes per job mode
Posted by Fuyao Li <fu...@oracle.com>.
Hello Yang,
I also created a PR for this issue. Please take a look.
Refer to https://github.com/apache/flink/pull/15602
Thanks,
Fuyao
From: Fuyao Li <fu...@oracle.com>
Date: Tuesday, April 13, 2021 at 18:23
To: Yang Wang <da...@gmail.com>
Cc: user <us...@flink.apache.org>
Subject: Re: [External] : Re: Conflict in the document - About native Kubernetes per job mode
Hello Yang,
I tried to create a ticket https://issues.apache.org/jira/browse/FLINK-22264<https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/FLINK-22264__;!!GqivPVa7Brio!JAL6kfYxsKxUWjI6RfOyzKgsII1Qvg0NKUSVfiwMVonKnBNQXm7kTHJekM03uUU$>
I just registered as a user and I can’t find a place to assign the task to myself… Any idea on this jira issue?
Thanks.
Best,
Fuyao
From: Yang Wang <da...@gmail.com>
Date: Tuesday, April 13, 2021 at 03:01
To: Fuyao Li <fu...@oracle.com>
Cc: user <us...@flink.apache.org>, Yan Wang <y....@oracle.com>
Subject: Re: [External] : Re: Conflict in the document - About native Kubernetes per job mode
I think it makes sense to have such a simple fix.
Could you please create a ticket and attach a PR?
Best,
Yang
Fuyao Li <fu...@oracle.com>> 于2021年4月13日周二 下午2:24写道：
Hello Yang,
It is very kind of you to give such a detailed explanation! Thanks for clarification.
For the small document fix I mentioned, what do you think?
Best,
Fuyao
From: Yang Wang <da...@gmail.com>>
Date: Monday, April 12, 2021 at 23:03
To: Fuyao Li <fu...@oracle.com>>
Cc: user <us...@flink.apache.org>>, Yan Wang <y....@oracle.com>>
Subject: [External] : Re: Conflict in the document - About native Kubernetes per job mode
Hi Fuyao,
Currently, Flink only supports perjob mode for Yarn. The standalone job cluster has been replaced with standalone application mode
after FLIP-85[1]. Both standalone Flink on K8s and native K8s integration do not support per-job mode.
In your attached video, it is a PoC implementation for presentation. We have introduced the Kubernetes application mode in release 1.11
and agree to not have a per-job cluster support. The only reason is that it is not very convenient to ship the job graphs, user jars, artifacts in
Kubernetes environment.
[1]. https://cwiki.apache.org/confluence/display/FLINK/FLIP-85+Flink+Application+Mode<https://urldefense.com/v3/__https:/cwiki.apache.org/confluence/display/FLINK/FLIP-85*Flink*Application*Mode__;Kysr!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_Bg9TMlg$>
Best,
Yang
Fuyao Li <fu...@oracle.com>> 于2021年4月13日周二 上午8:10写道：
Hello Community, Yang,
I noticed a conflict in the document for per-job mode support for Kubernetes.
In the doc here [1], it mentions
in a Flink Job Cluster, the available cluster manager (like YARN or Kubernetes) is used to spin up a cluster for each submitted job and this cluster is available to that job only.
It implies per job mode is supported in Kubernetes.
However, in the docs [2] and [3], it clearly points out per-job mode is not supported in Kubernetes.
This is a conflict statement and is kind of misleading. If needed, I can create an MR to delete the statement in [1] for Kubernetes.. It is a small fix.
I also noticed another thing in the video [4] at 25:08. Yang, you are executing a command with -e kubernetes-per-job flag. I tried and found such command is not supported in Flink distribution at all. I noticed the version you are using is 1.11-snapshot during the demo. Are you modifying the source code and generated an internal version of Flink….?
[1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html#flink-job-cluster<https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html*flink-job-cluster__;Iw!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_YCOfjX4$>
[2] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html#per-job-cluster-mode<https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html*per-job-cluster-mode__;Iw!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_-CMjlT8$>
[3] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html#per-job-cluster-mode<https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html*per-job-cluster-mode__;Iw!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_yXjZ_Mk$>
[4] https://www.youtube.com/watch?v=pdFPr_VOWTU&t=833s<https://urldefense.com/v3/__https:/www.youtube.com/watch?v=pdFPr_VOWTU&t=833s__;!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_J8hhHqE$>
Best,
Fuyao
Re: [External] : Re: Conflict in the document - About native
Kubernetes per job mode
Posted by Fuyao Li <fu...@oracle.com>.
Hello Yang,
I tried to create a ticket https://issues.apache.org/jira/browse/FLINK-22264
I just registered as a user and I can’t find a place to assign the task to myself… Any idea on this jira issue?
Thanks.
Best,
Fuyao
From: Yang Wang <da...@gmail.com>
Date: Tuesday, April 13, 2021 at 03:01
To: Fuyao Li <fu...@oracle.com>
Cc: user <us...@flink.apache.org>, Yan Wang <y....@oracle.com>
Subject: Re: [External] : Re: Conflict in the document - About native Kubernetes per job mode
I think it makes sense to have such a simple fix.
Could you please create a ticket and attach a PR?
Best,
Yang
Fuyao Li <fu...@oracle.com>> 于2021年4月13日周二 下午2:24写道：
Hello Yang,
It is very kind of you to give such a detailed explanation! Thanks for clarification.
For the small document fix I mentioned, what do you think?
Best,
Fuyao
From: Yang Wang <da...@gmail.com>>
Date: Monday, April 12, 2021 at 23:03
To: Fuyao Li <fu...@oracle.com>>
Cc: user <us...@flink.apache.org>>, Yan Wang <y....@oracle.com>>
Subject: [External] : Re: Conflict in the document - About native Kubernetes per job mode
Hi Fuyao,
Currently, Flink only supports perjob mode for Yarn. The standalone job cluster has been replaced with standalone application mode
after FLIP-85[1]. Both standalone Flink on K8s and native K8s integration do not support per-job mode.
In your attached video, it is a PoC implementation for presentation. We have introduced the Kubernetes application mode in release 1.11
and agree to not have a per-job cluster support. The only reason is that it is not very convenient to ship the job graphs, user jars, artifacts in
Kubernetes environment.
[1]. https://cwiki.apache.org/confluence/display/FLINK/FLIP-85+Flink+Application+Mode<https://urldefense.com/v3/__https:/cwiki.apache.org/confluence/display/FLINK/FLIP-85*Flink*Application*Mode__;Kysr!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_Bg9TMlg$>
Best,
Yang
Fuyao Li <fu...@oracle.com>> 于2021年4月13日周二 上午8:10写道：
Hello Community, Yang,
I noticed a conflict in the document for per-job mode support for Kubernetes.
In the doc here [1], it mentions
in a Flink Job Cluster, the available cluster manager (like YARN or Kubernetes) is used to spin up a cluster for each submitted job and this cluster is available to that job only.
It implies per job mode is supported in Kubernetes.
However, in the docs [2] and [3], it clearly points out per-job mode is not supported in Kubernetes.
This is a conflict statement and is kind of misleading. If needed, I can create an MR to delete the statement in [1] for Kubernetes.. It is a small fix.
I also noticed another thing in the video [4] at 25:08. Yang, you are executing a command with -e kubernetes-per-job flag. I tried and found such command is not supported in Flink distribution at all. I noticed the version you are using is 1.11-snapshot during the demo. Are you modifying the source code and generated an internal version of Flink….?
[1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html#flink-job-cluster<https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html*flink-job-cluster__;Iw!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_YCOfjX4$>
[2] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html#per-job-cluster-mode<https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html*per-job-cluster-mode__;Iw!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_-CMjlT8$>
[3] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html#per-job-cluster-mode<https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html*per-job-cluster-mode__;Iw!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_yXjZ_Mk$>
[4] https://www.youtube.com/watch?v=pdFPr_VOWTU&t=833s<https://urldefense.com/v3/__https:/www.youtube.com/watch?v=pdFPr_VOWTU&t=833s__;!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_J8hhHqE$>
Best,
Fuyao
Re: [External] : Re: Conflict in the document - About native
Kubernetes per job mode
Posted by Yang Wang <da...@gmail.com>.
I think it makes sense to have such a simple fix.
Could you please create a ticket and attach a PR?
Best,
Yang
Fuyao Li <fu...@oracle.com> 于2021年4月13日周二 下午2:24写道：
> Hello Yang,
>
>
>
> It is very kind of you to give such a detailed explanation! Thanks for
> clarification.
>
>
>
> For the small document fix I mentioned, what do you think?
>
>
>
> Best,
>
> Fuyao
>
>
>
> *From: *Yang Wang <da...@gmail.com>
> *Date: *Monday, April 12, 2021 at 23:03
> *To: *Fuyao Li <fu...@oracle.com>
> *Cc: *user <us...@flink.apache.org>, Yan Wang <y....@oracle.com>
> *Subject: *[External] : Re: Conflict in the document - About native
> Kubernetes per job mode
>
> Hi Fuyao,
>
>
>
> Currently, Flink only supports perjob mode for Yarn. The standalone job
> cluster has been replaced with standalone application mode
>
> after FLIP-85[1]. Both standalone Flink on K8s and native K8s integration
> do not support per-job mode.
>
>
>
> In your attached video, it is a PoC implementation for presentation. We
> have introduced the Kubernetes application mode in release 1.11
>
> and agree to not have a per-job cluster support. The only reason is that
> it is not very convenient to ship the job graphs, user jars, artifacts in
>
> Kubernetes environment.
>
>
>
> [1].
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-85+Flink+Application+Mode
> <https://urldefense.com/v3/__https:/cwiki.apache.org/confluence/display/FLINK/FLIP-85*Flink*Application*Mode__;Kysr!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_Bg9TMlg$>
>
>
>
> Best,
>
> Yang
>
>
>
> Fuyao Li <fu...@oracle.com> 于2021年4月13日周二 上午8:10写道：
>
> Hello Community, Yang,
>
>
>
> I noticed a conflict in the document for per-job mode support for
> Kubernetes.
>
> In the doc here [1], it mentions
>
> in a Flink Job Cluster, the available cluster manager (like YARN or
> Kubernetes) is used to spin up a cluster for each submitted job and this
> cluster is available to that job only.
>
> It implies per job mode is supported in Kubernetes.
>
>
>
> However, in the docs [2] and [3], it clearly points out per-job mode is
> not supported in Kubernetes.
>
>
>
> *This is a conflict statement and is kind of misleading. If needed, I can
> create an MR to delete the statement in [1] for Kubernetes.. It is a small
> fix. *
>
>
>
> I also noticed another thing in the video [4] at 25:08. Yang, you are
> executing a command with -e kubernetes-per-job flag. I tried and found such
> command is not supported in Flink distribution at all. I noticed the
> version you are using is 1.11-snapshot during the demo. Are you modifying
> the source code and generated an internal version of Flink….?
>
>
>
>
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html#flink-job-cluster
> <https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html*flink-job-cluster__;Iw!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_YCOfjX4$>
>
> [2]
> https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html#per-job-cluster-mode
> <https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html*per-job-cluster-mode__;Iw!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_-CMjlT8$>
>
> [3]
> https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html#per-job-cluster-mode
> <https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html*per-job-cluster-mode__;Iw!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_yXjZ_Mk$>
>
> [4] https://www.youtube.com/watch?v=pdFPr_VOWTU&t=833s
> <https://urldefense.com/v3/__https:/www.youtube.com/watch?v=pdFPr_VOWTU&t=833s__;!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_J8hhHqE$>
>
>
>
> Best,
>
> Fuyao
>
>
Re: [External] : Re: Conflict in the document - About native
Kubernetes per job mode
Posted by Fuyao Li <fu...@oracle.com>.
Hello Yang,
It is very kind of you to give such a detailed explanation! Thanks for clarification.
For the small document fix I mentioned, what do you think?
Best,
Fuyao
From: Yang Wang <da...@gmail.com>
Date: Monday, April 12, 2021 at 23:03
To: Fuyao Li <fu...@oracle.com>
Cc: user <us...@flink.apache.org>, Yan Wang <y....@oracle.com>
Subject: [External] : Re: Conflict in the document - About native Kubernetes per job mode
Hi Fuyao,
Currently, Flink only supports perjob mode for Yarn. The standalone job cluster has been replaced with standalone application mode
after FLIP-85[1]. Both standalone Flink on K8s and native K8s integration do not support per-job mode.
In your attached video, it is a PoC implementation for presentation. We have introduced the Kubernetes application mode in release 1.11
and agree to not have a per-job cluster support. The only reason is that it is not very convenient to ship the job graphs, user jars, artifacts in
Kubernetes environment.
[1]. https://cwiki.apache.org/confluence/display/FLINK/FLIP-85+Flink+Application+Mode<https://urldefense.com/v3/__https:/cwiki.apache.org/confluence/display/FLINK/FLIP-85*Flink*Application*Mode__;Kysr!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_Bg9TMlg$>
Best,
Yang
Fuyao Li <fu...@oracle.com>> 于2021年4月13日周二 上午8:10写道：
Hello Community, Yang,
I noticed a conflict in the document for per-job mode support for Kubernetes.
In the doc here [1], it mentions
in a Flink Job Cluster, the available cluster manager (like YARN or Kubernetes) is used to spin up a cluster for each submitted job and this cluster is available to that job only.
It implies per job mode is supported in Kubernetes.
However, in the docs [2] and [3], it clearly points out per-job mode is not supported in Kubernetes.
This is a conflict statement and is kind of misleading. If needed, I can create an MR to delete the statement in [1] for Kubernetes.. It is a small fix.
I also noticed another thing in the video [4] at 25:08. Yang, you are executing a command with -e kubernetes-per-job flag. I tried and found such command is not supported in Flink distribution at all. I noticed the version you are using is 1.11-snapshot during the demo. Are you modifying the source code and generated an internal version of Flink….?
[1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html#flink-job-cluster<https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html*flink-job-cluster__;Iw!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_YCOfjX4$>
[2] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html#per-job-cluster-mode<https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html*per-job-cluster-mode__;Iw!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_-CMjlT8$>
[3] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html#per-job-cluster-mode<https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html*per-job-cluster-mode__;Iw!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_yXjZ_Mk$>
[4] https://www.youtube.com/watch?v=pdFPr_VOWTU&t=833s<https://urldefense.com/v3/__https:/www.youtube.com/watch?v=pdFPr_VOWTU&t=833s__;!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_J8hhHqE$>
Best,
Fuyao
Re: Conflict in the document - About native Kubernetes per job mode
Posted by Yang Wang <da...@gmail.com>.
Hi Fuyao,
Currently, Flink only supports perjob mode for Yarn. The standalone job
cluster has been replaced with standalone application mode
after FLIP-85[1]. Both standalone Flink on K8s and native K8s integration
do not support per-job mode.
In your attached video, it is a PoC implementation for presentation. We
have introduced the Kubernetes application mode in release 1.11
and agree to not have a per-job cluster support. The only reason is that it
is not very convenient to ship the job graphs, user jars, artifacts in
Kubernetes environment.
[1].
https://cwiki.apache.org/confluence/display/FLINK/FLIP-85+Flink+Application+Mode
Best,
Yang
Fuyao Li <fu...@oracle.com> 于2021年4月13日周二 上午8:10写道：
> Hello Community, Yang,
>
>
>
> I noticed a conflict in the document for per-job mode support for
> Kubernetes.
>
> In the doc here [1], it mentions
>
> in a Flink Job Cluster, the available cluster manager (like YARN or
> Kubernetes) is used to spin up a cluster for each submitted job and this
> cluster is available to that job only.
>
> It implies per job mode is supported in Kubernetes.
>
>
>
> However, in the docs [2] and [3], it clearly points out per-job mode is
> not supported in Kubernetes.
>
>
>
> *This is a conflict statement and is kind of misleading. If needed, I can
> create an MR to delete the statement in [1] for Kubernetes.. It is a small
> fix. *
>
>
>
> I also noticed another thing in the video [4] at 25:08. Yang, you are
> executing a command with -e kubernetes-per-job flag. I tried and found such
> command is not supported in Flink distribution at all. I noticed the
> version you are using is 1.11-snapshot during the demo. Are you modifying
> the source code and generated an internal version of Flink….?
>
>
>
>
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html#flink-job-cluster
>
> [2]
> https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html#per-job-cluster-mode
>
> [3]
> https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html#per-job-cluster-mode
>
> [4] https://www.youtube.com/watch?v=pdFPr_VOWTU&t=833s
>
>
>
> Best,
>
> Fuyao
>