You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by bat man <ti...@gmail.com> on 2021/07/11 09:27:01 UTC
Flink 1.13 Native Kubernetes - Custom Pod Templates
Hi,
I am running a POC to run flink 1.13 on Native Kubernetes. Per
documentation [1] this supports custom pod templates. Are there any
references for sample pod templates which can be used as a baseline to
build upon. Could not find any references, documentation[1] has one sample
for pod-template.yaml which is for flink-main-container.
1. I would like to specify the task manager/job manager memory
configurations from pod templates.
2. Secondly, how can the metrics be captured, is it possible to use
service-monitor.
[1] -
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/
Thanks,
Hemant
Re: Flink 1.13 Native Kubernetes - Custom Pod Templates
Posted by bat man <ti...@gmail.com>.
Thanks Yang for the information.
Are all the commands applicable with Kubernetes integration like taking
savepoint, starting from savepoint. I see commands from here [1] for
savepoint and on yarn as well, nothing specific to kubernetes.
[1] -
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/state/savepoints/
Thanks,
Hemant
On Mon, Jul 12, 2021 at 9:03 AM Yang Wang <da...@gmail.com> wrote:
> Hi Hemant,
>
> Thanks for trying the native Kubernetes integration and share your
> feedback.
>
> Pod template is a supplementary feature for those advanced features(e.g.
> volume mounts, sidecar container, init container, etc.),
> which are not supported by Flink config options[1].
>
> By default, Flink is using an empty pod template to initialize the
> JobManager/TaskManager pod, then it is decorated internally.
> For example, set the pod resources(mem, cpu), the starting commands, as
> well as the image and etc.
>
> However, not all the fields could be configured via pod template. Like the
> pod resource you mentioned, it needs to be configured
> via Flink config options. I believe you could find the detailed
> information in the documentation[2].
>
> If you want to capture and store the metrics, the prometheus reporter
> should be right direction[3].
>
> [1].
> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/config/#kubernetes
> [2].
> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#fields-overwritten-by-flink
> [3].
> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/metric_reporters/#prometheus
>
> Best,
> Yang
>
>
> bat man <ti...@gmail.com> 于2021年7月11日周日 下午5:27写道:
>
>> Hi,
>>
>> I am running a POC to run flink 1.13 on Native Kubernetes. Per
>> documentation [1] this supports custom pod templates. Are there any
>> references for sample pod templates which can be used as a baseline to
>> build upon. Could not find any references, documentation[1] has one sample
>> for pod-template.yaml which is for flink-main-container.
>>
>> 1. I would like to specify the task manager/job manager memory
>> configurations from pod templates.
>> 2. Secondly, how can the metrics be captured, is it possible to use
>> service-monitor.
>>
>> [1] -
>> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/
>>
>> Thanks,
>> Hemant
>>
>
Re: Flink 1.13 Native Kubernetes - Custom Pod Templates
Posted by Yang Wang <da...@gmail.com>.
>
> Are all the commands applicable with Kubernetes integration like taking
> savepoint, starting from savepoint. I see commands from here [1] for
> savepoint and on yarn as well, nothing specific to kubernetes.
Yes. Taking savepoint, starting from savepoint are also available for
native Kubernetes integration.
> One more follow-up question on the custom pod-templates for JobManager and
> Taskmanager - As you mention pod template is for advanced features so is it
> that in custom template we just need to include the custom feature like if
> I want to include a volume mount or sid-car. I don't have to include any
> flink specific resource, since it is decorated internally, pod will come up
> with the combined resources - flink + custom.
Yes, you are right. But please remember that some fields could be specified
by both flink and custom pod template. The mechanism for resolving
effective field values could be found in the documentation[1].
[1].
https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#fields-overwritten-by-flink
Best,
Yang
bat man <ti...@gmail.com> 于2021年7月13日周二 下午3:27写道:
> Hi Yang,
>
> One more follow-up question on the custom pod-templates for JobManager and
> Taskmanager - As you mention pod template is for advanced features so is it
> that in custom template we just need to include the custom feature like if
> I want to include a volume mount or sid-car. I don't have to include any
> flink specific resource, since it is decorated internally, pod will come up
> with the combined resources - flink + custom.
>
> Let me know if my understanding is correct.
>
> Thanks,
> Hemant
>
> On Mon, Jul 12, 2021 at 9:03 AM Yang Wang <da...@gmail.com> wrote:
>
>> Hi Hemant,
>>
>> Thanks for trying the native Kubernetes integration and share your
>> feedback.
>>
>> Pod template is a supplementary feature for those advanced features(e.g.
>> volume mounts, sidecar container, init container, etc.),
>> which are not supported by Flink config options[1].
>>
>> By default, Flink is using an empty pod template to initialize the
>> JobManager/TaskManager pod, then it is decorated internally.
>> For example, set the pod resources(mem, cpu), the starting commands, as
>> well as the image and etc.
>>
>> However, not all the fields could be configured via pod template. Like
>> the pod resource you mentioned, it needs to be configured
>> via Flink config options. I believe you could find the detailed
>> information in the documentation[2].
>>
>> If you want to capture and store the metrics, the prometheus reporter
>> should be right direction[3].
>>
>> [1].
>> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/config/#kubernetes
>> [2].
>> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#fields-overwritten-by-flink
>> [3].
>> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/metric_reporters/#prometheus
>>
>> Best,
>> Yang
>>
>>
>> bat man <ti...@gmail.com> 于2021年7月11日周日 下午5:27写道:
>>
>>> Hi,
>>>
>>> I am running a POC to run flink 1.13 on Native Kubernetes. Per
>>> documentation [1] this supports custom pod templates. Are there any
>>> references for sample pod templates which can be used as a baseline to
>>> build upon. Could not find any references, documentation[1] has one sample
>>> for pod-template.yaml which is for flink-main-container.
>>>
>>> 1. I would like to specify the task manager/job manager memory
>>> configurations from pod templates.
>>> 2. Secondly, how can the metrics be captured, is it possible to use
>>> service-monitor.
>>>
>>> [1] -
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/
>>>
>>> Thanks,
>>> Hemant
>>>
>>
Re: Flink 1.13 Native Kubernetes - Custom Pod Templates
Posted by bat man <ti...@gmail.com>.
Hi Yang,
One more follow-up question on the custom pod-templates for JobManager and
Taskmanager - As you mention pod template is for advanced features so is it
that in custom template we just need to include the custom feature like if
I want to include a volume mount or sid-car. I don't have to include any
flink specific resource, since it is decorated internally, pod will come up
with the combined resources - flink + custom.
Let me know if my understanding is correct.
Thanks,
Hemant
On Mon, Jul 12, 2021 at 9:03 AM Yang Wang <da...@gmail.com> wrote:
> Hi Hemant,
>
> Thanks for trying the native Kubernetes integration and share your
> feedback.
>
> Pod template is a supplementary feature for those advanced features(e.g.
> volume mounts, sidecar container, init container, etc.),
> which are not supported by Flink config options[1].
>
> By default, Flink is using an empty pod template to initialize the
> JobManager/TaskManager pod, then it is decorated internally.
> For example, set the pod resources(mem, cpu), the starting commands, as
> well as the image and etc.
>
> However, not all the fields could be configured via pod template. Like the
> pod resource you mentioned, it needs to be configured
> via Flink config options. I believe you could find the detailed
> information in the documentation[2].
>
> If you want to capture and store the metrics, the prometheus reporter
> should be right direction[3].
>
> [1].
> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/config/#kubernetes
> [2].
> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#fields-overwritten-by-flink
> [3].
> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/metric_reporters/#prometheus
>
> Best,
> Yang
>
>
> bat man <ti...@gmail.com> 于2021年7月11日周日 下午5:27写道:
>
>> Hi,
>>
>> I am running a POC to run flink 1.13 on Native Kubernetes. Per
>> documentation [1] this supports custom pod templates. Are there any
>> references for sample pod templates which can be used as a baseline to
>> build upon. Could not find any references, documentation[1] has one sample
>> for pod-template.yaml which is for flink-main-container.
>>
>> 1. I would like to specify the task manager/job manager memory
>> configurations from pod templates.
>> 2. Secondly, how can the metrics be captured, is it possible to use
>> service-monitor.
>>
>> [1] -
>> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/
>>
>> Thanks,
>> Hemant
>>
>
Re: Flink 1.13 Native Kubernetes - Custom Pod Templates
Posted by Yang Wang <da...@gmail.com>.
Hi Hemant,
Thanks for trying the native Kubernetes integration and share your feedback.
Pod template is a supplementary feature for those advanced features(e.g.
volume mounts, sidecar container, init container, etc.),
which are not supported by Flink config options[1].
By default, Flink is using an empty pod template to initialize the
JobManager/TaskManager pod, then it is decorated internally.
For example, set the pod resources(mem, cpu), the starting commands, as
well as the image and etc.
However, not all the fields could be configured via pod template. Like the
pod resource you mentioned, it needs to be configured
via Flink config options. I believe you could find the detailed information
in the documentation[2].
If you want to capture and store the metrics, the prometheus reporter
should be right direction[3].
[1].
https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/config/#kubernetes
[2].
https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#fields-overwritten-by-flink
[3].
https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/metric_reporters/#prometheus
Best,
Yang
bat man <ti...@gmail.com> 于2021年7月11日周日 下午5:27写道:
> Hi,
>
> I am running a POC to run flink 1.13 on Native Kubernetes. Per
> documentation [1] this supports custom pod templates. Are there any
> references for sample pod templates which can be used as a baseline to
> build upon. Could not find any references, documentation[1] has one sample
> for pod-template.yaml which is for flink-main-container.
>
> 1. I would like to specify the task manager/job manager memory
> configurations from pod templates.
> 2. Secondly, how can the metrics be captured, is it possible to use
> service-monitor.
>
> [1] -
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/
>
> Thanks,
> Hemant
>