Re: Operators resource requirements on K8s Flink session cluster

[Michaël Melchiore <ro...@gmail.com>]

Hi Yang,

Thanks for your quick reply.

The Flink K8s documentation distinguishes between standalone and session
deployment mode. I think I will use the latter.
Since my previous mail, I found FLIP-53
<https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management>
which
is precisely the topic of my original question.

So, great progress has been already made to cover my needs. Unfortunately,
I am use DataStreams API which are currently not covered by the initial
implementation. I have asked on the dev mailing list if I could help
bridging this gap.

Regards,

Michaël

Le jeu. 19 déc. 2019 à 04:58, Yang Wang <da...@gmail.com> a écrit :

> Hi Michaël,
>
> Glad to hear that you are going to run Flink workload on Kubernetes.
> AFAIK, we have two
> deployment ways.
> 1. Running Flink standalone session/per-job cluster on K8s. You need to
> calculate how many
> taskmanagers you need and the <memory, cpu> per taskmanager. All the
> taskmanager
> will be started by a K8s deployment. You could find more information
> here[1]. In this mode,
> you could be `kubectl scale` to change the replicas of taskmanager if the
> resources are not
> enough for your job.
> 2. Natively running Flink session/per-job on K8s. The session mode has
> been support in
> master branch and will be released in 1.10. The per-job mode is in
> discussion. No matter
> session or per-job, the taskmanager will be allocated dynamically on
> demand. You could
> use a simple command to start a Flink cluster on K8s. More information
> could be found
> here[2].
>
>
> Best,
> Yang
>
> [1].
> https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/kubernetes.html
> [2].
> https://docs.google.com/document/d/1-jNzqGF6NfZuwVaFICoFQ5HFFXzF5NVIagUZByFMfBY/edit?usp=sharing
>
>
> Michaël Melchiore <ro...@gmail.com> 于2019年12月19日周四 上午1:11写道：
>
>> Hello,
>>
>> I plan to run topologies on a Flink session cluster on Kubernetes.
>> In my topologies, operators will have varying resource requirements in
>> term of CPU and RAM.
>> How can I make these informations available from Flink to Kubernetes so
>> the latter takes it into account to optimize its deployment ?
>>
>> I am trying to achieve something similar to Apache Storm/Trident Resource
>> Aware Scheduler
>> <https://storm.apache.org/releases/2.0.0/Trident-RAS-API.html>.
>>
>> Kind regards,
>>
>> Michaël
>>
>