Flink Kubernetes operator

[santhosh venkat <sa...@gmail.com>]

Hi,
I have a few questions about the Flink OSS Kubernetes operator. Apologies
if they’re basic.

   1. It seems like the auto-scaler logic is triggered as part of the operator
   reconciliation loop
   <https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/AbstractFlinkResourceReconciler.java#L132>.
   Does this mean the auto-scaler is expected to be applied as part of job
   deployment or when the state of any of the watched K8s objects is changed
   over time.
   2. From the implementation, it seems like OSS flink K8s operator changes
   the parallelism of flink-operators in the job depending upon processing
   rate and catchup rate configurations. Just wondering if these parallelism
   changes are preserved across job restarts/deployments or if the parallelism
   configured in the job for different flink operators is used upon restart.
   3. Just wondering how the auto-scaler checks if the kubernetes cluster
   has sufficient capacity before increasing the parallelism of
   flink operators.
   4. Are users expected to use HPA or VPA in conjunction with the
   auto-scaling module in Flink K8s operator. Or are these two unrelated to
   each other?

Thanks.

Re: Flink Kubernetes operator

[Gyula Fóra <gy...@gmail.com>]

Hey!

I think this question should have been sent to the user mailing list
instead of dev :)

In any case:

1: Yes the autoscaler logic is triggered in the reconcile loop, but that
just means that it is triggered every X seconds and only if there is no
other thing to do from the operator like performing the upgrade. This is to
ensure that the autoscaler does not interfere with upgrades or the other
way around.
2: The parallelism changes / overrides are stored in the autoscaler and are
always applied on the FlinkDeployment/SessionJob resource to make sure it
is preserved during restarts/upgrades etc.
3: It doesn't check that there are enough resources before scaling however
when used with in-place scaling the adaptive scheduler will not stop the
previous job until the resources are acquired.
4: You cannot use this autoscaler together with HPA/VPA. So you can either
use the autoscaler (which is much better in our opinion) or use the HPA
with reactive mode but that doesn't work well the kubernetes operator.

In any case I highly recommend to simply try and play around with this
locally, that will answer most of your questions.

Cheers,
Gyula

On Wed, Nov 6, 2024 at 2:11 AM santhosh venkat <sa...@gmail.com>
wrote:

> Hi,
> I have a few questions about the Flink OSS Kubernetes operator. Apologies
> if they’re basic.
>
>    1. It seems like the auto-scaler logic is triggered as part of the
> operator
>    reconciliation loop
>    <
> https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/AbstractFlinkResourceReconciler.java#L132
> >.
>    Does this mean the auto-scaler is expected to be applied as part of job
>    deployment or when the state of any of the watched K8s objects is
> changed
>    over time.
>    2. From the implementation, it seems like OSS flink K8s operator changes
>    the parallelism of flink-operators in the job depending upon processing
>    rate and catchup rate configurations. Just wondering if these
> parallelism
>    changes are preserved across job restarts/deployments or if the
> parallelism
>    configured in the job for different flink operators is used upon
> restart.
>    3. Just wondering how the auto-scaler checks if the kubernetes cluster
>    has sufficient capacity before increasing the parallelism of
>    flink operators.
>    4. Are users expected to use HPA or VPA in conjunction with the
>    auto-scaling module in Flink K8s operator. Or are these two unrelated to
>    each other?
>
> Thanks.
>