Difference between flink on kubernetes operator vs native kubernetes

[SAMPAD SAHA <sa...@gmail.com>]

I was trying to deploy Flink in Kubernetes environment and came across two things:

1. Kubernetes Flink control plane developed by google and Lyft
     - https://github.com/lyft/flinkk8soperator
     - https://github.com/GoogleCloudPlatform/flink-on-k8s-operator

2. Deploying Kubernetes natively.
    - https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/native_kubernetes.html

I would like to know the basic difference between the underlying working model of both.

I am able to understand the working structure of the first as well as the scope and features.
But for the second, 
             -Is it similar to the first? How the second is different? 
             -How can we handle multi-tenancy in the second? 
             -And since the second is still in the beta stage, what is the roadmap or the features I can expect 
              from the second?

Re: Difference between flink on kubernetes operator vs native kubernetes

[Yang Wang <da...@gmail.com>]

Inline comments for your questions.

Is it similar to the first? How the second is different?

The first leverages the K8s operator to make standalone job cluster running
on K8s easier. However,
the second is more Flink native. We have an embedded K8s client in Flink
and could use "kubernetes-session.sh"/"flink"
to start session/job clusters. The biggest difference is dynamic resource
allocation. All the TaskManager
pods will be allocated/released dynamically.

How can we handle multi-tenancy in the second?

The job cluster(new named with "application" mode) will be supported in
1.11[1]. It could provide better isolation. Each
application could be submitted with specified ServiceAccount, namespace,
resources.

And since the second is still in the beta stage, what is the roadmap or the
> features I can expect from the second?

Native Flink K8s integration is introduced in 1.10 and enriched with many
features in 1.11. After 1.11, the interfaces(
including command line, config options, etc.) will not change easily. And i
think it could used in production for some
cases. However, we still have some advanced features to complete(e.g.
native K8s HA, pod template, volume support, etc.).
And i hope they could be done in the next major release cycle(1.12).

Feel free to share your feedback. BTW, we also have a detailed introduction
"Integrate Flink with Kubernetes natively"
in last Flink Forward. You could find the video[2] here.

[1].
https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/native_kubernetes.html#flink-kubernetes-application

[2]. https://youtu.be/pdFPr_VOWTU


Best,
Yang

SAMPAD SAHA <sa...@gmail.com> 于2020年6月20日周六 上午10:43写道：

> I was trying to deploy Flink in Kubernetes environment and came across two
> things:
>
> 1. Kubernetes Flink control plane developed by google and Lyft
>      - https://github.com/lyft/flinkk8soperator
>      - https://github.com/GoogleCloudPlatform/flink-on-k8s-operator
>
> 2. Deploying Kubernetes natively.
>     -
> https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/native_kubernetes.html
>
> I would like to know the basic difference between the underlying working
> model of both.
>
> I am able to understand the working structure of the first as well as the
> scope and features.
> But for the second,
>              -Is it similar to the first? How the second is different?
>              -How can we handle multi-tenancy in the second?
>              -And since the second is still in the beta stage, what is the
> roadmap or the features I can expect
>               from the second?
>