You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by gaurav kulkarni <ku...@yahoo.com> on 2021/04/20 00:57:33 UTC
CRD compatible with native and standalone mode
Hi,
I plan to create a flink K8s operator which supports standalone mode, and and switch to native mode sometime later. I was wondering what are some of the approaches to ensure that CRD is compatible with both native and standalone mode?
Thanks
Re: CRD compatible with native and standalone mode
Posted by Yang Wang <da...@gmail.com>.
Exactly.
I think most of the fields could be shared by standalone and native mode.
Best,
Yang
gaurav kulkarni <ku...@yahoo.com> 于2021年4月21日周三 上午10:17写道：
> Thanks a lot for the response, folks! I appreciate it. I plan to use
> native mode in future mostly for the resource management it plans to offer.
> Let me go through the links provided.
>
> @Yang Wang "Since the CR is defined in yaml[2], native and standalone
> could have some dedicated fields. And you could easily parse them in your
> K8s operator.": Do you mean having fields that are needed for both native
> and standalone mode in the CRD (probably making them optional in the CRD)
> and each operator type (standalone/native) using fields that are relevant
> for it?
>
> Thanks,
> Gaurav
>
>
> On Tuesday, April 20, 2021, 08:23:34 AM PDT, Austin Cawley-Edwards <
> austin.cawley@gmail.com> wrote:
>
>
> Hi Gaurav,
>
> I think the name "Native Kubernetes" is a bit misleading – this just means
> that you can use the Flink CLI/ scripts to run Flink applications on
> Kubernetes without using the Kubernetes APIs/ kubectl directly. What
> features are you looking to use in the native mode?
>
> I think it would be difficult to use this directly inside an operator, but
> keeping "feature parity" with it is a good goal for your CRDs. Since CRDs
> are essentially just a new API, the design approach should be user/
> feature-first. By feature parity, I mean taking currently supported "Native
> Kubernetes" functionality as the feature list for your CRDs, for example:
> * Allowing Secrets to be mounted as files and environment variables [1]
> * Allowing templating of the JobManager and TaskManager Pods (containers,
> etc.) [2]
> * Easy use of built-in plugins [3]
> * etc.
>
> Other "Native Kubernetes" "features", like RBAC and logs, will come "out
> of the box" by defining Custom Resources.
>
> Yang's resources are a great place to start, though I'd suggest defining
> your API spec within the CRD explicitly[4], which will be clearer for your
> users and will allow for schema validation by other tools.
>
> Best,
> Austin
>
>
> [1]:
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#using-secrets
> [2]:
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#pod-template
> [3]:
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#using-plugins
> [4]:
> https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#specifying-a-structural-schema
>
> On Tue, Apr 20, 2021 at 2:13 AM Yang Wang <da...@gmail.com> wrote:
>
> I think the compatibility depends on you. For example, you could have the
> same
> CustomResourceDefinition for standalone and native Flink applications.
> They could
> look like this[1].
>
> Since the CR is defined in yaml[2], native and standalone could have some
> dedicated fields.
> And you could easily parse them in your K8s operator.
>
> [1].
> https://github.com/wangyang0918/flink-native-k8s-operator/blob/master/deploy/crd.yaml
> [2].
> https://github.com/wangyang0918/flink-native-k8s-operator/blob/master/deploy/cr.yaml
>
>
> Best,
> Yang
>
> gaurav kulkarni <ku...@yahoo.com> 于2021年4月20日周二 上午8:57写道：
>
> Hi,
>
> I plan to create a flink K8s operator which supports standalone mode, and
> and switch to native mode sometime later. I was wondering what are some of
> the approaches to ensure that CRD is compatible with both native and
> standalone mode?
>
> Thanks
>
>
Re: CRD compatible with native and standalone mode
Posted by gaurav kulkarni <ku...@yahoo.com>.
Thanks a lot for the response, folks! I appreciate it. I plan to use native mode in future mostly for the resource management it plans to offer. Let me go through the links provided.
@Yang Wang "Since the CR is defined in yaml[2], native and standalone could have some dedicated fields. And you could easily parse them in your K8s operator.": Do you mean having fields that are needed for both native and standalone mode in the CRD (probably making them optional in the CRD) and each operator type (standalone/native) using fields that are relevant for it?
Thanks,Gaurav
On Tuesday, April 20, 2021, 08:23:34 AM PDT, Austin Cawley-Edwards <au...@gmail.com> wrote:
Hi Gaurav,
I think the name "Native Kubernetes" is a bit misleading – this just means that you can use the Flink CLI/ scripts to run Flink applications on Kubernetes without using the Kubernetes APIs/ kubectl directly. What features are you looking to use in the native mode?
I think it would be difficult to use this directly inside an operator, but keeping "feature parity" with it is a good goal for your CRDs. Since CRDs are essentially just a new API, the design approach should be user/ feature-first. By feature parity, I mean taking currently supported "Native Kubernetes" functionality as the feature list for your CRDs, for example:* Allowing Secrets to be mounted as files and environment variables [1]* Allowing templating of the JobManager and TaskManager Pods (containers, etc.) [2]* Easy use of built-in plugins [3]* etc.
Other "Native Kubernetes" "features", like RBAC and logs, will come "out of the box" by defining Custom Resources.
Yang's resources are a great place to start, though I'd suggest defining your API spec within the CRD explicitly[4], which will be clearer for your users and will allow for schema validation by other tools.
Best,Austin
[1]: https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#using-secrets[2]: https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#pod-template[3]: https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#using-plugins[4]: https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#specifying-a-structural-schema
On Tue, Apr 20, 2021 at 2:13 AM Yang Wang <da...@gmail.com> wrote:
I think the compatibility depends on you. For example, you could have the sameCustomResourceDefinition for standalone and native Flink applications. They could
look like this[1].
Since the CR is defined in yaml[2], native and standalone could have some dedicated fields.And you could easily parse them in your K8s operator.
[1]. https://github.com/wangyang0918/flink-native-k8s-operator/blob/master/deploy/crd.yaml[2]. https://github.com/wangyang0918/flink-native-k8s-operator/blob/master/deploy/cr.yaml
Best,Yang
gaurav kulkarni <ku...@yahoo.com> 于2021年4月20日周二 上午8:57写道：
Hi,
I plan to create a flink K8s operator which supports standalone mode, and and switch to native mode sometime later. I was wondering what are some of the approaches to ensure that CRD is compatible with both native and standalone mode?
Thanks
Re: CRD compatible with native and standalone mode
Posted by Austin Cawley-Edwards <au...@gmail.com>.
Hi Gaurav,
I think the name "Native Kubernetes" is a bit misleading – this just means
that you can use the Flink CLI/ scripts to run Flink applications on
Kubernetes without using the Kubernetes APIs/ kubectl directly. What
features are you looking to use in the native mode?
I think it would be difficult to use this directly inside an operator, but
keeping "feature parity" with it is a good goal for your CRDs. Since CRDs
are essentially just a new API, the design approach should be user/
feature-first. By feature parity, I mean taking currently supported "Native
Kubernetes" functionality as the feature list for your CRDs, for example:
* Allowing Secrets to be mounted as files and environment variables [1]
* Allowing templating of the JobManager and TaskManager Pods (containers,
etc.) [2]
* Easy use of built-in plugins [3]
* etc.
Other "Native Kubernetes" "features", like RBAC and logs, will come "out of
the box" by defining Custom Resources.
Yang's resources are a great place to start, though I'd suggest defining
your API spec within the CRD explicitly[4], which will be clearer for your
users and will allow for schema validation by other tools.
Best,
Austin
[1]:
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#using-secrets
[2]:
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#pod-template
[3]:
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#using-plugins
[4]:
https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#specifying-a-structural-schema
On Tue, Apr 20, 2021 at 2:13 AM Yang Wang <da...@gmail.com> wrote:
> I think the compatibility depends on you. For example, you could have the
> same
> CustomResourceDefinition for standalone and native Flink applications.
> They could
> look like this[1].
>
> Since the CR is defined in yaml[2], native and standalone could have some
> dedicated fields.
> And you could easily parse them in your K8s operator.
>
> [1].
> https://github.com/wangyang0918/flink-native-k8s-operator/blob/master/deploy/crd.yaml
> [2].
> https://github.com/wangyang0918/flink-native-k8s-operator/blob/master/deploy/cr.yaml
>
>
> Best,
> Yang
>
> gaurav kulkarni <ku...@yahoo.com> 于2021年4月20日周二 上午8:57写道：
>
>> Hi,
>>
>> I plan to create a flink K8s operator which supports standalone mode, and
>> and switch to native mode sometime later. I was wondering what are some of
>> the approaches to ensure that CRD is compatible with both native and
>> standalone mode?
>>
>> Thanks
>>
>
Re: CRD compatible with native and standalone mode
Posted by Yang Wang <da...@gmail.com>.
I think the compatibility depends on you. For example, you could have the
same
CustomResourceDefinition for standalone and native Flink applications. They
could
look like this[1].
Since the CR is defined in yaml[2], native and standalone could have some
dedicated fields.
And you could easily parse them in your K8s operator.
[1].
https://github.com/wangyang0918/flink-native-k8s-operator/blob/master/deploy/crd.yaml
[2].
https://github.com/wangyang0918/flink-native-k8s-operator/blob/master/deploy/cr.yaml
Best,
Yang
gaurav kulkarni <ku...@yahoo.com> 于2021年4月20日周二 上午8:57写道：
> Hi,
>
> I plan to create a flink K8s operator which supports standalone mode, and
> and switch to native mode sometime later. I was wondering what are some of
> the approaches to ensure that CRD is compatible with both native and
> standalone mode?
>
> Thanks
>