[jira] [Comment Edited] (FLINK-16293) Document using plugins in Kubernetes

["Andrey Zagrebin (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-16293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17046733#comment-17046733 ] 

Andrey Zagrebin edited comment on FLINK-16293 at 2/27/20 3:26 PM:
------------------------------------------------------------------

[~nielsbasjes] Thanks for looking into this!

I would say it rather belongs to docker [doc section|https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/docker.html] because it is about how to build a custom docker image. The docker image can be used for many types of deployment, not only native Kubernetes. Now we have some confusion in our examples, different places in docs and Flink repo about docker and Kubernetes related topics because of historical reasons. Therefore, it is a bit difficult to integrate properly the plugin guide into the existing docs. Right now, we are preparing a new FLIP to address this docker topic misalignments and figure out a unified clear approach.

The common [docker entry|https://github.com/apache/flink-docker/blob/master/docker-entrypoint.sh] already has some scripts to setup plugins. In this particular case of native Kubernetes integration, the entry point gets overridden and plugin script gets lost. Therefore, the new idea is to prepare scripts/examples for this kind of standard actions and make it part of base docker image to call in inherited Dockerfile, like you do in your example. The plan is also to rewrite the mentioned general docker image docs with this new approach in mind including general example of plugin installation, hadoop etc. All other places (e.g. Kubernetes integration) should refer to this as a separate if user has to create a custom docker image.

Taking this into account, the addition to the docs, introduced in the current PR, will have to be completely replaced anyways by a reference to the main docker image guide with an example of a simple script call in custom Dockerfile or entry point.

What do you think?


was (Author: azagrebin):
[~nielsbasjes] Thanks for looking into this!

I would say it rather belongs to this [doc section|https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/docker.html] because it is about how to build a custom docker image. The docker image can be used for many types of deployment, not only native Kubernetes. Now we have some confusion in our examples, different places in docs and Flink repo about docker and Kubernetes related topics because of historical reasons. Therefore, it is a bit difficult to integrate properly the plugin guide into the existing docs. Right now, we are preparing a new FLIP to address this docker topic misalignments and figure out a unified clear approach.

The common [docker entry|https://github.com/apache/flink-docker/blob/master/docker-entrypoint.sh] already has some scripts to setup plugins. In this particular case of native Kubernetes integration, the entry point gets overridden and plugin script gets lost. Therefore, the new idea is to prepare scripts/examples for this kind of standard actions and make it part of base docker image to call in inherited Dockerfile, like you do in your example. The plan is also to rewrite the mentioned general docker image docs with this new approach in mind including general example of plugin installation, hadoop etc. All other places (e.g. Kubernetes integration) should refer to this as a separate if user has to create a custom docker image.

Taking this into account, the addition to the docs, introduced in the current PR, will have to be completely replaced anyways by a reference to the main docker image guide with an example of a simple script call in custom Dockerfile or entry point.

What do you think?

> Document using plugins in Kubernetes
> ------------------------------------
>
>                 Key: FLINK-16293
>                 URL: https://issues.apache.org/jira/browse/FLINK-16293
>             Project: Flink
>          Issue Type: Improvement
>          Components: Deployment / Kubernetes, FileSystems
>    Affects Versions: 1.10.0
>            Reporter: Niels Basjes
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> It took me some time to figure out how to enable plugins when running Flink on Kubernetes.
> So I'm writing some documentation to save other people trying the same a lot of time.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)