You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Zili Chen (Jira)" <ji...@apache.org> on 2020/03/05 01:21:00 UTC

[jira] [Closed] (FLINK-16194) Refactor the Kubernetes decorator design

     [ https://issues.apache.org/jira/browse/FLINK-16194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zili Chen closed FLINK-16194.
-----------------------------
    Resolution: Fixed

master via (reverse order)

dd8ce6c19eefeaf60c085365fe8bea960c1dace8
f6ad9685bcb8365277ffc2fa3fe1d54b2d445a28
10f53290dd07a64013623d7269c9d599dd7c2482
d29628ca6d64581f8627f81e81f05fa457a6b68d
4d8281a3c89514e16a923e20440095e974b9d91d
7f19c9cf14162c0f1b49e0b7a9bd7a0134cc7e3e
a50435d443395141fc526c6584ec9841e5968713
22735e3f3fdafc86b9f3c20b361a3830b30d44e3
8c7a17e6ef7e3da884e377437563d2508193a56d
20d0990f54272039e033f308cbb8a81fd0c7c000
0743b437c764a20b72ba4b14ad1e8f08755c2108
11fa0283d2abf7bfeca1a846a8163174dbfc3080

> Refactor the Kubernetes decorator design
> ----------------------------------------
>
>                 Key: FLINK-16194
>                 URL: https://issues.apache.org/jira/browse/FLINK-16194
>             Project: Flink
>          Issue Type: Improvement
>          Components: Deployment / Kubernetes
>    Affects Versions: 1.10.0
>            Reporter: Canbin Zheng
>            Assignee: Canbin Zheng
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.11.0
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> So far, Flink has made efforts for the native integration of Kubernetes. However, it is always essential to evaluate the existing design and consider alternatives that have better design and are easier to maintain in the long run. We have suffered from some problems while developing new features base on the current code. Here is some of them:
>  # We don’t have a unified monadic-step based orchestrator architecture to construct all the Kubernetes resources.
>  ** There are inconsistencies between the orchestrator architecture that client uses to create the Kubernetes resources, and the orchestrator architecture that the master uses to create Pods; this confuses new contributors, as there is a cognitive burden to understand two architectural philosophies instead of one; for another, maintenance and new feature development become quite challenging.
>  ** Pod construction is done in one step. With the introduction of new features for the Pod, the construction process could become far more complicated, and the functionality of a single class could explode, which hurts code readability, writability, and testability. At the moment, we have encountered such challenges and realized that it is not an easy thing to develop new features related to the Pod.
>  ** The implementations of a specific feature are usually scattered in multiple decoration classes. For example, the current design uses a decoration class chain that contains five Decorator class to mount a configuration file to the Pod. If people would like to introduce other configuration files support, such as Hadoop configuration or Keytab files, they have no choice but to repeat the same tedious and scattered process.
>  # We don’t have dedicated objects or tools for centrally parsing, verifying, and managing the Kubernetes parameters, which has raised some maintenance and inconsistency issues.
>  ** There are many duplicated parsing and validating code, including settings of Image, ImagePullPolicy, ClusterID, ConfDir, Labels, etc. It not only harms readability and testability but also is prone to mistakes. Refer to issue FLINK-16025 for inconsistent parsing of the same parameter.
>  ** The parameters are scattered so that some of the method signatures have to declare many unnecessary input parameters, such as FlinkMasterDeploymentDecorator#createJobManagerContainer.
>  
> For solving these issues, we propose to 
>  # Introduce a unified monadic-step based orchestrator architecture that has a better, cleaner and consistent abstraction for the Kubernetes resources construction process. 
>  # Add some dedicated tools for centrally parsing, verifying, and managing the Kubernetes parameters.
>  
> Refer to the design doc for the details, any feedback is welcome.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)