[jira] [Created] (FLINK-16194) Refactor the Kubernetes resouces construction architecture

["Canbin Zheng (Jira)" <ji...@apache.org>]

Canbin Zheng created FLINK-16194:
------------------------------------

             Summary: Refactor the Kubernetes resouces construction architecture
                 Key: FLINK-16194
                 URL: https://issues.apache.org/jira/browse/FLINK-16194
             Project: Flink
          Issue Type: Improvement
          Components: Deployment / Kubernetes
    Affects Versions: 1.10.0
            Reporter: Canbin Zheng
             Fix For: 1.11.0


So far, Flink has made efforts for the native integration of Kubernetes. However, it is always essential to evaluate the existing design and consider alternatives that have better design and are easier to maintain in the long run. We have suffered from some problems while developing new features base on the current code. Here is some of them:
 # We don’t have a unified monadic-step based orchestrator architecture to construct all the Kubernetes resources.
 ** There are inconsistencies between the orchestrator architecture that client uses to create the Kubernetes resources, and the orchestrator architecture that the master uses to create Pods; this confuses new contributors, as there is a cognitive burden to understand two architectural philosophies instead of one; for another, maintenance and new feature development become quite challenging.
 ** Pod construction is done in one step. With the introduction of new features for the Pod, the construction process could become far more complicated, and the functionality of a single class could explode, which hurts code readability, writability, and testability. At the moment, we have encountered such challenges and realized that it is not an easy thing to develop new features related to the Pod.
 ** The implementations of a specific feature are usually scattered in multiple decoration classes. For example, the current design uses a decoration class chain that contains five Decorator class to mount a configuration file to the Pod. If people would like to introduce other configuration files support, such as Hadoop configuration or Keytab files, they have no choice but to repeat the same tedious and scattered process.
 # We don’t have dedicated objects or tools for centrally parsing, verifying, and managing the Kubernetes parameters, which has raised some maintenance and inconsistency issues.
 ** There are many duplicated parsing and validating code, including settings of Image, ImagePullPolicy, ClusterID, ConfDir, Labels, etc. It not only harms readability and testability but also is prone to mistakes. Refer to issue FLINK-16025 for inconsistent parsing of the same parameter.
 ** The parameters are scattered so that some of the method signatures have to declare many unnecessary input parameters, such as FlinkMasterDeploymentDecorator#createJobManagerContainer.

 

For solving these issues, we propose to 
 # Introduce a unified monadic-step based orchestrator architecture that has a better, cleaner and consistent abstraction for the Kubernetes resources construction process. 
 # Add some dedicated tools for centrally parsing, verifying, and managing the Kubernetes parameters.

 

Refer to the design doc for the details, any feedback is welcome.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)