[jira] [Created] (NIFI-10757) Support Kubernetes as Leader Election Manager and State Provider

["David Handermann (Jira)" <ji...@apache.org>]

David Handermann created NIFI-10757:
---------------------------------------

             Summary: Support Kubernetes as Leader Election Manager and State Provider
                 Key: NIFI-10757
                 URL: https://issues.apache.org/jira/browse/NIFI-10757
             Project: Apache NiFi
          Issue Type: Epic
          Components: Core Framework
            Reporter: David Handermann
            Assignee: David Handermann


[Kubernetes|https://kubernetes.io/] is a common container deployment platform for Apache NiFi, with various efforts available to manage application orchestration.

Current versions of NiFi require ZooKeeper for clustered deployments, leveraging [Curator|https://curator.apache.org/] for the leader election implementation, and using ZookKeeper nodes for providing cluster state information. Kubernetes provides standard resources for leader election and state persistence that could serve as alternatives to ZooKeeper. New component implementations should be developed to enable Kubernetes-based clustered deployments without the need for ZooKeeper.

The {{StateProvider}} interface is defined in the public {{nifi-api}} module, so that component can be provided in an extension NAR. The Kubernetes implementation should leverage [ConfigMaps|https://kubernetes.io/docs/concepts/configuration/configmap/] to persist state based for each component identifier requested. ConfigMap size is limited to 1 MB, corresponding to the internal limit of etcd, which also matches the standard limit for ZooKeeper. The implementation will be configurable through the standard {{state-management.xml}} configuration.

The {{LeaderElectionManager}} interface is defined in {{{}nifi-framework-core{}}}, providing the contract for implementation. With Leader Election being a fundamental framework concern, this interface is not part of the public {{nifi-api}} open for extension. The Kubernetes implementation should incorporate module refactoring to separate the interface from the ZooKeeper and Kubernetes implementations. Several methods on the current ZooKeeper implementation can be abstracted for shared tracking for registration and election operations. The Kubernetes [Lease|https://kubernetes.io/docs/reference/kubernetes-api/cluster-resources/lease-v1/] resource should be used to implement a new Leader Election Manager. The Fabric8 and official Kubernetes Java client libraries also support standard abstractions for implementing the leader election pattern. The implementation approach should evaluate the transitive dependencies for client libraries to determine the best option. The new implementation should provide new settings for NiFi application properties to select Kubernetes instead of ZooKeeper for leader election.

In addition to component implementation, new toolkit features should be developed to support migration between ZooKeeper and Kubernetes. The existing ZooKeeper Migrator supports retrieving state information and sending it to a new ZooKeeper cluster, so a similar approach could be implemented for Kubernetes state information.

Implementation should also include documentation updates describing new properties and configuration examples.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)