You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@nifi.apache.org by e-...@gmx.fr on 2024/09/30 11:38:19 UTC

cluster nifi hosted in kubernetes


Hello all,



Until now we hosted our clusters nifi from VM installed.



We want now to create clusters nifi to kubernetes to have the possibility to
scale up and down more easier than just with VM.



Could you tell me it is possible with kubernetes and have we got some
documentations ?



I saw some works "Improve cluster configuration for dynamic scaling" in jira
https://issues.apache.org/jira/browse/NIFI-5443 and release note about
kubernetes

  * [[NIFI-10975](https://issues.apache.org/jira/browse/NIFI-10975)] - Add Kubernetes Leader Election Manager and State Provider



Thanks for helps if you have some informations for me.



Regards



Minh

Re: cluster nifi hosted in kubernetes

Posted by e-...@gmx.fr.

Hi David,

Thanks for reponse. Yes, I've already read this article from exceptionFactory.

I will check with my kubernetes team.

Thanks

**Envoy e:** lundi 30 septembre 2024 a 20:39  
**De:**  "David Handermann" <ex...@apache.org>  
**À :** users@nifi.apache.org  
**Objet:**  Re: cluster nifi hosted in kubernetes

Hi Minh,  

There are various approaches from vendors and open source projects  
around running NiFi on Kubernetes.  

The implementation work for NIFI-10975 improved framework support for  
clustering on Kubernetes, specifically enabling leader election using  
Kubernetes Leases instead of Apache ZooKeeper.  

The following post provides some of the technical details behind that  
implementation:  

<https://exceptionfactory.com/posts/2024/08/10/bringing-kubernetes-clustering-
to-apache-nifi/>  

Deploying on Kubernetes requires a number of additional infrastructure  
considerations when it comes to security and scalability.  

Regards,  
David Handermann  

On Mon, Sep 30, 2024 at 6:38 AM <e-...@gmx.fr> wrote:  
>  
>  
> Hello all,  
>  
> Until now we hosted our clusters nifi from VM installed.  
>  
> We want now to create clusters nifi to kubernetes to have the possibility to
> scale up and down more easier than just with VM.  
>  
> Could you tell me it is possible with kubernetes and have we got some
> documentations ?  
>  
> I saw some works "Improve cluster configuration for dynamic scaling" in jira
> <https://issues.apache.org/jira/browse/NIFI-5443> and release note about
> kubernetes  
>  
> [NIFI-10975] - Add Kubernetes Leader Election Manager and State Provider  
>  
>  
> Thanks for helps if you have some informations for me.  
>  
> Regards  
>  
> Minh

Re: cluster nifi hosted in kubernetes

Posted by David Handermann <ex...@apache.org>.

Hi Minh,

There are various approaches from vendors and open source projects
around running NiFi on Kubernetes.

The implementation work for NIFI-10975 improved framework support for
clustering on Kubernetes, specifically enabling leader election using
Kubernetes Leases instead of Apache ZooKeeper.

The following post provides some of the technical details behind that
implementation:

https://exceptionfactory.com/posts/2024/08/10/bringing-kubernetes-clustering-to-apache-nifi/

Deploying on Kubernetes requires a number of additional infrastructure
considerations when it comes to security and scalability.

Regards,
David Handermann

On Mon, Sep 30, 2024 at 6:38 AM <e-...@gmx.fr> wrote:
>
>
> Hello all,
>
> Until now we hosted our clusters nifi from VM installed.
>
> We want now to create clusters nifi to kubernetes to have the possibility to scale up and down more easier than just with VM.
>
> Could you tell me it is possible with kubernetes and have we got some documentations ?
>
> I saw some works "Improve cluster configuration for dynamic scaling" in jira https://issues.apache.org/jira/browse/NIFI-5443 and release note about kubernetes
>
> [NIFI-10975] - Add Kubernetes Leader Election Manager and State Provider
>
>
> Thanks for helps if you have some informations for me.
>
> Regards
>
> Minh