NIFI-5922 Can individual NiFi Operators run on different containers within Kubernetes

[Romeo Kienzler <ro...@gmail.com>]

Hi,

just read about NIFI-5922 which is very interesting.

But have an architectural question.

Which of the following is true?

1) A complete Nifi execution environment runs in a signle Kubernetes
Container, therefore unable to cross physical cluster boundaries

2) A complete Nifi execution environment runs in a signle Kubernetes POD
but on multiple containers which can therefore cross physical cluster
boundaries

3) Each processor runs in an individual POD or Container, therefore able
to cross physical cluster boundaries

thanks a lot!

- romeo

Re: NIFI-5922 Can individual NiFi Operators run on different containers within Kubernetes

[Erik Anderson <ea...@pobox.com>]

> 1) A complete Nifi execution environment runs in a signle Kubernetes
> Container, therefore unable to cross physical cluster boundaries

If you have high availability requirements like we do in Financial Service, let me let you in on a trick

Have your Kubernetes cluster span data centers. If you loose 1 data center you wont loose your Kubernetes cluster, thus whatever you are running will self-heal on the other data center (and your NiFi is happy as can be).

This works so long as you have low latency between your data centers.

I am assuming 2 kubernetes clusters (1 active, 1 hot standby) would work but you are doubling your resource requirements. This is a bad practice and WILL come back and haunt you.

Erik Anderson
Bloomberg