You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by James McMahon <js...@gmail.com> on 2024/07/06 18:09:48 UTC
Containerizing, clustering, Registry - how to begin?
I would like to build a containerized, scalable, highly available NiFi
architecture - likely docker, likely on EC2s. I intend to have dev, int,
and prod containerized groups, version controlled through NiFi Registry. I
am trying to understand how NiFi clusters, Registry, and containerization
get used in such a complex architecture.
Does one first containerize each and every NiFi node in the cluster
independently, each and every zookeeper node independently, and then group
them together in NiFi clusters as described in the Apache NiFi Admin Guide,
here:
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#clustering?
There doesn’t seem to be much in there about clustering and containerizing.
Am I thinking about this wrong? Does one abandon traditional nifi
clustering and instead achieve scalability somehow through
dockerized containers running single NiFi node instances, employing a load
balancer in front of the containers to distribute load - so no traditional
NiFi cluster, no zookeeper cluster, no coordination between NiFi nodes?
How well does NiFi Registry function when the underlying NiFi clusters are
containerized?
Has anyone accomplished anything similar? Could you help me understand how
to build out such a complex architecture: NiFi nodes in containers,
zookeeper nodes in containers, all clustered, NiFi Registry lording over
all for version control and process group promotion?
This may be helpful to get started:
https://sandundayananda.medium.com/deploy-apache-nifi-on-docker-with-aws-ec2-instance-and-connect-to-web-interface-3e516e06fe04
But it doesn't speak to clustering or to NiFi Registry in a containerized,
clustered architecture.
Thanks in advance for any thoughts.
Re: Containerizing, clustering, Registry - how to begin?
Posted by Ravi Teja Kondisetty <ko...@gmail.com>.
+1 to this. Would like to know if we can scale out and scale in based on the load.
On 9 Jul 2024, 7:32 PM +0530, Joe Obernberger <jo...@gmail.com>, wrote:
> Hi James - I too would like more information on this. I've been using NiFi in a kubernetes cluster for over a year now using cetic (https://github.com/cetic/helm-nifi) and have it working with NiFi 1.25.0. This works well with a 3 node cluster, and can successfully scale up. Scaling down can work, but (if I understand correctly), you'll need to make sure the queues on the node you are removing are empty, otherwise you'll lose that data. Nifi 2.0.0 looks like it has much better integration with kubernetes with the KubernetesLeaderElectionManager (https://issues.apache.org/jira/browse/NIFI-10757), which appears to take the place of zookeeper, but I've not seen any helm charts or nice ways to deploy a NiFi 2.0.0 cluster yet.
> Anyone?
> -Joe
> On 7/6/2024 2:09 PM, James McMahon wrote:
> > I would like to build a containerized, scalable, highly available NiFi architecture - likely docker, likely on EC2s. I intend to have dev, int, and prod containerized groups, version controlled through NiFi Registry. I am trying to understand how NiFi clusters, Registry, and containerization get used in such a complex architecture.
> >
> > Does one first containerize each and every NiFi node in the cluster independently, each and every zookeeper node independently, and then group them together in NiFi clusters as described in the Apache NiFi Admin Guide, here: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#clustering? There doesn’t seem to be much in there about clustering and containerizing.
> >
> > Am I thinking about this wrong? Does one abandon traditional nifi clustering and instead achieve scalability somehow through dockerized containers running single NiFi node instances, employing a load balancer in front of the containers to distribute load - so no traditional NiFi cluster, no zookeeper cluster, no coordination between NiFi nodes?
> >
> > How well does NiFi Registry function when the underlying NiFi clusters are containerized?
> >
> > Has anyone accomplished anything similar? Could you help me understand how to build out such a complex architecture: NiFi nodes in containers, zookeeper nodes in containers, all clustered, NiFi Registry lording over all for version control and process group promotion?
> >
> > This may be helpful to get started:
> >
> > https://sandundayananda.medium.com/deploy-apache-nifi-on-docker-with-aws-ec2-instance-and-connect-to-web-interface-3e516e06fe04
> >
> > But it doesn't speak to clustering or to NiFi Registry in a containerized, clustered architecture.
> >
> > Thanks in advance for any thoughts.
>
> Virus-free.www.avg.com
Re: Containerizing, clustering, Registry - how to begin?
Posted by Jim Halfpenny via users <us...@nifi.apache.org>.
Hi,
At Stackable we have Kubernetes operators and images for Apache NiFi and Apache ZooKeeper, so if you want avoid the effort of building all of this yourself it’s worth taking a look. There’s an experimental NiFi 2.0 Docker image in our repository if you’re keen to try that out, but that won’t be officially supported until we see a final NiFi 2.0 release.
Kind regards,
Jim
On 9 Jul 2024, at 15:01, Joe Obernberger <jo...@gmail.com> wrote:
>
> Hi James - I too would like more information on this. I've been using NiFi in a kubernetes cluster for over a year now using cetic (https://github.com/cetic/helm-nifi) and have it working with NiFi 1.25.0. This works well with a 3 node cluster, and can successfully scale up. Scaling down can work, but (if I understand correctly), you'll need to make sure the queues on the node you are removing are empty, otherwise you'll lose that data. Nifi 2.0.0 looks like it has much better integration with kubernetes with the KubernetesLeaderElectionManager (https://issues.apache.org/jira/browse/NIFI-10757), which appears to take the place of zookeeper, but I've not seen any helm charts or nice ways to deploy a NiFi 2.0.0 cluster yet.
> Anyone?
>
> -Joe
>
> On 7/6/2024 2:09 PM, James McMahon wrote:
>> I would like to build a containerized, scalable, highly available NiFi architecture - likely docker, likely on EC2s. I intend to have dev, int, and prod containerized groups, version controlled through NiFi Registry. I am trying to understand how NiFi clusters, Registry, and containerization get used in such a complex architecture.
>>
>> Does one first containerize each and every NiFi node in the cluster independently, each and every zookeeper node independently, and then group them together in NiFi clusters as described in the Apache NiFi Admin Guide, here: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#clustering? There doesn’t seem to be much in there about clustering and containerizing.
>>
>> Am I thinking about this wrong? Does one abandon traditional nifi clustering and instead achieve scalability somehow through dockerized containers running single NiFi node instances, employing a load balancer in front of the containers to distribute load - so no traditional NiFi cluster, no zookeeper cluster, no coordination between NiFi nodes?
>>
>> How well does NiFi Registry function when the underlying NiFi clusters are containerized?
>>
>> Has anyone accomplished anything similar? Could you help me understand how to build out such a complex architecture: NiFi nodes in containers, zookeeper nodes in containers, all clustered, NiFi Registry lording over all for version control and process group promotion?
>>
>> This may be helpful to get started:
>>
>> https://sandundayananda.medium.com/deploy-apache-nifi-on-docker-with-aws-ec2-instance-and-connect-to-web-interface-3e516e06fe04
>>
>> But it doesn't speak to clustering or to NiFi Registry in a containerized, clustered architecture.
>>
>> Thanks in advance for any thoughts.
>
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Virus-free.www.avg.com <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> <x-msg://1/#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Re: Containerizing, clustering, Registry - how to begin?
Posted by Joe Obernberger <jo...@gmail.com>.
Hi James - I too would like more information on this. I've been using
NiFi in a kubernetes cluster for over a year now using cetic
(https://github.com/cetic/helm-nifi) and have it working with NiFi
1.25.0. This works well with a 3 node cluster, and can successfully
scale up. Scaling down can work, but (if I understand correctly),
you'll need to make sure the queues on the node you are removing are
empty, otherwise you'll lose that data. Nifi 2.0.0 looks like it has
much better integration with kubernetes with the
KubernetesLeaderElectionManager
(https://issues.apache.org/jira/browse/NIFI-10757), which appears to
take the place of zookeeper, but I've not seen any helm charts or nice
ways to deploy a NiFi 2.0.0 cluster yet.
Anyone?
-Joe
On 7/6/2024 2:09 PM, James McMahon wrote:
>
> I would like to build a containerized, scalable, highly available NiFi
> architecture - likely docker, likely on EC2s. I intend to have dev,
> int, and prod containerized groups, version controlled through NiFi
> Registry. I am trying to understand how NiFi clusters, Registry, and
> containerization get used in such a complex architecture.
>
> Does one first containerize each and every NiFi node in the cluster
> independently, each and every zookeeper node independently, and then
> group them together in NiFi clusters as described in the Apache NiFi
> Admin Guide, here:
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#clustering?
> There doesn’t seem to be much in there about clustering and
> containerizing.
>
>
> Am I thinking about this wrong? Does one abandon traditional nifi
> clustering and instead achieve scalability somehow through
> dockerized containers running single NiFi node instances, employing a
> load balancer in front of the containers to distribute load - so no
> traditional NiFi cluster, no zookeeper cluster, no coordination
> between NiFi nodes?
>
> How well does NiFi Registry function when the underlying NiFi clusters
> are containerized?
>
>
> Has anyone accomplished anything similar? Could you help me understand
> how to build out such a complex architecture: NiFi nodes in
> containers, zookeeper nodes in containers, all clustered, NiFi
> Registry lording over all for version control and process group promotion?
>
> This may be helpful to get started:
>
> https://sandundayananda.medium.com/deploy-apache-nifi-on-docker-with-aws-ec2-instance-and-connect-to-web-interface-3e516e06fe04
>
> But it doesn't speak to clustering or to NiFi Registry in a
> containerized, clustered architecture.
>
> Thanks in advance for any thoughts.
--
This email has been checked for viruses by AVG antivirus software.
www.avg.com
RE: Containerizing, clustering, Registry - how to begin?
Posted by "stephen.hindmarch.bt.com via users" <us...@nifi.apache.org>.
James,
I run my own project at home that does this. Individual containers for each NiFi node, one for Registry, a zookeeper cluster of 3 containers and one container for nginx to proxy front end access. This to me is the Docker way of doing things. This then does look like a traditional NiFi cluster, but the emphasis is on using well designed and implemented images to create the containers, then use a management tool such as Compose, Swarm or Kubernetes to compose those containers into a single cluster.
For my own needs I have added in Kafka and Redis nodes and connected the cluster to a separate Elasticsearch cluster running on a different Docker network. I only use it for developing flow ideas, so it lacks production features such as volume persistence and secure connections. I have seen in the past a few people on this forum mention they have containerised solutions in production. They may wish to share their ideas too.
For what it is worth, I have it all on my personal GitHub account as a public repo.
GitHub - hindmasj/my-nifi-cluster: Quick project to create a NiFi cluster in Docker<https://github.com/hindmasj/my-nifi-cluster>
Regards
Steve Hindmarch
From: James McMahon <js...@gmail.com>
Sent: Saturday, July 6, 2024 7:10 PM
To: users <us...@nifi.apache.org>
Subject: Containerizing, clustering, Registry - how to begin?
I would like to build a containerized, scalable, highly available NiFi architecture - likely docker, likely on EC2s. I intend to have dev, int, and prod containerized groups, version controlled through NiFi Registry. I am trying to understand how NiFi clusters, Registry, and containerization get used in such a complex architecture.
Does one first containerize each and every NiFi node in the cluster independently, each and every zookeeper node independently, and then group them together in NiFi clusters as described in the Apache NiFi Admin Guide, here: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#clustering? There doesn't seem to be much in there about clustering and containerizing.
Am I thinking about this wrong? Does one abandon traditional nifi clustering and instead achieve scalability somehow through dockerized containers running single NiFi node instances, employing a load balancer in front of the containers to distribute load - so no traditional NiFi cluster, no zookeeper cluster, no coordination between NiFi nodes?
How well does NiFi Registry function when the underlying NiFi clusters are containerized?
Has anyone accomplished anything similar? Could you help me understand how to build out such a complex architecture: NiFi nodes in containers, zookeeper nodes in containers, all clustered, NiFi Registry lording over all for version control and process group promotion?
This may be helpful to get started:
https://sandundayananda.medium.com/deploy-apache-nifi-on-docker-with-aws-ec2-instance-and-connect-to-web-interface-3e516e06fe04
But it doesn't speak to clustering or to NiFi Registry in a containerized, clustered architecture.
Thanks in advance for any thoughts.