You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Bryan Bende <bb...@gmail.com> on 2017/12/20 18:53:57 UTC
Re: NiFi Clustering
Hello,
Each node must have its own repositories, you can't point multiple
nodes at the same repositories.
The repositories for a given node could be on local disk, or on a
mounted network drive, but local disk will likely have the best
performance.
Thanks,
Bryan
On Wed, Dec 20, 2017 at 1:25 PM, Karthik Kothareddy (karthikk) [CONT -
Type 2] <ka...@micron.com> wrote:
> All,
>
>
>
> We are trying to setup a new NiFi (1.4.0) cluster with 3 nodes, and trying
> to decide on how should we proceed with the repository structure. Should we
> be pointing the same repositories for all nodes (shared filers accessible by
> all 3 nodes) or give separate local repositories for all three nodes and let
> each node has its own data copy. Is there any recommendation for this for
> better performance and throughput?
>
>
>
> Thanks for your time.
>
>
>
> -Karthik
RE: [EXT] Re: NiFi Clustering
Posted by "Karthik Kothareddy (karthikk) [CONT - Type 2]" <ka...@micron.com>.
Thanks Bryan , we will use separate repos.
-Karthik
-----Original Message-----
From: Bryan Bende [mailto:bbende@gmail.com]
Sent: Wednesday, December 20, 2017 5:09 PM
To: users@nifi.apache.org
Subject: Re: [EXT] Re: NiFi Clustering
Karthik,
That cannot be done, none of the repositories are designed to have multiple NiFi instances writing to them.
Each node in a NiFi cluster has its own set of flow files, content, and provenance, thus the separate repos.
-Bryan
On Wed, Dec 20, 2017 at 6:59 PM, Karthik Kothareddy (karthikk) [CONT - Type 2] <ka...@micron.com> wrote:
> Joe,
>
>
>
> Sorry for the confusion, I meant to say, like a single network mounted
> drive accessible to all nodes and all nodes pointing to one “content_repository”
> directory in that mount. Can it be done?
>
>
>
> -Karthik
>
>
>
> From: Joe Witt [mailto:joe.witt@gmail.com]
> Sent: Wednesday, December 20, 2017 4:49 PM
> To: users@nifi.apache.org
> Subject: RE: [EXT] Re: NiFi Clustering
>
>
>
> karthik
>
>
>
> Terminology wise you cannot share a repository across nodes. So that
> is a big con :)
>
>
>
> But if you mean can i share something like a network mounted drive by
> giving each node its own mount pointa for its repos then sure that
> works fine. In such a case you need to have strong iops provisioning,
> you need to realize there will be higher roundtrip times for rw ops.
> But in general this can be managed.
>
>
>
> Thanks
>
> joe
>
>
>
> On Dec 20, 2017 6:45 PM, "Karthik Kothareddy (karthikk) [CONT - Type 2]"
> <ka...@micron.com> wrote:
>
> Bryan ,
>
> Thanks for your response , We are going with individual repositories.
> Can you explain what are the cons of having a shared content
> repository between multiple nodes ? I may be asking a dumb question
> but we are trying to understand if shared filers increase the throughput of the system.
>
> Thanks
> Karthik
>
> -----Original Message-----
> From: Bryan Bende [mailto:bbende@gmail.com]
> Sent: Wednesday, December 20, 2017 11:54 AM
> To: users@nifi.apache.org
> Subject: [EXT] Re: NiFi Clustering
>
> Hello,
>
> Each node must have its own repositories, you can't point multiple
> nodes at the same repositories.
>
> The repositories for a given node could be on local disk, or on a
> mounted network drive, but local disk will likely have the best performance.
>
> Thanks,
>
> Bryan
>
>
> On Wed, Dec 20, 2017 at 1:25 PM, Karthik Kothareddy (karthikk) [CONT -
> Type 2] <ka...@micron.com> wrote:
>> All,
>>
>>
>>
>> We are trying to setup a new NiFi (1.4.0) cluster with 3 nodes, and
>> trying to decide on how should we proceed with the repository
>> structure. Should we be pointing the same repositories for all nodes
>> (shared filers accessible by all 3 nodes) or give separate local
>> repositories for all three nodes and let each node has its own data
>> copy. Is there any recommendation for this for better performance and
>> throughput?
>>
>>
>>
>> Thanks for your time.
>>
>>
>>
>> -Karthik
Re: [EXT] Re: NiFi Clustering
Posted by Bryan Bende <bb...@gmail.com>.
Karthik,
That cannot be done, none of the repositories are designed to have
multiple NiFi instances writing to them.
Each node in a NiFi cluster has its own set of flow files, content,
and provenance, thus the separate repos.
-Bryan
On Wed, Dec 20, 2017 at 6:59 PM, Karthik Kothareddy (karthikk) [CONT -
Type 2] <ka...@micron.com> wrote:
> Joe,
>
>
>
> Sorry for the confusion, I meant to say, like a single network mounted drive
> accessible to all nodes and all nodes pointing to one “content_repository”
> directory in that mount. Can it be done?
>
>
>
> -Karthik
>
>
>
> From: Joe Witt [mailto:joe.witt@gmail.com]
> Sent: Wednesday, December 20, 2017 4:49 PM
> To: users@nifi.apache.org
> Subject: RE: [EXT] Re: NiFi Clustering
>
>
>
> karthik
>
>
>
> Terminology wise you cannot share a repository across nodes. So that is a
> big con :)
>
>
>
> But if you mean can i share something like a network mounted drive by giving
> each node its own mount pointa for its repos then sure that works fine. In
> such a case you need to have strong iops provisioning, you need to realize
> there will be higher roundtrip times for rw ops. But in general this can be
> managed.
>
>
>
> Thanks
>
> joe
>
>
>
> On Dec 20, 2017 6:45 PM, "Karthik Kothareddy (karthikk) [CONT - Type 2]"
> <ka...@micron.com> wrote:
>
> Bryan ,
>
> Thanks for your response , We are going with individual repositories. Can
> you explain what are the cons of having a shared content repository between
> multiple nodes ? I may be asking a dumb question but we are trying to
> understand if shared filers increase the throughput of the system.
>
> Thanks
> Karthik
>
> -----Original Message-----
> From: Bryan Bende [mailto:bbende@gmail.com]
> Sent: Wednesday, December 20, 2017 11:54 AM
> To: users@nifi.apache.org
> Subject: [EXT] Re: NiFi Clustering
>
> Hello,
>
> Each node must have its own repositories, you can't point multiple nodes at
> the same repositories.
>
> The repositories for a given node could be on local disk, or on a mounted
> network drive, but local disk will likely have the best performance.
>
> Thanks,
>
> Bryan
>
>
> On Wed, Dec 20, 2017 at 1:25 PM, Karthik Kothareddy (karthikk) [CONT - Type
> 2] <ka...@micron.com> wrote:
>> All,
>>
>>
>>
>> We are trying to setup a new NiFi (1.4.0) cluster with 3 nodes, and
>> trying to decide on how should we proceed with the repository
>> structure. Should we be pointing the same repositories for all nodes
>> (shared filers accessible by all 3 nodes) or give separate local
>> repositories for all three nodes and let each node has its own data
>> copy. Is there any recommendation for this for better performance and
>> throughput?
>>
>>
>>
>> Thanks for your time.
>>
>>
>>
>> -Karthik
RE: [EXT] Re: NiFi Clustering
Posted by "Karthik Kothareddy (karthikk) [CONT - Type 2]" <ka...@micron.com>.
Joe,
Sorry for the confusion, I meant to say, like a single network mounted drive accessible to all nodes and all nodes pointing to one “content_repository” directory in that mount. Can it be done?
-Karthik
From: Joe Witt [mailto:joe.witt@gmail.com]
Sent: Wednesday, December 20, 2017 4:49 PM
To: users@nifi.apache.org
Subject: RE: [EXT] Re: NiFi Clustering
karthik
Terminology wise you cannot share a repository across nodes. So that is a big con :)
But if you mean can i share something like a network mounted drive by giving each node its own mount pointa for its repos then sure that works fine. In such a case you need to have strong iops provisioning, you need to realize there will be higher roundtrip times for rw ops. But in general this can be managed.
Thanks
joe
On Dec 20, 2017 6:45 PM, "Karthik Kothareddy (karthikk) [CONT - Type 2]" <ka...@micron.com>> wrote:
Bryan ,
Thanks for your response , We are going with individual repositories. Can you explain what are the cons of having a shared content repository between multiple nodes ? I may be asking a dumb question but we are trying to understand if shared filers increase the throughput of the system.
Thanks
Karthik
-----Original Message-----
From: Bryan Bende [mailto:bbende@gmail.com<ma...@gmail.com>]
Sent: Wednesday, December 20, 2017 11:54 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: [EXT] Re: NiFi Clustering
Hello,
Each node must have its own repositories, you can't point multiple nodes at the same repositories.
The repositories for a given node could be on local disk, or on a mounted network drive, but local disk will likely have the best performance.
Thanks,
Bryan
On Wed, Dec 20, 2017 at 1:25 PM, Karthik Kothareddy (karthikk) [CONT - Type 2] <ka...@micron.com>> wrote:
> All,
>
>
>
> We are trying to setup a new NiFi (1.4.0) cluster with 3 nodes, and
> trying to decide on how should we proceed with the repository
> structure. Should we be pointing the same repositories for all nodes
> (shared filers accessible by all 3 nodes) or give separate local
> repositories for all three nodes and let each node has its own data
> copy. Is there any recommendation for this for better performance and throughput?
>
>
>
> Thanks for your time.
>
>
>
> -Karthik
RE: [EXT] Re: NiFi Clustering
Posted by Joe Witt <jo...@gmail.com>.
karthik
Terminology wise you cannot share a repository across nodes. So that is a
big con :)
But if you mean can i share something like a network mounted drive by
giving each node its own mount pointa for its repos then sure that works
fine. In such a case you need to have strong iops provisioning, you need
to realize there will be higher roundtrip times for rw ops. But in general
this can be managed.
Thanks
joe
On Dec 20, 2017 6:45 PM, "Karthik Kothareddy (karthikk) [CONT - Type 2]" <
karthikk@micron.com> wrote:
> Bryan ,
>
> Thanks for your response , We are going with individual repositories. Can
> you explain what are the cons of having a shared content repository between
> multiple nodes ? I may be asking a dumb question but we are trying to
> understand if shared filers increase the throughput of the system.
>
> Thanks
> Karthik
>
> -----Original Message-----
> From: Bryan Bende [mailto:bbende@gmail.com]
> Sent: Wednesday, December 20, 2017 11:54 AM
> To: users@nifi.apache.org
> Subject: [EXT] Re: NiFi Clustering
>
> Hello,
>
> Each node must have its own repositories, you can't point multiple nodes
> at the same repositories.
>
> The repositories for a given node could be on local disk, or on a mounted
> network drive, but local disk will likely have the best performance.
>
> Thanks,
>
> Bryan
>
>
> On Wed, Dec 20, 2017 at 1:25 PM, Karthik Kothareddy (karthikk) [CONT -
> Type 2] <ka...@micron.com> wrote:
> > All,
> >
> >
> >
> > We are trying to setup a new NiFi (1.4.0) cluster with 3 nodes, and
> > trying to decide on how should we proceed with the repository
> > structure. Should we be pointing the same repositories for all nodes
> > (shared filers accessible by all 3 nodes) or give separate local
> > repositories for all three nodes and let each node has its own data
> > copy. Is there any recommendation for this for better performance and
> throughput?
> >
> >
> >
> > Thanks for your time.
> >
> >
> >
> > -Karthik
>
RE: [EXT] Re: NiFi Clustering
Posted by "Karthik Kothareddy (karthikk) [CONT - Type 2]" <ka...@micron.com>.
Bryan ,
Thanks for your response , We are going with individual repositories. Can you explain what are the cons of having a shared content repository between multiple nodes ? I may be asking a dumb question but we are trying to understand if shared filers increase the throughput of the system.
Thanks
Karthik
-----Original Message-----
From: Bryan Bende [mailto:bbende@gmail.com]
Sent: Wednesday, December 20, 2017 11:54 AM
To: users@nifi.apache.org
Subject: [EXT] Re: NiFi Clustering
Hello,
Each node must have its own repositories, you can't point multiple nodes at the same repositories.
The repositories for a given node could be on local disk, or on a mounted network drive, but local disk will likely have the best performance.
Thanks,
Bryan
On Wed, Dec 20, 2017 at 1:25 PM, Karthik Kothareddy (karthikk) [CONT - Type 2] <ka...@micron.com> wrote:
> All,
>
>
>
> We are trying to setup a new NiFi (1.4.0) cluster with 3 nodes, and
> trying to decide on how should we proceed with the repository
> structure. Should we be pointing the same repositories for all nodes
> (shared filers accessible by all 3 nodes) or give separate local
> repositories for all three nodes and let each node has its own data
> copy. Is there any recommendation for this for better performance and throughput?
>
>
>
> Thanks for your time.
>
>
>
> -Karthik