You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Milan Das <md...@interset.com> on 2018/10/12 21:41:03 UTC
NIFI single node in cluster mode
Hello Nifi team,
Is it possible to run a single NIFI node in cluster mode ? I have this requirement because we will add other nodes soon down line.
I tried that by setting “nifi.cluster.is.node” but and zookeeper setting. But seems it waits ever for election.
Appreciate your thoughts.
Thanks,
Milan Das
Re: NIFI single node in cluster mode
Posted by Milan Das <md...@interset.com>.
Thanks all for the advise.
Found the problem. I was adding two Djava parameters in java.arg.N. I added then two different lines it worked.
Now I see the problem when I list Queue on Success Queye: My flow is simple GenerateFlowFile (success) --> Funnel.
Yes I added all policies at root level to user nifiadmin1. This works when I set the cluster to false.
NIFI version : 1.6.0
Error:
2018-10-14 15:03:21,620 INFO [NiFi Web Server-38] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for nifiadmin1@INTERSET.COM
2018-10-14 15:03:21,621 INFO [NiFi Web Server-38] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[nifiadmin1@INTERSET.COM], groups[] does not have permission to access the requested resource. Unable to view the data for Processor with ID 7312084e-0166-1000-0000-00006ef08dd3. Returning Forbidden response.
2018-10-14 15:03:21,623 INFO [NiFi Web Server-40] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[nifiadmin1@INTERSET.COM], groups[] does not have permission to access the requested resource. Node ip-172-30-1-235.ec2.internal:8443 is unable to fulfill this request due to: Unable to view the data for Processor with ID 7312084e-0166-1000-0000-00006ef08dd3. Contact the system administrator. Returning Forbidden response.
2018-10-14 15:03:21,633 INFO [NiFi Web Server-138] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<ni...@INTERSET.COM><CN=ip-172-30-1-235.ec2.internal, O=Interset, ST=California, C=US>) POST https://ip-172-30-1-235.ec2.internal:8443/nifi-api/flowfile-queues/73121f31-0166-1000-0000-000024972726/listing-requests (source ip: 172.30.1.235)
2018-10-14 15:03:21,633 INFO [NiFi Web Server-138] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for nifiadmin1@
Regards,
Milan Das
Milan Das
Sr. System Architect
email: mdas@interset.com
<https://www.linkedin.com/in/milandas/>
www.interset.com <http://www.interset.com/>
On 10/13/18, 2:39 PM, "Jeff" <jt...@gmail.com> wrote:
Milan,
If you haven't already done so, please take a look at the NiFi Admin
Guide's sections "Securing Zookeeper" [1] and "Kerberizing NiFi’s ZooKeeper
Client" [2], which should help you configure NiFi to use a kerberized
ZooKeeper.
[1]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#securing_zookeeper
[2]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#zk_kerberos_client
On Sat, Oct 13, 2018 at 9:38 AM Milan Das <md...@interset.com> wrote:
> Problem is I am using Kerbrized zookeeper and it is failing to create nifi
> basepath. Even if TGT is getting created Authentication is failing.
>
>
> 2018-10-13 13:33:53,573 INFO [Thread-12] org.apache.zookeeper.Login TGT
> refresh thread started.
> 2018-10-13 13:33:53,576 INFO [Thread-12] org.apache.zookeeper.Login TGT
> valid starting at: Sat Oct 13 13:33:53 UTC 2018
> 2018-10-13 13:33:53,576 INFO [Thread-12] org.apache.zookeeper.Login TGT
> expires: Sun Oct 14 13:33:53 UTC 2018
> 2018-10-13 13:33:53,577 INFO [Thread-12] org.apache.zookeeper.Login TGT
> refresh sleeping until: Sun Oct 14 09:38:53 UTC 2018
> 2018-10-13 13:33:53,577 INFO
> [main-SendThread(ip-172-30-1-132.ec2.internal:2181)]
> o.a.zookeeper.client.ZooKeeperSaslClient Client will use GSSAPI as SASL
> mechanism.
> 2018-10-13 13:33:53,606 INFO [main-EventThread]
> o.a.c.f.state.ConnectionStateManager State change: CONNECTED
> 2018-10-13 13:33:53,616 ERROR
> [main-SendThread(ip-172-30-1-132.ec2.internal:2181)]
> o.a.zookeeper.client.ZooKeeperSaslClient SASL authentication failed using
> login context 'Client'.
> 2018-10-13 13:33:53,723 WARN [main]
> o.a.n.c.l.e.CuratorLeaderElectionManager Unable to determine the Elected
> Leader for role 'Cluster Coordinator' due to
> org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode =
> AuthFailed for /nifi/leaders/Cluster Coordinator; assuming no leader has
> been elected
> 2018-10-13 13:33:53,724 INFO [Curator-Framework-0]
> o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
> 2018-10-13 13:33:53,726 INFO [main]
> o.apache.nifi.controller.FlowController It appears that no Cluster
> Coordinator has been Elected yet. Registering for Cluster Coordinator Role.
>
>
> Thanks,
> Milan Das
>
>
> On 10/12/18, 6:26 PM, "Bryan Bende" <bb...@gmail.com> wrote:
>
> There is also another property for the # of candidates to wait for when
> voting, if it sees the # of candidates first it will short circuit the
> time
> period. So setting the candidates to 1 for a single node cluster should
> start immediately.
>
> On Fri, Oct 12, 2018 at 5:59 PM Jon Logan <jm...@buffalo.edu> wrote:
>
> > It waits for election for a specific period of time, which if I
> recall is
> > fairly high (I think 5 minutes?). If you lower this it'll still wait
> for an
> > election but will complete faster (we do 30 seconds, but you could do
> > lower). There's a property controlling this.
> >
> > On Fri, Oct 12, 2018 at 5:41 PM Milan Das <md...@interset.com> wrote:
> >
> > > Hello Nifi team,
> > >
> > > Is it possible to run a single NIFI node in cluster mode ? I have
> this
> > > requirement because we will add other nodes soon down line.
> > >
> > > I tried that by setting “nifi.cluster.is.node” but and zookeeper
> > setting.
> > > But seems it waits ever for election.
> > >
> > >
> > >
> > > Appreciate your thoughts.
> > >
> > >
> > >
> > > Thanks,
> > >
> > > Milan Das
> > >
> > >
> >
> --
> Sent from Gmail Mobile
>
>
>
>
Re: NIFI single node in cluster mode
Posted by Jeff <jt...@gmail.com>.
Milan,
If you haven't already done so, please take a look at the NiFi Admin
Guide's sections "Securing Zookeeper" [1] and "Kerberizing NiFi’s ZooKeeper
Client" [2], which should help you configure NiFi to use a kerberized
ZooKeeper.
[1]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#securing_zookeeper
[2]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#zk_kerberos_client
On Sat, Oct 13, 2018 at 9:38 AM Milan Das <md...@interset.com> wrote:
> Problem is I am using Kerbrized zookeeper and it is failing to create nifi
> basepath. Even if TGT is getting created Authentication is failing.
>
>
> 2018-10-13 13:33:53,573 INFO [Thread-12] org.apache.zookeeper.Login TGT
> refresh thread started.
> 2018-10-13 13:33:53,576 INFO [Thread-12] org.apache.zookeeper.Login TGT
> valid starting at: Sat Oct 13 13:33:53 UTC 2018
> 2018-10-13 13:33:53,576 INFO [Thread-12] org.apache.zookeeper.Login TGT
> expires: Sun Oct 14 13:33:53 UTC 2018
> 2018-10-13 13:33:53,577 INFO [Thread-12] org.apache.zookeeper.Login TGT
> refresh sleeping until: Sun Oct 14 09:38:53 UTC 2018
> 2018-10-13 13:33:53,577 INFO
> [main-SendThread(ip-172-30-1-132.ec2.internal:2181)]
> o.a.zookeeper.client.ZooKeeperSaslClient Client will use GSSAPI as SASL
> mechanism.
> 2018-10-13 13:33:53,606 INFO [main-EventThread]
> o.a.c.f.state.ConnectionStateManager State change: CONNECTED
> 2018-10-13 13:33:53,616 ERROR
> [main-SendThread(ip-172-30-1-132.ec2.internal:2181)]
> o.a.zookeeper.client.ZooKeeperSaslClient SASL authentication failed using
> login context 'Client'.
> 2018-10-13 13:33:53,723 WARN [main]
> o.a.n.c.l.e.CuratorLeaderElectionManager Unable to determine the Elected
> Leader for role 'Cluster Coordinator' due to
> org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode =
> AuthFailed for /nifi/leaders/Cluster Coordinator; assuming no leader has
> been elected
> 2018-10-13 13:33:53,724 INFO [Curator-Framework-0]
> o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
> 2018-10-13 13:33:53,726 INFO [main]
> o.apache.nifi.controller.FlowController It appears that no Cluster
> Coordinator has been Elected yet. Registering for Cluster Coordinator Role.
>
>
> Thanks,
> Milan Das
>
>
> On 10/12/18, 6:26 PM, "Bryan Bende" <bb...@gmail.com> wrote:
>
> There is also another property for the # of candidates to wait for when
> voting, if it sees the # of candidates first it will short circuit the
> time
> period. So setting the candidates to 1 for a single node cluster should
> start immediately.
>
> On Fri, Oct 12, 2018 at 5:59 PM Jon Logan <jm...@buffalo.edu> wrote:
>
> > It waits for election for a specific period of time, which if I
> recall is
> > fairly high (I think 5 minutes?). If you lower this it'll still wait
> for an
> > election but will complete faster (we do 30 seconds, but you could do
> > lower). There's a property controlling this.
> >
> > On Fri, Oct 12, 2018 at 5:41 PM Milan Das <md...@interset.com> wrote:
> >
> > > Hello Nifi team,
> > >
> > > Is it possible to run a single NIFI node in cluster mode ? I have
> this
> > > requirement because we will add other nodes soon down line.
> > >
> > > I tried that by setting “nifi.cluster.is.node” but and zookeeper
> > setting.
> > > But seems it waits ever for election.
> > >
> > >
> > >
> > > Appreciate your thoughts.
> > >
> > >
> > >
> > > Thanks,
> > >
> > > Milan Das
> > >
> > >
> >
> --
> Sent from Gmail Mobile
>
>
>
>
Re: NIFI single node in cluster mode
Posted by Milan Das <md...@interset.com>.
Problem is I am using Kerbrized zookeeper and it is failing to create nifi basepath. Even if TGT is getting created Authentication is failing.
2018-10-13 13:33:53,573 INFO [Thread-12] org.apache.zookeeper.Login TGT refresh thread started.
2018-10-13 13:33:53,576 INFO [Thread-12] org.apache.zookeeper.Login TGT valid starting at: Sat Oct 13 13:33:53 UTC 2018
2018-10-13 13:33:53,576 INFO [Thread-12] org.apache.zookeeper.Login TGT expires: Sun Oct 14 13:33:53 UTC 2018
2018-10-13 13:33:53,577 INFO [Thread-12] org.apache.zookeeper.Login TGT refresh sleeping until: Sun Oct 14 09:38:53 UTC 2018
2018-10-13 13:33:53,577 INFO [main-SendThread(ip-172-30-1-132.ec2.internal:2181)] o.a.zookeeper.client.ZooKeeperSaslClient Client will use GSSAPI as SASL mechanism.
2018-10-13 13:33:53,606 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2018-10-13 13:33:53,616 ERROR [main-SendThread(ip-172-30-1-132.ec2.internal:2181)] o.a.zookeeper.client.ZooKeeperSaslClient SASL authentication failed using login context 'Client'.
2018-10-13 13:33:53,723 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager Unable to determine the Elected Leader for role 'Cluster Coordinator' due to org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /nifi/leaders/Cluster Coordinator; assuming no leader has been elected
2018-10-13 13:33:53,724 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
2018-10-13 13:33:53,726 INFO [main] o.apache.nifi.controller.FlowController It appears that no Cluster Coordinator has been Elected yet. Registering for Cluster Coordinator Role.
Thanks,
Milan Das
On 10/12/18, 6:26 PM, "Bryan Bende" <bb...@gmail.com> wrote:
There is also another property for the # of candidates to wait for when
voting, if it sees the # of candidates first it will short circuit the time
period. So setting the candidates to 1 for a single node cluster should
start immediately.
On Fri, Oct 12, 2018 at 5:59 PM Jon Logan <jm...@buffalo.edu> wrote:
> It waits for election for a specific period of time, which if I recall is
> fairly high (I think 5 minutes?). If you lower this it'll still wait for an
> election but will complete faster (we do 30 seconds, but you could do
> lower). There's a property controlling this.
>
> On Fri, Oct 12, 2018 at 5:41 PM Milan Das <md...@interset.com> wrote:
>
> > Hello Nifi team,
> >
> > Is it possible to run a single NIFI node in cluster mode ? I have this
> > requirement because we will add other nodes soon down line.
> >
> > I tried that by setting “nifi.cluster.is.node” but and zookeeper
> setting.
> > But seems it waits ever for election.
> >
> >
> >
> > Appreciate your thoughts.
> >
> >
> >
> > Thanks,
> >
> > Milan Das
> >
> >
>
--
Sent from Gmail Mobile
Re: NIFI single node in cluster mode
Posted by Bryan Bende <bb...@gmail.com>.
There is also another property for the # of candidates to wait for when
voting, if it sees the # of candidates first it will short circuit the time
period. So setting the candidates to 1 for a single node cluster should
start immediately.
On Fri, Oct 12, 2018 at 5:59 PM Jon Logan <jm...@buffalo.edu> wrote:
> It waits for election for a specific period of time, which if I recall is
> fairly high (I think 5 minutes?). If you lower this it'll still wait for an
> election but will complete faster (we do 30 seconds, but you could do
> lower). There's a property controlling this.
>
> On Fri, Oct 12, 2018 at 5:41 PM Milan Das <md...@interset.com> wrote:
>
> > Hello Nifi team,
> >
> > Is it possible to run a single NIFI node in cluster mode ? I have this
> > requirement because we will add other nodes soon down line.
> >
> > I tried that by setting “nifi.cluster.is.node” but and zookeeper
> setting.
> > But seems it waits ever for election.
> >
> >
> >
> > Appreciate your thoughts.
> >
> >
> >
> > Thanks,
> >
> > Milan Das
> >
> >
>
--
Sent from Gmail Mobile
Re: NIFI single node in cluster mode
Posted by Jon Logan <jm...@buffalo.edu>.
It waits for election for a specific period of time, which if I recall is
fairly high (I think 5 minutes?). If you lower this it'll still wait for an
election but will complete faster (we do 30 seconds, but you could do
lower). There's a property controlling this.
On Fri, Oct 12, 2018 at 5:41 PM Milan Das <md...@interset.com> wrote:
> Hello Nifi team,
>
> Is it possible to run a single NIFI node in cluster mode ? I have this
> requirement because we will add other nodes soon down line.
>
> I tried that by setting “nifi.cluster.is.node” but and zookeeper setting.
> But seems it waits ever for election.
>
>
>
> Appreciate your thoughts.
>
>
>
> Thanks,
>
> Milan Das
>
>