You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by dan young <da...@gmail.com> on 2020/08/05 18:02:08 UTC
cluster stick in "Attempted to register Leader Election for role
'Cluster Coordinator' but this role is already registered"
Hello,
Running nifi 1.11.4, 3 X secure cluster mode and have enabled
kerberos/sasl, upon trying to startup the cluster, they seem to get stuck
in :
2020-08-05 17:10:18,907 WARN [main] o.a.nifi.controller.StandardFlowService
There is currently no Cluster Coordinator. This often happens upon restart
of NiFi
when running an embedded ZooKeeper. Will register this node to become the
active Cluster Coordinator and will attempt to connect to cluster again
2020-08-05 17:10:18,907 INFO [main]
o.a.n.c.l.e.CuratorLeaderElectionManager
CuratorLeaderElectionManager[stopped=false] Attempted to register Leader
Election
for role 'Cluster Coordinator' but this role is already registered
I've checked zookeeper and I can see that the /nifi znode has been created,
although empty, and the ACL seem to look correct
zk: nifi1-5.X.net:2181(CONNECTED) 3] getAcl /nifi
'sasl,'nifi@X.NET
: cdrwa
'world,'anyone
: r
relevant Nifi config settings
nifi.properties:
nifi.zookeeper.auth.type=sasl
nifi.zookeeper.kerberos.removeHostFromPrincipal=true
nifi.zookeeper.kerberos.removeRealmFromPrincipal=false
# kerberos #
nifi.kerberos.krb5.file=/etc/krb5.conf
# kerberos service principal #
nifi.kerberos.service.principal=nifi@X.NET
nifi.kerberos.service.keytab.location=/opt/nifi/conf/nifi.keytab
state-management.xml
<cluster-provider>
<id>zk-provider</id>
<class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
<property name="Root Node">/nifi</property>
<property name="Session Timeout">30 seconds</property>
<property name="Access Control">CreatorOnly</property>
<property name="Connect String">X:2181,Y:2181,Z:2181</property>
</cluster-provider>
KRB5_TRACE=/dev/stdout kinit -k -t /opt/nifi/conf/nifi.keytab nifi@X.NET
...
...
klist
Ticket cache: FILE:/tmp/krb5cc_2004
Default principal: nifi@X.NET
Valid starting Expires Service principal
08/05/2020 17:57:02 08/06/2020 03:57:02 krbtgt/X.NET@X.NET
renew until 08/06/2020 17:57:02
As a side note, secure NiFi was working fine before the kerberos bit, I've
been beating my head against the wall with it for the day, but the
kerberos/zookeeper stuff seems to be working now....
do we need to have Server-Server zookeeper auth working for this?
Appreciate any insight....
Regards,
Dano
Re: cluster stick in "Attempted to register Leader Election for role
'Cluster Coordinator' but this role is already registered"
Posted by dan young <da...@gmail.com>.
Hello Bryan,
Same issue. I have a jaas.conf and config in the bootstrap.cof
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/opt/nifi/conf/nifi.keytab"
storeKey=true
useTicketCache=false
principal="nifi@X.net <ni...@lookercorp.net>";
};
java.arg.16=-Djavax.security.auth.useSubjectCredsOnly=true
# Zookeeper 3.5 now includes an Admin Server that starts on port 8080,
since NiFi is already using that port disable by default.
# Please see
https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_adminserver_config
for
configuration options.
java.arg.17=-Dzookeeper.admin.enableServer=false
java.arg.18=-Djava.security.auth.login.config=/opt/nifi/conf/jaas.conf
java.arg.19=-Dsun.security.krb5.debug=true
I just noticed that the realm here is in lowercase, let me change
that.....maybe that's an issue
ReplyForward
On Wed, Aug 5, 2020 at 1:12 PM Bryan Bende <bb...@gmail.com> wrote:
> I don't see how this would relate to the problem, but shouldn't the ACL be
> set to "Creator" when Sasl/Kerberos is setup correctly?
>
> In addition to the nifi configs you showed, you would also need a jaas
> conf file specified in bootstrap.conf and in that file you would need the
> jaas entry for the ZK client.
>
> On Wed, Aug 5, 2020 at 3:02 PM dan young <da...@gmail.com> wrote:
>
>> Hello Mark,
>>
>> Attached is a dump from one of the nodes....I replaced the domain related
>> entries with X/x. I'm not sure if it's relevant or not, but I did notice
>> that in the log there's entries "Looking for keys for nifi@x.net" the x
>> (domain) is lowercase whereas in the keytab file it's uppercase X. Also
>> not sure if the Found unsupported keytype (1) is meaningful. Not that when
>> I delete the znode in zookeeper=, at least the initial znode is created
>> /nifi, but we never see the other typical suspect, i.e Coordinator,
>> Primary, etc...
>>
>> Seems to be something stuck in Curator???
>>
>> Regards.
>>
>> Dano
>>
>> On Wed, Aug 5, 2020 at 12:20 PM Mark Payne <ma...@hotmail.com> wrote:
>>
>>> Dan,
>>>
>>> Can you grab a thread dump and provide that? Specifically, the “main”
>>> thread is the important one with startup. The note that the role is already
>>> registered is normal. It probably could be changed to a DEBUG level,
>>> really. It should not be concerning. A thread dump, though, would show us
>>> exactly where it’s at.
>>>
>>> Thanks
>>> -Mark
>>>
>>>
>>> On Aug 5, 2020, at 2:02 PM, dan young <da...@gmail.com> wrote:
>>>
>>> Hello,
>>> Running nifi 1.11.4, 3 X secure cluster mode and have enabled
>>> kerberos/sasl, upon trying to startup the cluster, they seem to get stuck
>>> in :
>>>
>>> 2020-08-05 17:10:18,907 WARN [main]
>>> o.a.nifi.controller.StandardFlowService There is currently no Cluster
>>> Coordinator. This often happens upon restart of NiFi
>>> when running an embedded ZooKeeper. Will register this node to become
>>> the active Cluster Coordinator and will attempt to connect to cluster again
>>> 2020-08-05 17:10:18,907 INFO [main]
>>> o.a.n.c.l.e.CuratorLeaderElectionManager
>>> CuratorLeaderElectionManager[stopped=false] Attempted to register Leader
>>> Election
>>> for role 'Cluster Coordinator' but this role is already registered
>>>
>>>
>>>
>>> I've checked zookeeper and I can see that the /nifi znode has been
>>> created, although empty, and the ACL seem to look correct
>>> zk: nifi1-5.X.net:2181 <http://nifi1-5.x.net:2181>(CONNECTED) 3] getAcl
>>> /nifi
>>> 'sasl,'nifi@X.NET
>>> : cdrwa
>>> 'world,'anyone
>>> : r
>>>
>>>
>>> relevant Nifi config settings
>>>
>>> nifi.properties:
>>>
>>> nifi.zookeeper.auth.type=sasl
>>> nifi.zookeeper.kerberos.removeHostFromPrincipal=true
>>> nifi.zookeeper.kerberos.removeRealmFromPrincipal=false
>>>
>>> # kerberos #
>>> nifi.kerberos.krb5.file=/etc/krb5.conf
>>>
>>> # kerberos service principal #
>>> nifi.kerberos.service.principal=nifi@X.NET
>>> nifi.kerberos.service.keytab.location=/opt/nifi/conf/nifi.keytab
>>>
>>>
>>> state-management.xml
>>> <cluster-provider>
>>> <id>zk-provider</id>
>>>
>>> <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
>>> <property name="Root Node">/nifi</property>
>>> <property name="Session Timeout">30 seconds</property>
>>> <property name="Access Control">CreatorOnly</property>
>>> <property name="Connect String">X:2181,Y:2181,Z:2181</property>
>>> </cluster-provider>
>>>
>>>
>>>
>>> KRB5_TRACE=/dev/stdout kinit -k -t /opt/nifi/conf/nifi.keytab nifi@X.NET
>>> ...
>>> ...
>>>
>>> klist
>>> Ticket cache: FILE:/tmp/krb5cc_2004
>>> Default principal: nifi@X.NET
>>>
>>> Valid starting Expires Service principal
>>> 08/05/2020 17:57:02 08/06/2020 03:57:02 krbtgt/X.NET@X.NET
>>> renew until 08/06/2020 17:57:02
>>>
>>>
>>>
>>>
>>> As a side note, secure NiFi was working fine before the kerberos bit,
>>> I've been beating my head against the wall with it for the day, but the
>>> kerberos/zookeeper stuff seems to be working now....
>>> do we need to have Server-Server zookeeper auth working for this?
>>>
>>>
>>> Appreciate any insight....
>>>
>>> Regards,
>>>
>>> Dano
>>>
>>>
>>>
Re: cluster stick in "Attempted to register Leader Election for role
'Cluster Coordinator' but this role is already registered"
Posted by dan young <da...@gmail.com>.
I'll make that change, was going off the comments in the
state-management.xml
-Open
-CreatorOnly
Let me try Creator...
On Wed, Aug 5, 2020 at 1:12 PM Bryan Bende <bb...@gmail.com> wrote:
> I don't see how this would relate to the problem, but shouldn't the ACL be
> set to "Creator" when Sasl/Kerberos is setup correctly?
>
> In addition to the nifi configs you showed, you would also need a jaas
> conf file specified in bootstrap.conf and in that file you would need the
> jaas entry for the ZK client.
>
> On Wed, Aug 5, 2020 at 3:02 PM dan young <da...@gmail.com> wrote:
>
>> Hello Mark,
>>
>> Attached is a dump from one of the nodes....I replaced the domain related
>> entries with X/x. I'm not sure if it's relevant or not, but I did notice
>> that in the log there's entries "Looking for keys for nifi@x.net" the x
>> (domain) is lowercase whereas in the keytab file it's uppercase X. Also
>> not sure if the Found unsupported keytype (1) is meaningful. Not that when
>> I delete the znode in zookeeper=, at least the initial znode is created
>> /nifi, but we never see the other typical suspect, i.e Coordinator,
>> Primary, etc...
>>
>> Seems to be something stuck in Curator???
>>
>> Regards.
>>
>> Dano
>>
>> On Wed, Aug 5, 2020 at 12:20 PM Mark Payne <ma...@hotmail.com> wrote:
>>
>>> Dan,
>>>
>>> Can you grab a thread dump and provide that? Specifically, the “main”
>>> thread is the important one with startup. The note that the role is already
>>> registered is normal. It probably could be changed to a DEBUG level,
>>> really. It should not be concerning. A thread dump, though, would show us
>>> exactly where it’s at.
>>>
>>> Thanks
>>> -Mark
>>>
>>>
>>> On Aug 5, 2020, at 2:02 PM, dan young <da...@gmail.com> wrote:
>>>
>>> Hello,
>>> Running nifi 1.11.4, 3 X secure cluster mode and have enabled
>>> kerberos/sasl, upon trying to startup the cluster, they seem to get stuck
>>> in :
>>>
>>> 2020-08-05 17:10:18,907 WARN [main]
>>> o.a.nifi.controller.StandardFlowService There is currently no Cluster
>>> Coordinator. This often happens upon restart of NiFi
>>> when running an embedded ZooKeeper. Will register this node to become
>>> the active Cluster Coordinator and will attempt to connect to cluster again
>>> 2020-08-05 17:10:18,907 INFO [main]
>>> o.a.n.c.l.e.CuratorLeaderElectionManager
>>> CuratorLeaderElectionManager[stopped=false] Attempted to register Leader
>>> Election
>>> for role 'Cluster Coordinator' but this role is already registered
>>>
>>>
>>>
>>> I've checked zookeeper and I can see that the /nifi znode has been
>>> created, although empty, and the ACL seem to look correct
>>> zk: nifi1-5.X.net:2181 <http://nifi1-5.x.net:2181>(CONNECTED) 3] getAcl
>>> /nifi
>>> 'sasl,'nifi@X.NET
>>> : cdrwa
>>> 'world,'anyone
>>> : r
>>>
>>>
>>> relevant Nifi config settings
>>>
>>> nifi.properties:
>>>
>>> nifi.zookeeper.auth.type=sasl
>>> nifi.zookeeper.kerberos.removeHostFromPrincipal=true
>>> nifi.zookeeper.kerberos.removeRealmFromPrincipal=false
>>>
>>> # kerberos #
>>> nifi.kerberos.krb5.file=/etc/krb5.conf
>>>
>>> # kerberos service principal #
>>> nifi.kerberos.service.principal=nifi@X.NET
>>> nifi.kerberos.service.keytab.location=/opt/nifi/conf/nifi.keytab
>>>
>>>
>>> state-management.xml
>>> <cluster-provider>
>>> <id>zk-provider</id>
>>>
>>> <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
>>> <property name="Root Node">/nifi</property>
>>> <property name="Session Timeout">30 seconds</property>
>>> <property name="Access Control">CreatorOnly</property>
>>> <property name="Connect String">X:2181,Y:2181,Z:2181</property>
>>> </cluster-provider>
>>>
>>>
>>>
>>> KRB5_TRACE=/dev/stdout kinit -k -t /opt/nifi/conf/nifi.keytab nifi@X.NET
>>> ...
>>> ...
>>>
>>> klist
>>> Ticket cache: FILE:/tmp/krb5cc_2004
>>> Default principal: nifi@X.NET
>>>
>>> Valid starting Expires Service principal
>>> 08/05/2020 17:57:02 08/06/2020 03:57:02 krbtgt/X.NET@X.NET
>>> renew until 08/06/2020 17:57:02
>>>
>>>
>>>
>>>
>>> As a side note, secure NiFi was working fine before the kerberos bit,
>>> I've been beating my head against the wall with it for the day, but the
>>> kerberos/zookeeper stuff seems to be working now....
>>> do we need to have Server-Server zookeeper auth working for this?
>>>
>>>
>>> Appreciate any insight....
>>>
>>> Regards,
>>>
>>> Dano
>>>
>>>
>>>
Re: cluster stick in "Attempted to register Leader Election for role
'Cluster Coordinator' but this role is already registered"
Posted by Andy LoPresto <al...@apache.org>.
Dan,
Thanks for reporting this. Case-sensitivity in these kinds of things is important but it also seems like low-hanging fruit for us to at least detect & alert on when errors occur. “Failed to connect to external service X with principal nifi@x.net <ma...@x.net> …. Did you mean NIFI@X.NET <ma...@X.NET>?” Or even potentially trying to do case-conversion internally as a fallback.
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Aug 5, 2020, at 2:50 PM, dan young <da...@gmail.com> wrote:
>
> On a related note, I noticed that the ACL are getting set, but also for each znode under the /nifi, the Read ACL for world is being set. Is there a way to have nifi only set with the sasl?
>
> zk: nifi1-5.X.net:2181(CONNECTED) 12] getAcl /nifi
> 'sasl,'nifi@X.NET <ma...@X.NET>
> : cdrwa
> 'world,'anyone
> : r
>
> On Wed, Aug 5, 2020 at 1:56 PM Mark Payne <markap14@hotmail.com <ma...@hotmail.com>> wrote:
> No worries, thanks for following up and letting us know!
>
> Thanks
> -Mark
>
>
>> On Aug 5, 2020, at 3:42 PM, dan young <danoyoung@gmail.com <ma...@gmail.com>> wrote:
>>
>> Hello,
>>
>> Sorry for all the noise...dooh....was due to the realm in the jaas.conf being lowercase...i'm a knucklehead...
>>
>> Dano
>>
>> On Wed, Aug 5, 2020 at 1:12 PM Bryan Bende <bbende@gmail.com <ma...@gmail.com>> wrote:
>> I don't see how this would relate to the problem, but shouldn't the ACL be set to "Creator" when Sasl/Kerberos is setup correctly?
>>
>> In addition to the nifi configs you showed, you would also need a jaas conf file specified in bootstrap.conf and in that file you would need the jaas entry for the ZK client.
>>
>> On Wed, Aug 5, 2020 at 3:02 PM dan young <danoyoung@gmail.com <ma...@gmail.com>> wrote:
>> Hello Mark,
>>
>> Attached is a dump from one of the nodes....I replaced the domain related entries with X/x. I'm not sure if it's relevant or not, but I did notice that in the log there's entries "Looking for keys for nifi@x.net <ma...@x.net>" the x (domain) is lowercase whereas in the keytab file it's uppercase X. Also not sure if the Found unsupported keytype (1) is meaningful. Not that when I delete the znode in zookeeper=, at least the initial znode is created /nifi, but we never see the other typical suspect, i.e Coordinator, Primary, etc...
>>
>> Seems to be something stuck in Curator???
>>
>> Regards.
>>
>> Dano
>>
>> On Wed, Aug 5, 2020 at 12:20 PM Mark Payne <markap14@hotmail.com <ma...@hotmail.com>> wrote:
>> Dan,
>>
>> Can you grab a thread dump and provide that? Specifically, the “main” thread is the important one with startup. The note that the role is already registered is normal. It probably could be changed to a DEBUG level, really. It should not be concerning. A thread dump, though, would show us exactly where it’s at.
>>
>> Thanks
>> -Mark
>>
>>
>>> On Aug 5, 2020, at 2:02 PM, dan young <danoyoung@gmail.com <ma...@gmail.com>> wrote:
>>>
>>> Hello,
>>> Running nifi 1.11.4, 3 X secure cluster mode and have enabled kerberos/sasl, upon trying to startup the cluster, they seem to get stuck in :
>>>
>>> 2020-08-05 17:10:18,907 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi
>>> when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again
>>> 2020-08-05 17:10:18,907 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election
>>> for role 'Cluster Coordinator' but this role is already registered
>>>
>>>
>>>
>>> I've checked zookeeper and I can see that the /nifi znode has been created, although empty, and the ACL seem to look correct
>>> zk: nifi1-5.X.net:2181 <http://nifi1-5.x.net:2181/>(CONNECTED) 3] getAcl /nifi
>>> 'sasl,'nifi@X.NET <ma...@X.NET>
>>> : cdrwa
>>> 'world,'anyone
>>> : r
>>>
>>>
>>> relevant Nifi config settings
>>>
>>> nifi.properties:
>>>
>>> nifi.zookeeper.auth.type=sasl
>>> nifi.zookeeper.kerberos.removeHostFromPrincipal=true
>>> nifi.zookeeper.kerberos.removeRealmFromPrincipal=false
>>>
>>> # kerberos #
>>> nifi.kerberos.krb5.file=/etc/krb5.conf
>>>
>>> # kerberos service principal #
>>> nifi.kerberos.service.principal=nifi@X.NET <ma...@X.NET>
>>> nifi.kerberos.service.keytab.location=/opt/nifi/conf/nifi.keytab
>>>
>>>
>>> state-management.xml
>>> <cluster-provider>
>>> <id>zk-provider</id>
>>> <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
>>> <property name="Root Node">/nifi</property>
>>> <property name="Session Timeout">30 seconds</property>
>>> <property name="Access Control">CreatorOnly</property>
>>> <property name="Connect String">X:2181,Y:2181,Z:2181</property>
>>> </cluster-provider>
>>>
>>>
>>>
>>> KRB5_TRACE=/dev/stdout kinit -k -t /opt/nifi/conf/nifi.keytab nifi@X.NET <ma...@X.NET>
>>> ...
>>> ...
>>>
>>> klist
>>> Ticket cache: FILE:/tmp/krb5cc_2004
>>> Default principal: nifi@X.NET <ma...@X.NET>
>>>
>>> Valid starting Expires Service principal
>>> 08/05/2020 17:57:02 08/06/2020 03:57:02 krbtgt/X.NET@X.NET <ma...@X.NET>
>>> renew until 08/06/2020 17:57:02
>>>
>>>
>>>
>>>
>>> As a side note, secure NiFi was working fine before the kerberos bit, I've been beating my head against the wall with it for the day, but the kerberos/zookeeper stuff seems to be working now....
>>> do we need to have Server-Server zookeeper auth working for this?
>>>
>>>
>>> Appreciate any insight....
>>>
>>> Regards,
>>>
>>> Dano
>>
>
Re: cluster stick in "Attempted to register Leader Election for role
'Cluster Coordinator' but this role is already registered"
Posted by dan young <da...@gmail.com>.
On a related note, I noticed that the ACL are getting set, but also for
each znode under the /nifi, the Read ACL for world is being set. Is there
a way to have nifi only set with the sasl?
zk: nifi1-5.X.net:2181(CONNECTED) 12] getAcl /nifi
'sasl,'nifi@X.NET
: cdrwa
'world,'anyone
: r
On Wed, Aug 5, 2020 at 1:56 PM Mark Payne <ma...@hotmail.com> wrote:
> No worries, thanks for following up and letting us know!
>
> Thanks
> -Mark
>
>
> On Aug 5, 2020, at 3:42 PM, dan young <da...@gmail.com> wrote:
>
> Hello,
>
> Sorry for all the noise...dooh....was due to the realm in the
> jaas.conf being lowercase...i'm a knucklehead...
>
> Dano
>
> On Wed, Aug 5, 2020 at 1:12 PM Bryan Bende <bb...@gmail.com> wrote:
>
>> I don't see how this would relate to the problem, but shouldn't the ACL
>> be set to "Creator" when Sasl/Kerberos is setup correctly?
>>
>> In addition to the nifi configs you showed, you would also need a jaas
>> conf file specified in bootstrap.conf and in that file you would need the
>> jaas entry for the ZK client.
>>
>> On Wed, Aug 5, 2020 at 3:02 PM dan young <da...@gmail.com> wrote:
>>
>>> Hello Mark,
>>>
>>> Attached is a dump from one of the nodes....I replaced the domain
>>> related entries with X/x. I'm not sure if it's relevant or not, but I did
>>> notice that in the log there's entries "Looking for keys for nifi@x.net"
>>> the x (domain) is lowercase whereas in the keytab file it's uppercase X.
>>> Also not sure if the Found unsupported keytype (1) is meaningful. Not that
>>> when I delete the znode in zookeeper=, at least the initial znode is
>>> created /nifi, but we never see the other typical suspect, i.e Coordinator,
>>> Primary, etc...
>>>
>>> Seems to be something stuck in Curator???
>>>
>>> Regards.
>>>
>>> Dano
>>>
>>> On Wed, Aug 5, 2020 at 12:20 PM Mark Payne <ma...@hotmail.com> wrote:
>>>
>>>> Dan,
>>>>
>>>> Can you grab a thread dump and provide that? Specifically, the “main”
>>>> thread is the important one with startup. The note that the role is already
>>>> registered is normal. It probably could be changed to a DEBUG level,
>>>> really. It should not be concerning. A thread dump, though, would show us
>>>> exactly where it’s at.
>>>>
>>>> Thanks
>>>> -Mark
>>>>
>>>>
>>>> On Aug 5, 2020, at 2:02 PM, dan young <da...@gmail.com> wrote:
>>>>
>>>> Hello,
>>>> Running nifi 1.11.4, 3 X secure cluster mode and have enabled
>>>> kerberos/sasl, upon trying to startup the cluster, they seem to get stuck
>>>> in :
>>>>
>>>> 2020-08-05 17:10:18,907 WARN [main]
>>>> o.a.nifi.controller.StandardFlowService There is currently no Cluster
>>>> Coordinator. This often happens upon restart of NiFi
>>>> when running an embedded ZooKeeper. Will register this node to become
>>>> the active Cluster Coordinator and will attempt to connect to cluster again
>>>> 2020-08-05 17:10:18,907 INFO [main]
>>>> o.a.n.c.l.e.CuratorLeaderElectionManager
>>>> CuratorLeaderElectionManager[stopped=false] Attempted to register Leader
>>>> Election
>>>> for role 'Cluster Coordinator' but this role is already registered
>>>>
>>>>
>>>>
>>>> I've checked zookeeper and I can see that the /nifi znode has been
>>>> created, although empty, and the ACL seem to look correct
>>>> zk: nifi1-5.X.net:2181 <http://nifi1-5.x.net:2181/>(CONNECTED) 3]
>>>> getAcl /nifi
>>>> 'sasl,'nifi@X.NET
>>>> : cdrwa
>>>> 'world,'anyone
>>>> : r
>>>>
>>>>
>>>> relevant Nifi config settings
>>>>
>>>> nifi.properties:
>>>>
>>>> nifi.zookeeper.auth.type=sasl
>>>> nifi.zookeeper.kerberos.removeHostFromPrincipal=true
>>>> nifi.zookeeper.kerberos.removeRealmFromPrincipal=false
>>>>
>>>> # kerberos #
>>>> nifi.kerberos.krb5.file=/etc/krb5.conf
>>>>
>>>> # kerberos service principal #
>>>> nifi.kerberos.service.principal=nifi@X.NET
>>>> nifi.kerberos.service.keytab.location=/opt/nifi/conf/nifi.keytab
>>>>
>>>>
>>>> state-management.xml
>>>> <cluster-provider>
>>>> <id>zk-provider</id>
>>>>
>>>> <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
>>>> <property name="Root Node">/nifi</property>
>>>> <property name="Session Timeout">30 seconds</property>
>>>> <property name="Access Control">CreatorOnly</property>
>>>> <property name="Connect String">X:2181,Y:2181,Z:2181</property>
>>>> </cluster-provider>
>>>>
>>>>
>>>>
>>>> KRB5_TRACE=/dev/stdout kinit -k -t /opt/nifi/conf/nifi.keytab
>>>> nifi@X.NET
>>>> ...
>>>> ...
>>>>
>>>> klist
>>>> Ticket cache: FILE:/tmp/krb5cc_2004
>>>> Default principal: nifi@X.NET
>>>>
>>>> Valid starting Expires Service principal
>>>> 08/05/2020 17:57:02 08/06/2020 03:57:02 krbtgt/X.NET@X.NET
>>>> renew until 08/06/2020 17:57:02
>>>>
>>>>
>>>>
>>>>
>>>> As a side note, secure NiFi was working fine before the kerberos bit,
>>>> I've been beating my head against the wall with it for the day, but the
>>>> kerberos/zookeeper stuff seems to be working now....
>>>> do we need to have Server-Server zookeeper auth working for this?
>>>>
>>>>
>>>> Appreciate any insight....
>>>>
>>>> Regards,
>>>>
>>>> Dano
>>>>
>>>>
>>>>
>
Re: cluster stick in "Attempted to register Leader Election for role
'Cluster Coordinator' but this role is already registered"
Posted by Mark Payne <ma...@hotmail.com>.
No worries, thanks for following up and letting us know!
Thanks
-Mark
On Aug 5, 2020, at 3:42 PM, dan young <da...@gmail.com>> wrote:
Hello,
Sorry for all the noise...dooh....was due to the realm in the jaas.conf being lowercase...i'm a knucklehead...
Dano
On Wed, Aug 5, 2020 at 1:12 PM Bryan Bende <bb...@gmail.com>> wrote:
I don't see how this would relate to the problem, but shouldn't the ACL be set to "Creator" when Sasl/Kerberos is setup correctly?
In addition to the nifi configs you showed, you would also need a jaas conf file specified in bootstrap.conf and in that file you would need the jaas entry for the ZK client.
On Wed, Aug 5, 2020 at 3:02 PM dan young <da...@gmail.com>> wrote:
Hello Mark,
Attached is a dump from one of the nodes....I replaced the domain related entries with X/x. I'm not sure if it's relevant or not, but I did notice that in the log there's entries "Looking for keys for nifi@x.net<ma...@x.net>" the x (domain) is lowercase whereas in the keytab file it's uppercase X. Also not sure if the Found unsupported keytype (1) is meaningful. Not that when I delete the znode in zookeeper=, at least the initial znode is created /nifi, but we never see the other typical suspect, i.e Coordinator, Primary, etc...
Seems to be something stuck in Curator???
Regards.
Dano
On Wed, Aug 5, 2020 at 12:20 PM Mark Payne <ma...@hotmail.com>> wrote:
Dan,
Can you grab a thread dump and provide that? Specifically, the “main” thread is the important one with startup. The note that the role is already registered is normal. It probably could be changed to a DEBUG level, really. It should not be concerning. A thread dump, though, would show us exactly where it’s at.
Thanks
-Mark
On Aug 5, 2020, at 2:02 PM, dan young <da...@gmail.com>> wrote:
Hello,
Running nifi 1.11.4, 3 X secure cluster mode and have enabled kerberos/sasl, upon trying to startup the cluster, they seem to get stuck in :
2020-08-05 17:10:18,907 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi
when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again
2020-08-05 17:10:18,907 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election
for role 'Cluster Coordinator' but this role is already registered
I've checked zookeeper and I can see that the /nifi znode has been created, although empty, and the ACL seem to look correct
zk: nifi1-5.X.net:2181<http://nifi1-5.x.net:2181/>(CONNECTED) 3] getAcl /nifi
'sasl,'nifi@X.NET<ma...@X.NET>
: cdrwa
'world,'anyone
: r
relevant Nifi config settings
nifi.properties:
nifi.zookeeper.auth.type=sasl
nifi.zookeeper.kerberos.removeHostFromPrincipal=true
nifi.zookeeper.kerberos.removeRealmFromPrincipal=false
# kerberos #
nifi.kerberos.krb5.file=/etc/krb5.conf
# kerberos service principal #
nifi.kerberos.service.principal=nifi@X.NET<ma...@X.NET>
nifi.kerberos.service.keytab.location=/opt/nifi/conf/nifi.keytab
state-management.xml
<cluster-provider>
<id>zk-provider</id>
<class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
<property name="Root Node">/nifi</property>
<property name="Session Timeout">30 seconds</property>
<property name="Access Control">CreatorOnly</property>
<property name="Connect String">X:2181,Y:2181,Z:2181</property>
</cluster-provider>
KRB5_TRACE=/dev/stdout kinit -k -t /opt/nifi/conf/nifi.keytab nifi@X.NET<ma...@X.NET>
...
...
klist
Ticket cache: FILE:/tmp/krb5cc_2004
Default principal: nifi@X.NET<ma...@X.NET>
Valid starting Expires Service principal
08/05/2020 17:57:02 08/06/2020 03:57:02 krbtgt/X.NET@X.NET<ma...@X.NET>
renew until 08/06/2020 17:57:02
As a side note, secure NiFi was working fine before the kerberos bit, I've been beating my head against the wall with it for the day, but the kerberos/zookeeper stuff seems to be working now....
do we need to have Server-Server zookeeper auth working for this?
Appreciate any insight....
Regards,
Dano
Re: cluster stick in "Attempted to register Leader Election for role
'Cluster Coordinator' but this role is already registered"
Posted by dan young <da...@gmail.com>.
Hello,
Sorry for all the noise...dooh....was due to the realm in the
jaas.conf being lowercase...i'm a knucklehead...
Dano
On Wed, Aug 5, 2020 at 1:12 PM Bryan Bende <bb...@gmail.com> wrote:
> I don't see how this would relate to the problem, but shouldn't the ACL be
> set to "Creator" when Sasl/Kerberos is setup correctly?
>
> In addition to the nifi configs you showed, you would also need a jaas
> conf file specified in bootstrap.conf and in that file you would need the
> jaas entry for the ZK client.
>
> On Wed, Aug 5, 2020 at 3:02 PM dan young <da...@gmail.com> wrote:
>
>> Hello Mark,
>>
>> Attached is a dump from one of the nodes....I replaced the domain related
>> entries with X/x. I'm not sure if it's relevant or not, but I did notice
>> that in the log there's entries "Looking for keys for nifi@x.net" the x
>> (domain) is lowercase whereas in the keytab file it's uppercase X. Also
>> not sure if the Found unsupported keytype (1) is meaningful. Not that when
>> I delete the znode in zookeeper=, at least the initial znode is created
>> /nifi, but we never see the other typical suspect, i.e Coordinator,
>> Primary, etc...
>>
>> Seems to be something stuck in Curator???
>>
>> Regards.
>>
>> Dano
>>
>> On Wed, Aug 5, 2020 at 12:20 PM Mark Payne <ma...@hotmail.com> wrote:
>>
>>> Dan,
>>>
>>> Can you grab a thread dump and provide that? Specifically, the “main”
>>> thread is the important one with startup. The note that the role is already
>>> registered is normal. It probably could be changed to a DEBUG level,
>>> really. It should not be concerning. A thread dump, though, would show us
>>> exactly where it’s at.
>>>
>>> Thanks
>>> -Mark
>>>
>>>
>>> On Aug 5, 2020, at 2:02 PM, dan young <da...@gmail.com> wrote:
>>>
>>> Hello,
>>> Running nifi 1.11.4, 3 X secure cluster mode and have enabled
>>> kerberos/sasl, upon trying to startup the cluster, they seem to get stuck
>>> in :
>>>
>>> 2020-08-05 17:10:18,907 WARN [main]
>>> o.a.nifi.controller.StandardFlowService There is currently no Cluster
>>> Coordinator. This often happens upon restart of NiFi
>>> when running an embedded ZooKeeper. Will register this node to become
>>> the active Cluster Coordinator and will attempt to connect to cluster again
>>> 2020-08-05 17:10:18,907 INFO [main]
>>> o.a.n.c.l.e.CuratorLeaderElectionManager
>>> CuratorLeaderElectionManager[stopped=false] Attempted to register Leader
>>> Election
>>> for role 'Cluster Coordinator' but this role is already registered
>>>
>>>
>>>
>>> I've checked zookeeper and I can see that the /nifi znode has been
>>> created, although empty, and the ACL seem to look correct
>>> zk: nifi1-5.X.net:2181 <http://nifi1-5.x.net:2181>(CONNECTED) 3] getAcl
>>> /nifi
>>> 'sasl,'nifi@X.NET
>>> : cdrwa
>>> 'world,'anyone
>>> : r
>>>
>>>
>>> relevant Nifi config settings
>>>
>>> nifi.properties:
>>>
>>> nifi.zookeeper.auth.type=sasl
>>> nifi.zookeeper.kerberos.removeHostFromPrincipal=true
>>> nifi.zookeeper.kerberos.removeRealmFromPrincipal=false
>>>
>>> # kerberos #
>>> nifi.kerberos.krb5.file=/etc/krb5.conf
>>>
>>> # kerberos service principal #
>>> nifi.kerberos.service.principal=nifi@X.NET
>>> nifi.kerberos.service.keytab.location=/opt/nifi/conf/nifi.keytab
>>>
>>>
>>> state-management.xml
>>> <cluster-provider>
>>> <id>zk-provider</id>
>>>
>>> <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
>>> <property name="Root Node">/nifi</property>
>>> <property name="Session Timeout">30 seconds</property>
>>> <property name="Access Control">CreatorOnly</property>
>>> <property name="Connect String">X:2181,Y:2181,Z:2181</property>
>>> </cluster-provider>
>>>
>>>
>>>
>>> KRB5_TRACE=/dev/stdout kinit -k -t /opt/nifi/conf/nifi.keytab nifi@X.NET
>>> ...
>>> ...
>>>
>>> klist
>>> Ticket cache: FILE:/tmp/krb5cc_2004
>>> Default principal: nifi@X.NET
>>>
>>> Valid starting Expires Service principal
>>> 08/05/2020 17:57:02 08/06/2020 03:57:02 krbtgt/X.NET@X.NET
>>> renew until 08/06/2020 17:57:02
>>>
>>>
>>>
>>>
>>> As a side note, secure NiFi was working fine before the kerberos bit,
>>> I've been beating my head against the wall with it for the day, but the
>>> kerberos/zookeeper stuff seems to be working now....
>>> do we need to have Server-Server zookeeper auth working for this?
>>>
>>>
>>> Appreciate any insight....
>>>
>>> Regards,
>>>
>>> Dano
>>>
>>>
>>>
Re: cluster stick in "Attempted to register Leader Election for role
'Cluster Coordinator' but this role is already registered"
Posted by Bryan Bende <bb...@gmail.com>.
I don't see how this would relate to the problem, but shouldn't the ACL be
set to "Creator" when Sasl/Kerberos is setup correctly?
In addition to the nifi configs you showed, you would also need a jaas conf
file specified in bootstrap.conf and in that file you would need the jaas
entry for the ZK client.
On Wed, Aug 5, 2020 at 3:02 PM dan young <da...@gmail.com> wrote:
> Hello Mark,
>
> Attached is a dump from one of the nodes....I replaced the domain related
> entries with X/x. I'm not sure if it's relevant or not, but I did notice
> that in the log there's entries "Looking for keys for nifi@x.net" the x
> (domain) is lowercase whereas in the keytab file it's uppercase X. Also
> not sure if the Found unsupported keytype (1) is meaningful. Not that when
> I delete the znode in zookeeper=, at least the initial znode is created
> /nifi, but we never see the other typical suspect, i.e Coordinator,
> Primary, etc...
>
> Seems to be something stuck in Curator???
>
> Regards.
>
> Dano
>
> On Wed, Aug 5, 2020 at 12:20 PM Mark Payne <ma...@hotmail.com> wrote:
>
>> Dan,
>>
>> Can you grab a thread dump and provide that? Specifically, the “main”
>> thread is the important one with startup. The note that the role is already
>> registered is normal. It probably could be changed to a DEBUG level,
>> really. It should not be concerning. A thread dump, though, would show us
>> exactly where it’s at.
>>
>> Thanks
>> -Mark
>>
>>
>> On Aug 5, 2020, at 2:02 PM, dan young <da...@gmail.com> wrote:
>>
>> Hello,
>> Running nifi 1.11.4, 3 X secure cluster mode and have enabled
>> kerberos/sasl, upon trying to startup the cluster, they seem to get stuck
>> in :
>>
>> 2020-08-05 17:10:18,907 WARN [main]
>> o.a.nifi.controller.StandardFlowService There is currently no Cluster
>> Coordinator. This often happens upon restart of NiFi
>> when running an embedded ZooKeeper. Will register this node to become
>> the active Cluster Coordinator and will attempt to connect to cluster again
>> 2020-08-05 17:10:18,907 INFO [main]
>> o.a.n.c.l.e.CuratorLeaderElectionManager
>> CuratorLeaderElectionManager[stopped=false] Attempted to register Leader
>> Election
>> for role 'Cluster Coordinator' but this role is already registered
>>
>>
>>
>> I've checked zookeeper and I can see that the /nifi znode has been
>> created, although empty, and the ACL seem to look correct
>> zk: nifi1-5.X.net:2181 <http://nifi1-5.x.net:2181>(CONNECTED) 3] getAcl
>> /nifi
>> 'sasl,'nifi@X.NET
>> : cdrwa
>> 'world,'anyone
>> : r
>>
>>
>> relevant Nifi config settings
>>
>> nifi.properties:
>>
>> nifi.zookeeper.auth.type=sasl
>> nifi.zookeeper.kerberos.removeHostFromPrincipal=true
>> nifi.zookeeper.kerberos.removeRealmFromPrincipal=false
>>
>> # kerberos #
>> nifi.kerberos.krb5.file=/etc/krb5.conf
>>
>> # kerberos service principal #
>> nifi.kerberos.service.principal=nifi@X.NET
>> nifi.kerberos.service.keytab.location=/opt/nifi/conf/nifi.keytab
>>
>>
>> state-management.xml
>> <cluster-provider>
>> <id>zk-provider</id>
>>
>> <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
>> <property name="Root Node">/nifi</property>
>> <property name="Session Timeout">30 seconds</property>
>> <property name="Access Control">CreatorOnly</property>
>> <property name="Connect String">X:2181,Y:2181,Z:2181</property>
>> </cluster-provider>
>>
>>
>>
>> KRB5_TRACE=/dev/stdout kinit -k -t /opt/nifi/conf/nifi.keytab nifi@X.NET
>> ...
>> ...
>>
>> klist
>> Ticket cache: FILE:/tmp/krb5cc_2004
>> Default principal: nifi@X.NET
>>
>> Valid starting Expires Service principal
>> 08/05/2020 17:57:02 08/06/2020 03:57:02 krbtgt/X.NET@X.NET
>> renew until 08/06/2020 17:57:02
>>
>>
>>
>>
>> As a side note, secure NiFi was working fine before the kerberos bit,
>> I've been beating my head against the wall with it for the day, but the
>> kerberos/zookeeper stuff seems to be working now....
>> do we need to have Server-Server zookeeper auth working for this?
>>
>>
>> Appreciate any insight....
>>
>> Regards,
>>
>> Dano
>>
>>
>>
Re: cluster stick in "Attempted to register Leader Election for role
'Cluster Coordinator' but this role is already registered"
Posted by dan young <da...@gmail.com>.
Hello Mark,
Attached is a dump from one of the nodes....I replaced the domain related
entries with X/x. I'm not sure if it's relevant or not, but I did notice
that in the log there's entries "Looking for keys for nifi@x.net" the x
(domain) is lowercase whereas in the keytab file it's uppercase X. Also
not sure if the Found unsupported keytype (1) is meaningful. Not that when
I delete the znode in zookeeper=, at least the initial znode is created
/nifi, but we never see the other typical suspect, i.e Coordinator,
Primary, etc...
Seems to be something stuck in Curator???
Regards.
Dano
On Wed, Aug 5, 2020 at 12:20 PM Mark Payne <ma...@hotmail.com> wrote:
> Dan,
>
> Can you grab a thread dump and provide that? Specifically, the “main”
> thread is the important one with startup. The note that the role is already
> registered is normal. It probably could be changed to a DEBUG level,
> really. It should not be concerning. A thread dump, though, would show us
> exactly where it’s at.
>
> Thanks
> -Mark
>
>
> On Aug 5, 2020, at 2:02 PM, dan young <da...@gmail.com> wrote:
>
> Hello,
> Running nifi 1.11.4, 3 X secure cluster mode and have enabled
> kerberos/sasl, upon trying to startup the cluster, they seem to get stuck
> in :
>
> 2020-08-05 17:10:18,907 WARN [main]
> o.a.nifi.controller.StandardFlowService There is currently no Cluster
> Coordinator. This often happens upon restart of NiFi
> when running an embedded ZooKeeper. Will register this node to become the
> active Cluster Coordinator and will attempt to connect to cluster again
> 2020-08-05 17:10:18,907 INFO [main]
> o.a.n.c.l.e.CuratorLeaderElectionManager
> CuratorLeaderElectionManager[stopped=false] Attempted to register Leader
> Election
> for role 'Cluster Coordinator' but this role is already registered
>
>
>
> I've checked zookeeper and I can see that the /nifi znode has been
> created, although empty, and the ACL seem to look correct
> zk: nifi1-5.X.net:2181 <http://nifi1-5.x.net:2181>(CONNECTED) 3] getAcl
> /nifi
> 'sasl,'nifi@X.NET
> : cdrwa
> 'world,'anyone
> : r
>
>
> relevant Nifi config settings
>
> nifi.properties:
>
> nifi.zookeeper.auth.type=sasl
> nifi.zookeeper.kerberos.removeHostFromPrincipal=true
> nifi.zookeeper.kerberos.removeRealmFromPrincipal=false
>
> # kerberos #
> nifi.kerberos.krb5.file=/etc/krb5.conf
>
> # kerberos service principal #
> nifi.kerberos.service.principal=nifi@X.NET
> nifi.kerberos.service.keytab.location=/opt/nifi/conf/nifi.keytab
>
>
> state-management.xml
> <cluster-provider>
> <id>zk-provider</id>
>
> <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
> <property name="Root Node">/nifi</property>
> <property name="Session Timeout">30 seconds</property>
> <property name="Access Control">CreatorOnly</property>
> <property name="Connect String">X:2181,Y:2181,Z:2181</property>
> </cluster-provider>
>
>
>
> KRB5_TRACE=/dev/stdout kinit -k -t /opt/nifi/conf/nifi.keytab nifi@X.NET
> ...
> ...
>
> klist
> Ticket cache: FILE:/tmp/krb5cc_2004
> Default principal: nifi@X.NET
>
> Valid starting Expires Service principal
> 08/05/2020 17:57:02 08/06/2020 03:57:02 krbtgt/X.NET@X.NET
> renew until 08/06/2020 17:57:02
>
>
>
>
> As a side note, secure NiFi was working fine before the kerberos bit, I've
> been beating my head against the wall with it for the day, but the
> kerberos/zookeeper stuff seems to be working now....
> do we need to have Server-Server zookeeper auth working for this?
>
>
> Appreciate any insight....
>
> Regards,
>
> Dano
>
>
>
Re: cluster stick in "Attempted to register Leader Election for role
'Cluster Coordinator' but this role is already registered"
Posted by Mark Payne <ma...@hotmail.com>.
Dan,
Can you grab a thread dump and provide that? Specifically, the “main” thread is the important one with startup. The note that the role is already registered is normal. It probably could be changed to a DEBUG level, really. It should not be concerning. A thread dump, though, would show us exactly where it’s at.
Thanks
-Mark
On Aug 5, 2020, at 2:02 PM, dan young <da...@gmail.com>> wrote:
Hello,
Running nifi 1.11.4, 3 X secure cluster mode and have enabled kerberos/sasl, upon trying to startup the cluster, they seem to get stuck in :
2020-08-05 17:10:18,907 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi
when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again
2020-08-05 17:10:18,907 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election
for role 'Cluster Coordinator' but this role is already registered
I've checked zookeeper and I can see that the /nifi znode has been created, although empty, and the ACL seem to look correct
zk: nifi1-5.X.net:2181<http://nifi1-5.x.net:2181>(CONNECTED) 3] getAcl /nifi
'sasl,'nifi@X.NET<ma...@X.NET>
: cdrwa
'world,'anyone
: r
relevant Nifi config settings
nifi.properties:
nifi.zookeeper.auth.type=sasl
nifi.zookeeper.kerberos.removeHostFromPrincipal=true
nifi.zookeeper.kerberos.removeRealmFromPrincipal=false
# kerberos #
nifi.kerberos.krb5.file=/etc/krb5.conf
# kerberos service principal #
nifi.kerberos.service.principal=nifi@X.NET<ma...@X.NET>
nifi.kerberos.service.keytab.location=/opt/nifi/conf/nifi.keytab
state-management.xml
<cluster-provider>
<id>zk-provider</id>
<class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
<property name="Root Node">/nifi</property>
<property name="Session Timeout">30 seconds</property>
<property name="Access Control">CreatorOnly</property>
<property name="Connect String">X:2181,Y:2181,Z:2181</property>
</cluster-provider>
KRB5_TRACE=/dev/stdout kinit -k -t /opt/nifi/conf/nifi.keytab nifi@X.NET<ma...@X.NET>
...
...
klist
Ticket cache: FILE:/tmp/krb5cc_2004
Default principal: nifi@X.NET<ma...@X.NET>
Valid starting Expires Service principal
08/05/2020 17:57:02 08/06/2020 03:57:02 krbtgt/X.NET@X.NET<ma...@X.NET>
renew until 08/06/2020 17:57:02
As a side note, secure NiFi was working fine before the kerberos bit, I've been beating my head against the wall with it for the day, but the kerberos/zookeeper stuff seems to be working now....
do we need to have Server-Server zookeeper auth working for this?
Appreciate any insight....
Regards,
Dano